Union with serialized bitmap

[forsaken628]

This PR is related to #263 , #281 and the computation process was refactored using the Visitor pattern to facilitate future extensions

[Kerollmops]

Hey @forsaken628 👋

Thank you for the proposal. However, I doubt that there is a performance gain. Could you please set up some benchmarks and run them?

Have a nice day 🌵

[forsaken628]

There are performance improvements for heavy computation and some regressions for light computation

pairwise_intersection_with_serialized_unchecked/ref_own/census-income
                        time:   [1.3265 ms 1.3423 ms 1.3624 ms]
                        change: [-3.9036% -2.6485% -1.1383%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) high mild
  2 (2.00%) high severe
Benchmarking pairwise_intersection_with_serialized_unchecked/ref_own/census-income_srt: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 8.0s, enable flat sampling, or reduce sample count to 50.
pairwise_intersection_with_serialized_unchecked/ref_own/census-income_srt
                        time:   [762.64 µs 775.15 µs 785.99 µs]
                        change: [-2.0298% +0.5733% +3.1849%] (p = 0.66 > 0.05)
                        No change in performance detected.
Benchmarking pairwise_intersection_with_serialized_unchecked/ref_own/census1881: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.4s, enable flat sampling, or reduce sample count to 60.
pairwise_intersection_with_serialized_unchecked/ref_own/census1881
                        time:   [156.06 µs 160.07 µs 163.52 µs]
                        change: [-22.497% -5.9705% +6.8403%] (p = 0.69 > 0.05)
                        No change in performance detected.
pairwise_intersection_with_serialized_unchecked/ref_own/census1881_srt
                        time:   [103.83 µs 106.85 µs 109.57 µs]
                        change: [+1.7995% +10.292% +16.093%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
  7 (7.00%) high mild
pairwise_intersection_with_serialized_unchecked/ref_own/weather_sept_85
                        time:   [5.8185 ms 5.8632 ms 5.9071 ms]
                        change: [-2.6190% -1.9004% -1.2175%] (p = 0.00 < 0.05)
                        Performance has improved.
pairwise_intersection_with_serialized_unchecked/ref_own/weather_sept_85_srt
                        time:   [1.3805 ms 1.3910 ms 1.4014 ms]
                        change: [+0.0140% +0.7875% +1.6099%] (p = 0.06 > 0.05)
                        No change in performance detected.
pairwise_intersection_with_serialized_unchecked/ref_own/wikileaks-noquotes
                        time:   [314.08 µs 317.08 µs 320.27 µs]
                        change: [+1.3629% +2.8094% +5.0484%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
  4 (4.00%) high mild
  1 (1.00%) high severe
pairwise_intersection_with_serialized_unchecked/ref_own/wikileaks-noquotes_srt
                        time:   [153.52 µs 155.01 µs 156.37 µs]
                        change: [+3.5969% +4.7044% +5.9526%] (p = 0.00 < 0.05)
                        Performance has regressed.

[Kerollmops]

Hey @forsaken628 👋

Thanks for the proposal. However, I am wondering whether the approach you took is what I expected. I see that you introduced new internal types to read the containers and skip the unnecessary ones, and that you modified the code in the intersection part to use your new visitor structs.

However, I would prefer that you don't modify the existing code for now and introduce a new union_with_serialized_unchecked method akin to intersection_with_serialized_unchecked first, please.

Have a great day 🌵