Use BooleanBufferBuilder rather than Vec<bool> in ArrowBytesViewMap by etk18 · Pull Request #20064 · apache/datafusion

etk18 · 2026-01-29T15:30:25Z

Which issue does this PR close?

Closes #20053

Rationale for this change

ArrowBytesViewMap previously used Vec<bool> to track null values.
This PR replaces it with BooleanBufferBuilder, which is significantly more
memory-efficient and faster, and aligns with Apache Arrow best practices for
building validity bitmaps.

This change improves performance and memory usage without changing behavior.

What changed

Replaced Vec<bool> with BooleanBufferBuilder for null tracking
Updated null buffer construction to use Arrow-native buffers
Kept ordering and semantics unchanged

Tests

cargo test -p datafusion-physical-expr-common

Note: Full workspace tests require protoc; this PR was validated with the affected crate tests.

Screen.Recording.2026-01-29.at.8.34.41.PM.mov

etk18 · 2026-01-29T15:33:44Z

Thanks for the review!
Happy to make any adjustments if needed.

Dandandan · 2026-01-29T15:55:35Z

run benchmarks

alamb-ghbot · 2026-01-29T15:55:41Z

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing etk18/issue-20053/use-boolean-buffer-builder (fefefc8) to ed0a060 diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

datafusion/physical-expr-common/src/binary_view_map.rs

CONTRIBUTING.md

etk18 · 2026-01-29T16:09:10Z

Thanks for the review and suggestions!
I’ll update the implementation to use NullBufferBuilder and write validity bits directly to avoid the .not() call, and address the formatting suggestion as well.
I’ll push the updates by tomorrow.

Co-authored-by: Daniël Heres <danielheres@gmail.com>

alamb-ghbot · 2026-01-29T16:37:00Z

🤖: Benchmark completed

Details

Comparing HEAD and etk18_issue-20053_use-boolean-buffer-builder
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query    ┃        HEAD ┃ etk18_issue-20053_use-boolean-buffer-builder ┃        Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │  2345.81 ms │                                   2401.27 ms │     no change │
│ QQuery 1 │   912.49 ms │                                    868.90 ms │     no change │
│ QQuery 2 │  1802.68 ms │                                   1707.85 ms │ +1.06x faster │
│ QQuery 3 │  1072.79 ms │                                   1077.82 ms │     no change │
│ QQuery 4 │  2223.18 ms │                                   2244.09 ms │     no change │
│ QQuery 5 │ 28598.66 ms │                                  28989.60 ms │     no change │
│ QQuery 6 │  4028.85 ms │                                   4018.85 ms │     no change │
│ QQuery 7 │  2649.16 ms │                                   2745.97 ms │     no change │
└──────────┴─────────────┴──────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                           ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                           │ 43633.61ms │
│ Total Time (etk18_issue-20053_use-boolean-buffer-builder)   │ 44054.35ms │
│ Average Time (HEAD)                                         │  5454.20ms │
│ Average Time (etk18_issue-20053_use-boolean-buffer-builder) │  5506.79ms │
│ Queries Faster                                              │          1 │
│ Queries Slower                                              │          0 │
│ Queries with No Change                                      │          7 │
│ Queries with Failure                                        │          0 │
└─────────────────────────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃        HEAD ┃ etk18_issue-20053_use-boolean-buffer-builder ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │     2.73 ms │                                      3.19 ms │  1.17x slower │
│ QQuery 1  │    55.51 ms │                                     55.65 ms │     no change │
│ QQuery 2  │   145.61 ms │                                    136.79 ms │ +1.06x faster │
│ QQuery 3  │   160.85 ms │                                    160.89 ms │     no change │
│ QQuery 4  │  1040.78 ms │                                   1171.56 ms │  1.13x slower │
│ QQuery 5  │  1311.79 ms │                                   1394.04 ms │  1.06x slower │
│ QQuery 6  │    12.35 ms │                                     11.04 ms │ +1.12x faster │
│ QQuery 7  │    67.86 ms │                                     60.13 ms │ +1.13x faster │
│ QQuery 8  │  1456.20 ms │                                   1538.62 ms │  1.06x slower │
│ QQuery 9  │  1934.24 ms │                                   2001.71 ms │     no change │
│ QQuery 10 │   364.64 ms │                                    366.57 ms │     no change │
│ QQuery 11 │   417.96 ms │                                    418.66 ms │     no change │
│ QQuery 12 │  1287.00 ms │                                   1323.65 ms │     no change │
│ QQuery 13 │  1976.52 ms │                                   2014.34 ms │     no change │
│ QQuery 14 │  1237.20 ms │                                   1317.34 ms │  1.06x slower │
│ QQuery 15 │  1242.57 ms │                                   1298.77 ms │     no change │
│ QQuery 16 │  2638.81 ms │                                   2631.92 ms │     no change │
│ QQuery 17 │  2532.79 ms │                                   2565.09 ms │     no change │
│ QQuery 18 │  5869.19 ms │                                   4965.52 ms │ +1.18x faster │
│ QQuery 19 │   128.26 ms │                                    126.55 ms │     no change │
│ QQuery 20 │  1987.52 ms │                                   1949.88 ms │     no change │
│ QQuery 21 │  2273.91 ms │                                   2213.97 ms │     no change │
│ QQuery 22 │  4248.08 ms │                                   3741.98 ms │ +1.14x faster │
│ QQuery 23 │ 18434.78 ms │                                  12273.81 ms │ +1.50x faster │
│ QQuery 24 │   230.40 ms │                                    214.08 ms │ +1.08x faster │
│ QQuery 25 │   482.44 ms │                                    468.97 ms │     no change │
│ QQuery 26 │   223.48 ms │                                    235.84 ms │  1.06x slower │
│ QQuery 27 │  2736.69 ms │                                   2641.92 ms │     no change │
│ QQuery 28 │ 22225.19 ms │                                  21818.84 ms │     no change │
│ QQuery 29 │  1000.12 ms │                                    994.86 ms │     no change │
│ QQuery 30 │  1319.46 ms │                                   1309.25 ms │     no change │
│ QQuery 31 │  1383.40 ms │                                   1342.82 ms │     no change │
│ QQuery 32 │  5345.04 ms │                                   4231.51 ms │ +1.26x faster │
│ QQuery 33 │  5915.75 ms │                                   5504.05 ms │ +1.07x faster │
│ QQuery 34 │  8238.28 ms │                                   6022.30 ms │ +1.37x faster │
│ QQuery 35 │  2016.57 ms │                                   1986.94 ms │     no change │
│ QQuery 36 │   207.37 ms │                                    201.99 ms │     no change │
│ QQuery 37 │    84.25 ms │                                     81.96 ms │     no change │
│ QQuery 38 │   128.19 ms │                                    121.13 ms │ +1.06x faster │
│ QQuery 39 │   374.62 ms │                                    354.26 ms │ +1.06x faster │
│ QQuery 40 │    47.71 ms │                                     48.57 ms │     no change │
│ QQuery 41 │    41.80 ms │                                     40.37 ms │     no change │
│ QQuery 42 │    36.72 ms │                                     35.47 ms │     no change │
└───────────┴─────────────┴──────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary                                           ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD)                                           │ 102864.63ms │
│ Total Time (etk18_issue-20053_use-boolean-buffer-builder)   │  91396.79ms │
│ Average Time (HEAD)                                         │   2392.20ms │
│ Average Time (etk18_issue-20053_use-boolean-buffer-builder) │   2125.51ms │
│ Queries Faster                                              │          12 │
│ Queries Slower                                              │           6 │
│ Queries with No Change                                      │          25 │
│ Queries with Failure                                        │           0 │
└─────────────────────────────────────────────────────────────┴─────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃      HEAD ┃ etk18_issue-20053_use-boolean-buffer-builder ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │ 113.31 ms │                                    104.95 ms │ +1.08x faster │
│ QQuery 2  │  36.64 ms │                                     35.11 ms │     no change │
│ QQuery 3  │  43.10 ms │                                     43.92 ms │     no change │
│ QQuery 4  │  34.51 ms │                                     32.81 ms │     no change │
│ QQuery 5  │  99.79 ms │                                     96.42 ms │     no change │
│ QQuery 6  │  21.75 ms │                                     21.43 ms │     no change │
│ QQuery 7  │ 178.79 ms │                                    162.01 ms │ +1.10x faster │
│ QQuery 8  │  40.65 ms │                                     43.85 ms │  1.08x slower │
│ QQuery 9  │ 112.30 ms │                                    112.41 ms │     no change │
│ QQuery 10 │  73.43 ms │                                     68.21 ms │ +1.08x faster │
│ QQuery 11 │  20.00 ms │                                     18.97 ms │ +1.05x faster │
│ QQuery 12 │  54.05 ms │                                     53.11 ms │     no change │
│ QQuery 13 │  52.98 ms │                                     53.58 ms │     no change │
│ QQuery 14 │  16.54 ms │                                     15.79 ms │     no change │
│ QQuery 15 │  31.86 ms │                                     31.04 ms │     no change │
│ QQuery 16 │  30.29 ms │                                     29.55 ms │     no change │
│ QQuery 17 │ 154.24 ms │                                    153.93 ms │     no change │
│ QQuery 18 │ 305.35 ms │                                    298.06 ms │     no change │
│ QQuery 19 │  41.46 ms │                                     42.24 ms │     no change │
│ QQuery 20 │  55.98 ms │                                     57.47 ms │     no change │
│ QQuery 21 │ 201.01 ms │                                    199.98 ms │     no change │
│ QQuery 22 │  24.03 ms │                                     24.23 ms │     no change │
└───────────┴───────────┴──────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                                           ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                                           │ 1742.09ms │
│ Total Time (etk18_issue-20053_use-boolean-buffer-builder)   │ 1699.09ms │
│ Average Time (HEAD)                                         │   79.19ms │
│ Average Time (etk18_issue-20053_use-boolean-buffer-builder) │   77.23ms │
│ Queries Faster                                              │         4 │
│ Queries Slower                                              │         1 │
│ Queries with No Change                                      │        17 │
│ Queries with Failure                                        │         0 │
└─────────────────────────────────────────────────────────────┴───────────┘

alamb

Thanks @etk18 and @Dandandan -- this looks cool -- Using NullBufferBuilder will be even better. Thank you

CONTRIBUTING.md

etk18 · 2026-01-30T05:26:50Z

Thanks for the reviews!

I’ve updated the PR to use NullBufferBuilder and addressed the feedback.
CI issues on my local machine were due to missing system tools, but the code changes themselves are complete.

Happy to rebase or make further adjustments if needed.

etk18 · 2026-01-30T05:53:00Z

Thanks for the review!

I’ve pushed updates to:

Fix Prettier formatting in CONTRIBUTING.md
Address the review feedback

CI is rerunning now. Please let me know if anything else is needed.

Dandandan · 2026-01-30T06:04:06Z

run benchmarks

alamb-ghbot · 2026-01-30T06:04:12Z

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing etk18/issue-20053/use-boolean-buffer-builder (c954c1a) to ed0a060 diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb-ghbot · 2026-01-30T06:44:59Z

🤖: Benchmark completed

Details

Comparing HEAD and etk18_issue-20053_use-boolean-buffer-builder
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query    ┃        HEAD ┃ etk18_issue-20053_use-boolean-buffer-builder ┃        Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │  2759.67 ms │                                   2601.01 ms │ +1.06x faster │
│ QQuery 1 │   988.35 ms │                                    972.23 ms │     no change │
│ QQuery 2 │  2011.85 ms │                                   1984.20 ms │     no change │
│ QQuery 3 │  1149.92 ms │                                   1112.58 ms │     no change │
│ QQuery 4 │  2412.59 ms │                                   2400.34 ms │     no change │
│ QQuery 5 │ 31165.64 ms │                                  30873.38 ms │     no change │
│ QQuery 6 │  4225.10 ms │                                   4166.18 ms │     no change │
│ QQuery 7 │  3136.36 ms │                                   3049.33 ms │     no change │
└──────────┴─────────────┴──────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                           ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                           │ 47849.47ms │
│ Total Time (etk18_issue-20053_use-boolean-buffer-builder)   │ 47159.25ms │
│ Average Time (HEAD)                                         │  5981.18ms │
│ Average Time (etk18_issue-20053_use-boolean-buffer-builder) │  5894.91ms │
│ Queries Faster                                              │          1 │
│ Queries Slower                                              │          0 │
│ Queries with No Change                                      │          7 │
│ Queries with Failure                                        │          0 │
└─────────────────────────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃        HEAD ┃ etk18_issue-20053_use-boolean-buffer-builder ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │     2.56 ms │                                      3.32 ms │  1.30x slower │
│ QQuery 1  │    51.93 ms │                                     57.74 ms │  1.11x slower │
│ QQuery 2  │   134.10 ms │                                    142.75 ms │  1.06x slower │
│ QQuery 3  │   155.22 ms │                                    167.08 ms │  1.08x slower │
│ QQuery 4  │  1020.33 ms │                                   1166.67 ms │  1.14x slower │
│ QQuery 5  │  1278.56 ms │                                   1453.50 ms │  1.14x slower │
│ QQuery 6  │     9.10 ms │                                     10.41 ms │  1.14x slower │
│ QQuery 7  │    56.78 ms │                                     61.42 ms │  1.08x slower │
│ QQuery 8  │  1398.49 ms │                                   1555.69 ms │  1.11x slower │
│ QQuery 9  │  1821.36 ms │                                   2104.13 ms │  1.16x slower │
│ QQuery 10 │   351.11 ms │                                    388.65 ms │  1.11x slower │
│ QQuery 11 │   409.12 ms │                                    440.11 ms │  1.08x slower │
│ QQuery 12 │  1209.03 ms │                                   1380.48 ms │  1.14x slower │
│ QQuery 13 │  1919.06 ms │                                   2163.13 ms │  1.13x slower │
│ QQuery 14 │  1212.45 ms │                                   1393.66 ms │  1.15x slower │
│ QQuery 15 │  1177.86 ms │                                   1348.67 ms │  1.15x slower │
│ QQuery 16 │  2585.89 ms │                                   2819.85 ms │  1.09x slower │
│ QQuery 17 │  2499.67 ms │                                   2707.05 ms │  1.08x slower │
│ QQuery 18 │  6306.19 ms │                                   5355.51 ms │ +1.18x faster │
│ QQuery 19 │   133.52 ms │                                    134.33 ms │     no change │
│ QQuery 20 │  2134.01 ms │                                   2051.07 ms │     no change │
│ QQuery 21 │  2396.51 ms │                                   2374.97 ms │     no change │
│ QQuery 22 │  4337.13 ms │                                   4039.35 ms │ +1.07x faster │
│ QQuery 23 │ 13109.83 ms │                                  12752.73 ms │     no change │
│ QQuery 24 │   221.47 ms │                                    230.81 ms │     no change │
│ QQuery 25 │   494.34 ms │                                    515.35 ms │     no change │
│ QQuery 26 │   244.92 ms │                                    234.06 ms │     no change │
│ QQuery 27 │  2888.58 ms │                                   2823.09 ms │     no change │
│ QQuery 28 │ 23100.78 ms │                                  23715.55 ms │     no change │
│ QQuery 29 │  1043.13 ms │                                    979.42 ms │ +1.07x faster │
│ QQuery 30 │  1380.56 ms │                                   1365.30 ms │     no change │
│ QQuery 31 │  1453.36 ms │                                   1411.38 ms │     no change │
│ QQuery 32 │  5257.16 ms │                                   4706.19 ms │ +1.12x faster │
│ QQuery 33 │  6267.01 ms │                                   5789.88 ms │ +1.08x faster │
│ QQuery 34 │  6644.80 ms │                                   6268.60 ms │ +1.06x faster │
│ QQuery 35 │  2201.15 ms │                                   2101.08 ms │     no change │
│ QQuery 36 │   231.24 ms │                                    222.24 ms │     no change │
│ QQuery 37 │    91.79 ms │                                     86.21 ms │ +1.06x faster │
│ QQuery 38 │   131.99 ms │                                    129.31 ms │     no change │
│ QQuery 39 │   426.34 ms │                                    419.99 ms │     no change │
│ QQuery 40 │    52.61 ms │                                     55.37 ms │  1.05x slower │
│ QQuery 41 │    45.72 ms │                                     43.53 ms │     no change │
│ QQuery 42 │    39.48 ms │                                     39.37 ms │     no change │
└───────────┴─────────────┴──────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                           ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                           │ 97926.22ms │
│ Total Time (etk18_issue-20053_use-boolean-buffer-builder)   │ 97209.01ms │
│ Average Time (HEAD)                                         │  2277.35ms │
│ Average Time (etk18_issue-20053_use-boolean-buffer-builder) │  2260.67ms │
│ Queries Faster                                              │          7 │
│ Queries Slower                                              │         19 │
│ Queries with No Change                                      │         17 │
│ Queries with Failure                                        │          0 │
└─────────────────────────────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query     ┃      HEAD ┃ etk18_issue-20053_use-boolean-buffer-builder ┃       Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1  │ 102.10 ms │                                    100.50 ms │    no change │
│ QQuery 2  │  32.23 ms │                                     33.04 ms │    no change │
│ QQuery 3  │  40.23 ms │                                     39.51 ms │    no change │
│ QQuery 4  │  30.94 ms │                                     31.06 ms │    no change │
│ QQuery 5  │  90.92 ms │                                     89.11 ms │    no change │
│ QQuery 6  │  20.74 ms │                                     20.83 ms │    no change │
│ QQuery 7  │ 160.07 ms │                                    154.04 ms │    no change │
│ QQuery 8  │  38.87 ms │                                     40.44 ms │    no change │
│ QQuery 9  │  95.79 ms │                                    104.79 ms │ 1.09x slower │
│ QQuery 10 │  66.84 ms │                                     67.71 ms │    no change │
│ QQuery 11 │  18.25 ms │                                     18.23 ms │    no change │
│ QQuery 12 │  52.13 ms │                                     51.28 ms │    no change │
│ QQuery 13 │  49.10 ms │                                     47.51 ms │    no change │
│ QQuery 14 │  15.22 ms │                                     15.02 ms │    no change │
│ QQuery 15 │  29.68 ms │                                     30.61 ms │    no change │
│ QQuery 16 │  28.22 ms │                                     29.37 ms │    no change │
│ QQuery 17 │ 142.81 ms │                                    143.59 ms │    no change │
│ QQuery 18 │ 274.93 ms │                                    281.89 ms │    no change │
│ QQuery 19 │  39.37 ms │                                     39.84 ms │    no change │
│ QQuery 20 │  54.80 ms │                                     57.34 ms │    no change │
│ QQuery 21 │ 192.97 ms │                                    189.52 ms │    no change │
│ QQuery 22 │  22.95 ms │                                     23.14 ms │    no change │
└───────────┴───────────┴──────────────────────────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                                           ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                                           │ 1599.17ms │
│ Total Time (etk18_issue-20053_use-boolean-buffer-builder)   │ 1608.38ms │
│ Average Time (HEAD)                                         │   72.69ms │
│ Average Time (etk18_issue-20053_use-boolean-buffer-builder) │   73.11ms │
│ Queries Faster                                              │         0 │
│ Queries Slower                                              │         1 │
│ Queries with No Change                                      │        21 │
│ Queries with Failure                                        │         0 │
└─────────────────────────────────────────────────────────────┴───────────┘

Dandandan · 2026-01-30T10:34:57Z

Thank you @etk18 !

alamb · 2026-01-30T11:43:58Z

datafusion/physical-expr-common/src/binary_view_map.rs

-        } else {
-            None
-        };
+        let null_buffer = self.nulls.finish();


alamb · 2026-01-30T11:45:43Z

datafusion/physical-expr-common/src/binary_view_map.rs

        let in_progress_size = self.in_progress.capacity();
        let completed_size: usize = self.completed.iter().map(|b| b.len()).sum();
-        let nulls_size = self.nulls.len();
+        let nulls_size = self.nulls.len() / 8;


I think this should use https://docs.rs/arrow/latest/arrow/array/struct.NullBufferBuilder.html#method.allocated_size as it is meant to track the allocated size (not the used size)

Filed Improve memory accounting for ArrowBytesViewMap #20074

Use BooleanBufferBuilder rather than Vec<bool> in ArrowBytesViewMap

fefefc8

github-actions bot added the physical-expr Changes to the physical-expr crates label Jan 29, 2026

Dandandan reviewed Jan 29, 2026

View reviewed changes

datafusion/physical-expr-common/src/binary_view_map.rs Outdated Show resolved Hide resolved

Dandandan reviewed Jan 29, 2026

View reviewed changes

datafusion/physical-expr-common/src/binary_view_map.rs Outdated Show resolved Hide resolved

Dandandan reviewed Jan 29, 2026

View reviewed changes

CONTRIBUTING.md Outdated Show resolved Hide resolved

Update CONTRIBUTING.md

c01d32b

Co-authored-by: Daniël Heres <danielheres@gmail.com>

alamb reviewed Jan 29, 2026

View reviewed changes

CONTRIBUTING.md Outdated Show resolved Hide resolved

Eesh Sagar Singh added 3 commits January 30, 2026 09:48

Remove stray character from CONTRIBUTING.md

8d9b6a0

Remove stray character from CONTRIBUTING.md

63e4ea2

Use NullBufferBuilder for null tracking in ArrowBytesViewMap

9e6cc0b

Run cargo fmt

c954c1a

Dandandan approved these changes Jan 30, 2026

View reviewed changes

Dandandan added this pull request to the merge queue Jan 30, 2026

Merged via the queue into apache:main with commit f0de02f Jan 30, 2026
32 checks passed

alamb reviewed Jan 30, 2026

View reviewed changes

alamb mentioned this pull request Jan 30, 2026

Improve memory accounting for ArrowBytesViewMap #20074

Closed

Comments

Conversation

etk18 commented Jan 29, 2026

Which issue does this PR close?

Rationale for this change

What changed

Tests

Uh oh!

etk18 commented Jan 29, 2026

Uh oh!

Dandandan commented Jan 29, 2026

Uh oh!

alamb-ghbot commented Jan 29, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

etk18 commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb-ghbot commented Jan 29, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

etk18 commented Jan 30, 2026

Uh oh!

etk18 commented Jan 30, 2026

Uh oh!

Dandandan commented Jan 30, 2026

Uh oh!

alamb-ghbot commented Jan 30, 2026

Uh oh!

alamb-ghbot commented Jan 30, 2026

Uh oh!

Dandandan commented Jan 30, 2026

Uh oh!

Uh oh!

alamb Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

etk18 commented Jan 29, 2026 •

edited

Loading