Introduce morsel-driven Parquet scan by Dandandan · Pull Request #20481 · apache/datafusion

Dandandan · 2026-02-22T17:48:59Z

Which issue does this PR close?

Closes Introduce morsel-driven Parquet scan #20529

TODO:

Make sure workqueue is closed after execute stream is dropped / not shared between two concurrent instances (major design issue)
Test changing planning again to only plan whole files instead of "early splitting" (got some slowdown on this - perhaps it removes some parallelism)
Update test to work with morsel setting enabled
Fix tests
Compare different approaches
Update docs
Simplify / address feedback

Rationale for this change

Current parelllization of Parquet scan is bounded by the thread that has the most data / is the slowest to execute, which means in the case of data skew (driven by either larger partitions or less selective filters during pruning / filter pushdown..., variable object store latency), the parallelism will be significantly limited.

We can change the strategy by morsel-driven parallelism like described in https://db.in.tum.de/~leis/papers/morsels.pdf.

Doing so is faster for a lot of queries, when there is an amount of skew (such as clickbench) and we have enough row filters to spread out the work.
For clickbench_partitioned / clickbench_pushdown it seems up to ~2x as fast for some queries, on a 10 core machine.

It seems to have almost no regressions, perhaps some due to different file scanning order(?) - so different statistics that can be used to prune and thus some minor variation.

Morsel-Driven Execution Architecture (partly claude-generated)

This branch implements a morsel-driven execution model for Parquet scans, based on the concept
from the Morsel-Driven Parallelism paper (Leis et al.). The core idea: instead of statically
assigning files to partitions, all work is pooled in a shared queue that all partition streams pull
from dynamically.

The Problem It Solves

In the traditional model, partition 0 might get a 1 GB file while partition 1 gets nothing --
partition 1 idles while 0 is busy. Currently we already try to statically spread out work to n partitions / threads based on stats (which works very well on perfectly distributed scans on SSDs (e.g. TPCH running locally), this doesn't work well when there is any data skew caused by any of those:

filters being more selective on part of the data
high variation in object store response times
overhead of filter pushdown + more serial/smaller reads and file operations

Morsel-driven execution prevents this by sharing work dynamically.

Key Types

ParquetMorsel -- datafusion/datasource-parquet/src/opener.rs:129

A morsel = one row group of a Parquet file. Stored as an extension on PartitionedFile.

  pub struct ParquetMorsel {
      pub metadata: Arc<ParquetMetaData>,   // cached, shared across morsels from same file
      pub access_plan: ParquetAccessPlan,   // which row groups to read
  }

WorkQueue -- datafusion/datasource/src/file_stream.rs:410

The shared, thread-safe queue. Each partition stream calls pull() which returns:

Work(file) -- here's a file/morsel to process
Wait -- queue is empty but workers are still morselizing (wait for notification)
Done -- all work consumed

MorselState -- datafusion/datasource/src/source.rs:240

Tracks the shared queue lifecycle. A new queue is created once per execution cycle when all
partition streams have opened.

struct MorselState {
    queue: Option<Arc<WorkQueue>>,
    streams_opened: usize,
    expected_streams: usize,
}

MorselizingGuard -- datafusion/datasource/src/file_stream.rs:49

RAII wrapper that atomically decrements morselizing_count when a worker finishes -- enabling
WorkStatus::Wait vs Done decisions.

`FileOpener` Trait Extension -- `datafusion/datasource/src/file_stream.rs:498`

A new morselize() method is added to FileOpener. The default implementation is a no-op
(returns the file as-is). ParquetOpener overrides it to split files by row group.

pub trait FileOpener {
    fn open(&self, file: PartitionedFile) -> Result<FileOpenFuture>;

    // NEW: split a file into morsels (default = no-op)
    fn morselize(&self, file: PartitionedFile)
        -> BoxFuture<'static, Result<Vec<PartitionedFile>>> { ... }
}

ParquetOpener::morselize() at opener.rs:232:

Loads Parquet metadata once (shared via Arc across all resulting morsels)
Applies file-level and row-group-level statistics pruning
Returns one PartitionedFile per surviving row group, each carrying a ParquetMorsel extension

`FileStream` State Machine -- `datafusion/datasource/src/file_stream.rs:141`

The morsel-driven path adds two new states (Morselizing and Waiting):

  Idle
   +-- morsel_driven=true  -> queue.pull()
   |    +-- Work(file) -> Morselizing (run morselize())
   |    +-- Wait       -> Waiting (yield, wake on notify)
   |    +-- Done       -> return None
   +-- morsel_driven=false -> traditional local file_iter

  Morselizing
   +-- morsels > 1 -> push_many() back to queue -> Idle
   +-- morsel = 1  -> Open

  Waiting
   +-- notified -> Idle

Configuration

datafusion.execution.parquet.allow_morsel_driven -- datafusion/common/src/config.rs:748

Default: true. Can be disabled per-session.

FileScanConfig::morsel_driven -- datafusion/datasource/src/file_scan_config.rs:211

Automatically disabled when:

partitioned_by_file_group = true (breaks hash-partitioning guarantees)
preserve_order = true (breaks SortPreservingMerge guarantees)

Benchmark results

Summary: both clickbench, clickbench_partitioned

Details


🤖: Benchmark completed

Details

Comparing HEAD and parquet-morsel-driven-execution-237164415184908839
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Query    ┃        HEAD ┃ parquet-morsel-driven-execution-237164415184908839 ┃          Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ QQuery 0 │  2445.21 ms │                                         2334.07 ms │       no change │
│ QQuery 1 │   984.42 ms │                                          831.92 ms │   +1.18x faster │
│ QQuery 2 │  1847.13 ms │                                         1609.38 ms │   +1.15x faster │
│ QQuery 3 │  1078.65 ms │                                          973.60 ms │   +1.11x faster │
│ QQuery 4 │  2279.11 ms │                                         2070.83 ms │   +1.10x faster │
│ QQuery 5 │ 26832.66 ms │                                        26954.54 ms │       no change │
│ QQuery 6 │  3852.58 ms │                                            8.28 ms │ +465.50x faster │
│ QQuery 7 │  2619.05 ms │                                         2547.13 ms │       no change │
└──────────┴─────────────┴────────────────────────────────────────────────────┴─────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                                 ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                                 │ 41938.81ms │
│ Total Time (parquet-morsel-driven-execution-237164415184908839)   │ 37329.75ms │
│ Average Time (HEAD)                                               │  5242.35ms │
│ Average Time (parquet-morsel-driven-execution-237164415184908839) │  4666.22ms │
│ Queries Faster                                                    │          5 │
│ Queries Slower                                                    │          0 │
│ Queries with No Change                                            │          3 │
│ Queries with Failure                                              │          0 │
└───────────────────────────────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃        HEAD ┃ parquet-morsel-driven-execution-237164415184908839 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │     2.59 ms │                                            2.59 ms │     no change │
│ QQuery 1  │    52.70 ms │                                           53.83 ms │     no change │
│ QQuery 2  │   170.22 ms │                                          157.23 ms │ +1.08x faster │
│ QQuery 3  │   165.42 ms │                                          167.38 ms │     no change │
│ QQuery 4  │  1090.62 ms │                                         1035.57 ms │ +1.05x faster │
│ QQuery 5  │  1240.29 ms │                                         1229.64 ms │     no change │
│ QQuery 6  │     6.51 ms │                                            7.08 ms │  1.09x slower │
│ QQuery 7  │    49.50 ms │                                           60.72 ms │  1.23x slower │
│ QQuery 8  │  1388.32 ms │                                         1441.47 ms │     no change │
│ QQuery 9  │  1743.04 ms │                                         1723.40 ms │     no change │
│ QQuery 10 │   337.39 ms │                                          335.23 ms │     no change │
│ QQuery 11 │   386.04 ms │                                          380.30 ms │     no change │
│ QQuery 12 │  1183.09 ms │                                         1140.56 ms │     no change │
│ QQuery 13 │  1871.04 ms │                                         1609.71 ms │ +1.16x faster │
│ QQuery 14 │  1231.91 ms │                                         1158.10 ms │ +1.06x faster │
│ QQuery 15 │  1172.84 ms │                                         1210.94 ms │     no change │
│ QQuery 16 │  2418.93 ms │                                         2444.96 ms │     no change │
│ QQuery 17 │  2394.23 ms │                                         2427.85 ms │     no change │
│ QQuery 18 │  4803.83 ms │                                         4575.30 ms │     no change │
│ QQuery 19 │   123.68 ms │                                          124.92 ms │     no change │
│ QQuery 20 │  1849.23 ms │                                         1689.57 ms │ +1.09x faster │
│ QQuery 21 │  2137.19 ms │                                         1979.95 ms │ +1.08x faster │
│ QQuery 22 │  3705.39 ms │                                         3533.51 ms │     no change │
│ QQuery 23 │ 16569.27 ms │                                        12289.08 ms │ +1.35x faster │
│ QQuery 24 │   212.07 ms │                                          137.98 ms │ +1.54x faster │
│ QQuery 25 │   476.64 ms │                                          406.70 ms │ +1.17x faster │
│ QQuery 26 │   205.82 ms │                                          141.86 ms │ +1.45x faster │
│ QQuery 27 │  2643.51 ms │                                         2462.51 ms │ +1.07x faster │
│ QQuery 28 │ 23612.99 ms │                                        22934.78 ms │     no change │
│ QQuery 29 │  1037.09 ms │                                         1008.12 ms │     no change │
│ QQuery 30 │  1237.34 ms │                                         1236.45 ms │     no change │
│ QQuery 31 │  1316.76 ms │                                         1286.78 ms │     no change │
│ QQuery 32 │  4511.21 ms │                                         4096.49 ms │ +1.10x faster │
│ QQuery 33 │  5462.02 ms │                                         5437.33 ms │     no change │
│ QQuery 34 │  5792.16 ms │                                         5907.89 ms │     no change │
│ QQuery 35 │  1885.60 ms │                                         1896.64 ms │     no change │
│ QQuery 36 │   197.18 ms │                                          127.31 ms │ +1.55x faster │
│ QQuery 37 │    72.42 ms │                                           55.88 ms │ +1.30x faster │
│ QQuery 38 │   112.79 ms │                                           68.44 ms │ +1.65x faster │
│ QQuery 39 │   336.37 ms │                                          224.29 ms │ +1.50x faster │
│ QQuery 40 │    41.22 ms │                                           28.57 ms │ +1.44x faster │
│ QQuery 41 │    34.57 ms │                                           27.29 ms │ +1.27x faster │
│ QQuery 42 │    30.75 ms │                                           21.99 ms │ +1.40x faster │
└───────────┴─────────────┴────────────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                                 ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                                 │ 95311.78ms │
│ Total Time (parquet-morsel-driven-execution-237164415184908839)   │ 88286.18ms │
│ Average Time (HEAD)                                               │  2216.55ms │
│ Average Time (parquet-morsel-driven-execution-237164415184908839) │  2053.17ms │
│ Queries Faster                                                    │         19 │
│ Queries Slower                                                    │          2 │
│ Queries with No Change                                            │         22 │
│ Queries with Failure                                              │          0 │
└───────────────────────────────────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃      HEAD ┃ parquet-morsel-driven-execution-237164415184908839 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │ 125.24 ms │                                          129.21 ms │     no change │
│ QQuery 2  │  33.04 ms │                                           32.48 ms │     no change │
│ QQuery 3  │  39.89 ms │                                           41.09 ms │     no change │
│ QQuery 4  │  34.35 ms │                                           35.63 ms │     no change │
│ QQuery 5  │  91.73 ms │                                           91.67 ms │     no change │
│ QQuery 6  │  23.21 ms │                                           20.64 ms │ +1.12x faster │
│ QQuery 7  │ 157.14 ms │                                          162.48 ms │     no change │
│ QQuery 8  │  40.93 ms │                                           42.22 ms │     no change │
│ QQuery 9  │ 111.58 ms │                                          115.04 ms │     no change │
│ QQuery 10 │  72.12 ms │                                           74.28 ms │     no change │
│ QQuery 11 │  18.96 ms │                                           19.11 ms │     no change │
│ QQuery 12 │  56.87 ms │                                           54.87 ms │     no change │
│ QQuery 13 │  55.66 ms │                                           54.46 ms │     no change │
│ QQuery 14 │  15.64 ms │                                           15.74 ms │     no change │
│ QQuery 15 │  34.30 ms │                                           34.93 ms │     no change │
│ QQuery 16 │  30.91 ms │                                           31.02 ms │     no change │
│ QQuery 17 │ 169.58 ms │                                          162.26 ms │     no change │
│ QQuery 18 │ 305.11 ms │                                          297.61 ms │     no change │
│ QQuery 19 │  48.79 ms │                                           53.73 ms │  1.10x slower │
│ QQuery 20 │  62.46 ms │                                           63.48 ms │     no change │
│ QQuery 21 │ 200.87 ms │                                          203.27 ms │     no change │
│ QQuery 22 │  24.72 ms │                                           23.78 ms │     no change │
└───────────┴───────────┴────────────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                                                 ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                                                 │ 1753.10ms │
│ Total Time (parquet-morsel-driven-execution-237164415184908839)   │ 1758.99ms │
│ Average Time (HEAD)                                               │   79.69ms │
│ Average Time (parquet-morsel-driven-execution-237164415184908839) │   79.95ms │
│ Queries Faster                                                    │         1 │
│ Queries Slower                                                    │         1 │
│ Queries with No Change                                            │        20 │
│ Queries with Failure                                              │         0 │
└───────────────────────────────────────────────────────────────────┴───────────┘

Acknowledgements

I heavily used AI (Jules / Claude) for this PR, but reviewed the code myself

This PR implements morsel-driven execution for Parquet files in DataFusion, enabling row-group level work sharing across partitions to mitigate data skew. Key changes: - Introduced `WorkQueue` in `datafusion/datasource/src/file_stream.rs` for shared pool of work. - Added `morselize` method to `FileOpener` trait to allow dynamic splitting of files into morsels. - Implemented `morselize` for `ParquetOpener` to split files into individual row groups. - Cached `ParquetMetaData` in `ParquetMorsel` extensions to avoid redundant I/O. - Modified `FileStream` to support work stealing from the shared queue. - Implemented `Weak` pointer pattern for `WorkQueue` in `FileScanConfig` to support plan re-executability. - Added `MorselizingGuard` to ensure shared state consistency on cancellation. - Added `allow_morsel_driven` configuration option (enabled by default for Parquet). - Implemented row-group pruning during the morselization phase for better efficiency. Tests: - Added `parquet_morsel_driven_execution` test to verify work distribution and re-executability. - Added `parquet_morsel_driven_enabled_by_default` to verify the default configuration. Co-authored-by: Dandandan <163737+Dandandan@users.noreply.github.com>

…en-execution-237164415184908839

Dandandan · 2026-02-23T10:14:24Z

run benchmarks

alamb-ghbot · 2026-03-12T18:12:50Z

🤖: Benchmark completed

Details

Comparing HEAD and parquet-morsel-driven-execution-237164415184908839
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query    ┃        HEAD ┃ parquet-morsel-driven-execution-237164415184908839 ┃        Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │  2268.28 ms │                                         2350.80 ms │     no change │
│ QQuery 1 │   884.50 ms │                                          765.86 ms │ +1.15x faster │
│ QQuery 2 │  1709.92 ms │                                         1530.23 ms │ +1.12x faster │
│ QQuery 3 │  1144.38 ms │                                          942.56 ms │ +1.21x faster │
│ QQuery 4 │  2113.57 ms │                                         2156.01 ms │     no change │
│ QQuery 5 │ 26686.11 ms │                                        27480.63 ms │     no change │
│ QQuery 6 │  3841.79 ms │                                         3683.99 ms │     no change │
│ QQuery 7 │  3020.89 ms │                                         2683.61 ms │ +1.13x faster │
└──────────┴─────────────┴────────────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                                 ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                                 │ 41669.44ms │
│ Total Time (parquet-morsel-driven-execution-237164415184908839)   │ 41593.67ms │
│ Average Time (HEAD)                                               │  5208.68ms │
│ Average Time (parquet-morsel-driven-execution-237164415184908839) │  5199.21ms │
│ Queries Faster                                                    │          4 │
│ Queries Slower                                                    │          0 │
│ Queries with No Change                                            │          4 │
│ Queries with Failure                                              │          0 │
└───────────────────────────────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃        HEAD ┃ parquet-morsel-driven-execution-237164415184908839 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │     2.74 ms │                                            2.62 ms │     no change │
│ QQuery 1  │    48.03 ms │                                           44.25 ms │ +1.09x faster │
│ QQuery 2  │   152.10 ms │                                          139.07 ms │ +1.09x faster │
│ QQuery 3  │   141.67 ms │                                          150.22 ms │  1.06x slower │
│ QQuery 4  │   983.06 ms │                                          998.36 ms │     no change │
│ QQuery 5  │  1233.66 ms │                                         1127.12 ms │ +1.09x faster │
│ QQuery 6  │     6.48 ms │                                            8.24 ms │  1.27x slower │
│ QQuery 7  │    51.14 ms │                                           50.33 ms │     no change │
│ QQuery 8  │  1347.16 ms │                                         1368.09 ms │     no change │
│ QQuery 9  │  1734.86 ms │                                         1534.33 ms │ +1.13x faster │
│ QQuery 10 │   323.52 ms │                                          297.14 ms │ +1.09x faster │
│ QQuery 11 │   377.52 ms │                                          337.25 ms │ +1.12x faster │
│ QQuery 12 │  1141.33 ms │                                         1039.67 ms │ +1.10x faster │
│ QQuery 13 │  1787.86 ms │                                         1532.83 ms │ +1.17x faster │
│ QQuery 14 │  1162.55 ms │                                         1071.69 ms │ +1.08x faster │
│ QQuery 15 │  1163.46 ms │                                         1141.82 ms │     no change │
│ QQuery 16 │  2432.59 ms │                                         2347.30 ms │     no change │
│ QQuery 17 │  2341.62 ms │                                         2339.29 ms │     no change │
│ QQuery 18 │  5732.39 ms │                                         4599.96 ms │ +1.25x faster │
│ QQuery 19 │   117.47 ms │                                          109.01 ms │ +1.08x faster │
│ QQuery 20 │  1865.69 ms │                                         1583.79 ms │ +1.18x faster │
│ QQuery 21 │  2138.32 ms │                                         1913.58 ms │ +1.12x faster │
│ QQuery 22 │  3710.47 ms │                                         3456.27 ms │ +1.07x faster │
│ QQuery 23 │ 21666.53 ms │                                        12330.89 ms │ +1.76x faster │
│ QQuery 24 │   192.45 ms │                                          198.33 ms │     no change │
│ QQuery 25 │   438.52 ms │                                          372.90 ms │ +1.18x faster │
│ QQuery 26 │   191.14 ms │                                          203.33 ms │  1.06x slower │
│ QQuery 27 │  2641.84 ms │                                         2378.14 ms │ +1.11x faster │
│ QQuery 28 │ 23501.26 ms │                                        23533.27 ms │     no change │
│ QQuery 29 │   991.63 ms │                                          923.98 ms │ +1.07x faster │
│ QQuery 30 │  1223.75 ms │                                         1119.97 ms │ +1.09x faster │
│ QQuery 31 │  1299.38 ms │                                         1159.33 ms │ +1.12x faster │
│ QQuery 32 │  4653.73 ms │                                         4482.69 ms │     no change │
│ QQuery 33 │  5319.79 ms │                                         5556.16 ms │     no change │
│ QQuery 34 │  5571.67 ms │                                         5740.76 ms │     no change │
│ QQuery 35 │  1104.05 ms │                                         1060.62 ms │     no change │
│ QQuery 36 │   184.83 ms │                                          115.26 ms │ +1.60x faster │
│ QQuery 37 │    71.80 ms │                                           58.62 ms │ +1.22x faster │
│ QQuery 38 │   115.97 ms │                                           62.01 ms │ +1.87x faster │
│ QQuery 39 │   323.11 ms │                                          220.31 ms │ +1.47x faster │
│ QQuery 40 │    37.69 ms │                                           27.63 ms │ +1.36x faster │
│ QQuery 41 │    33.70 ms │                                           26.25 ms │ +1.28x faster │
│ QQuery 42 │    30.37 ms │                                           23.07 ms │ +1.32x faster │
└───────────┴─────────────┴────────────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                                 ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                                 │ 99588.89ms │
│ Total Time (parquet-morsel-driven-execution-237164415184908839)   │ 86785.73ms │
│ Average Time (HEAD)                                               │  2316.02ms │
│ Average Time (parquet-morsel-driven-execution-237164415184908839) │  2018.27ms │
│ Queries Faster                                                    │         27 │
│ Queries Slower                                                    │          3 │
│ Queries with No Change                                            │         13 │
│ Queries with Failure                                              │          0 │
└───────────────────────────────────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃      HEAD ┃ parquet-morsel-driven-execution-237164415184908839 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │ 102.19 ms │                                          104.58 ms │     no change │
│ QQuery 2  │  32.33 ms │                                           31.84 ms │     no change │
│ QQuery 3  │  38.07 ms │                                           39.72 ms │     no change │
│ QQuery 4  │  30.17 ms │                                           31.15 ms │     no change │
│ QQuery 5  │  81.34 ms │                                           82.43 ms │     no change │
│ QQuery 6  │  20.00 ms │                                           20.12 ms │     no change │
│ QQuery 7  │ 138.43 ms │                                          147.55 ms │  1.07x slower │
│ QQuery 8  │  38.92 ms │                                           40.25 ms │     no change │
│ QQuery 9  │  98.64 ms │                                           97.05 ms │     no change │
│ QQuery 10 │  63.09 ms │                                           64.34 ms │     no change │
│ QQuery 11 │  18.01 ms │                                           18.19 ms │     no change │
│ QQuery 12 │  53.39 ms │                                           52.98 ms │     no change │
│ QQuery 13 │  49.20 ms │                                           47.51 ms │     no change │
│ QQuery 14 │  14.46 ms │                                           14.18 ms │     no change │
│ QQuery 15 │  29.14 ms │                                           29.48 ms │     no change │
│ QQuery 16 │  26.91 ms │                                           27.64 ms │     no change │
│ QQuery 17 │ 143.61 ms │                                          146.75 ms │     no change │
│ QQuery 18 │ 267.64 ms │                                          268.33 ms │     no change │
│ QQuery 19 │  42.07 ms │                                           41.41 ms │     no change │
│ QQuery 20 │  54.49 ms │                                           53.97 ms │     no change │
│ QQuery 21 │ 188.97 ms │                                          191.71 ms │     no change │
│ QQuery 22 │  22.84 ms │                                           21.37 ms │ +1.07x faster │
└───────────┴───────────┴────────────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                                                 ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                                                 │ 1553.88ms │
│ Total Time (parquet-morsel-driven-execution-237164415184908839)   │ 1572.54ms │
│ Average Time (HEAD)                                               │   70.63ms │
│ Average Time (parquet-morsel-driven-execution-237164415184908839) │   71.48ms │
│ Queries Faster                                                    │         1 │
│ Queries Slower                                                    │         1 │
│ Queries with No Change                                            │        20 │
│ Queries with Failure                                              │         0 │
└───────────────────────────────────────────────────────────────────┴───────────┘

This reverts commit 4fc8edd.

Dandandan · 2026-03-13T07:31:54Z

run benchmarks

adriangbot · 2026-03-13T07:31:59Z

Benchmark job started for this request (job bench-c4053325926-193). Results will be posted here when complete.

adriangbot · 2026-03-13T07:32:00Z

Benchmark job started for this request (job bench-c4053325926-194). Results will be posted here when complete.

adriangbot · 2026-03-13T07:32:01Z

Benchmark job started for this request (job bench-c4053325926-195). Results will be posted here when complete.

adriangbot · 2026-03-13T07:34:19Z

🤖 Benchmark running (GKE) | trigger
Linux bench-c4053325926-195-rrpqn 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing parquet-morsel-driven-execution-237164415184908839 (b02a9c0) to ed793f0 (merge-base) diff using: tpch
Results will be posted here when complete

adriangbot · 2026-03-13T07:34:49Z

🤖 Benchmark running (GKE) | trigger
Linux bench-c4053325926-193-m68lq 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing parquet-morsel-driven-execution-237164415184908839 (b02a9c0) to ed793f0 (merge-base) diff using: clickbench_partitioned
Results will be posted here when complete

adriangbot · 2026-03-13T07:34:50Z

🤖 Benchmark running (GKE) | trigger
Linux bench-c4053325926-194-xb2kg 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing parquet-morsel-driven-execution-237164415184908839 (b02a9c0) to ed793f0 (merge-base) diff using: tpcds
Results will be posted here when complete

adriangbot · 2026-03-13T07:47:24Z

🤖 Benchmark completed (GKE) | trigger

Details

Comparing HEAD and parquet-morsel-driven-execution-237164415184908839
--------------------
Benchmark tpch_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                           HEAD ┃ parquet-morsel-driven-execution-237164415184908839 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │ 47.12 / 47.78 ±0.70 / 49.13 ms │                     47.69 / 48.29 ±0.73 / 49.71 ms │     no change │
│ QQuery 2  │ 25.34 / 25.57 ±0.15 / 25.81 ms │                     23.30 / 24.12 ±0.88 / 25.82 ms │ +1.06x faster │
│ QQuery 3  │ 34.37 / 34.64 ±0.24 / 34.95 ms │                     33.66 / 34.00 ±0.26 / 34.36 ms │     no change │
│ QQuery 4  │ 22.42 / 23.54 ±0.85 / 24.60 ms │                     23.04 / 23.24 ±0.16 / 23.46 ms │     no change │
│ QQuery 5  │ 53.42 / 55.11 ±1.00 / 56.15 ms │                     48.77 / 50.43 ±1.36 / 52.15 ms │ +1.09x faster │
│ QQuery 6  │ 18.05 / 18.64 ±0.90 / 20.43 ms │                     19.75 / 20.17 ±0.31 / 20.66 ms │  1.08x slower │
│ QQuery 7  │ 61.12 / 61.77 ±0.49 / 62.63 ms │                     56.09 / 56.37 ±0.19 / 56.66 ms │ +1.10x faster │
│ QQuery 8  │ 52.89 / 53.60 ±0.50 / 54.17 ms │                     51.37 / 51.96 ±0.41 / 52.62 ms │     no change │
│ QQuery 9  │ 57.25 / 59.06 ±1.09 / 60.63 ms │                     59.79 / 60.42 ±0.77 / 61.82 ms │     no change │
│ QQuery 10 │ 75.49 / 76.52 ±0.86 / 77.58 ms │                     50.84 / 51.20 ±0.45 / 52.06 ms │ +1.49x faster │
│ QQuery 11 │ 17.56 / 17.73 ±0.20 / 18.12 ms │                     16.69 / 16.83 ±0.14 / 17.07 ms │ +1.05x faster │
│ QQuery 12 │ 29.30 / 29.63 ±0.34 / 30.24 ms │                     30.41 / 31.42 ±0.52 / 31.90 ms │  1.06x slower │
│ QQuery 13 │ 41.02 / 43.41 ±3.37 / 50.06 ms │                     37.18 / 38.02 ±1.24 / 40.47 ms │ +1.14x faster │
│ QQuery 14 │ 30.06 / 30.18 ±0.12 / 30.34 ms │                     28.92 / 30.35 ±1.43 / 32.17 ms │     no change │
│ QQuery 15 │ 37.27 / 37.81 ±0.31 / 38.17 ms │                     38.95 / 39.16 ±0.24 / 39.58 ms │     no change │
│ QQuery 16 │ 18.33 / 18.68 ±0.24 / 19.08 ms │                     17.04 / 17.64 ±0.42 / 18.33 ms │ +1.06x faster │
│ QQuery 17 │ 77.95 / 79.21 ±1.28 / 81.33 ms │                     85.18 / 86.17 ±1.00 / 88.00 ms │  1.09x slower │
│ QQuery 18 │ 83.71 / 84.86 ±0.84 / 86.16 ms │                     77.55 / 80.64 ±2.51 / 83.57 ms │     no change │
│ QQuery 19 │ 39.07 / 39.91 ±0.65 / 40.88 ms │                     41.79 / 42.35 ±0.63 / 43.39 ms │  1.06x slower │
│ QQuery 20 │ 44.36 / 45.21 ±0.49 / 45.77 ms │                     40.81 / 41.54 ±0.45 / 42.17 ms │ +1.09x faster │
│ QQuery 21 │ 70.72 / 72.66 ±1.96 / 76.16 ms │                     72.65 / 75.05 ±1.84 / 77.45 ms │     no change │
│ QQuery 22 │ 20.98 / 21.23 ±0.23 / 21.63 ms │                     14.17 / 15.38 ±1.17 / 16.82 ms │ +1.38x faster │
└───────────┴────────────────────────────────┴────────────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark Summary                                                 ┃          ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time (HEAD)                                                 │ 976.75ms │
│ Total Time (parquet-morsel-driven-execution-237164415184908839)   │ 934.77ms │
│ Average Time (HEAD)                                               │  44.40ms │
│ Average Time (parquet-morsel-driven-execution-237164415184908839) │  42.49ms │
│ Queries Faster                                                    │        9 │
│ Queries Slower                                                    │        4 │
│ Queries with No Change                                            │        9 │
│ Queries with Failure                                              │        0 │
└───────────────────────────────────────────────────────────────────┴──────────┘

Resource Usage

tpch — base (merge-base)

Metric	Value
Wall time	5.2s
Peak memory	4.1 GiB
Avg memory	3.7 GiB
CPU user	34.9s
CPU sys	3.3s
Disk read	0 B
Disk write	136.0 KiB

tpch — branch

Metric	Value
Wall time	4.9s
Peak memory	4.2 GiB
Avg memory	3.7 GiB
CPU user	35.6s
CPU sys	3.2s
Disk read	0 B
Disk write	56.0 KiB

adriangbot · 2026-03-13T07:51:55Z

🤖 Benchmark completed (GKE) | trigger

Details

Comparing HEAD and parquet-morsel-driven-execution-237164415184908839
--------------------
Benchmark tpcds_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                     HEAD ┃ parquet-morsel-driven-execution-237164415184908839 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1  │           48.95 / 51.06 ±1.09 / 52.09 ms │                     50.22 / 51.36 ±0.81 / 52.34 ms │     no change │
│ QQuery 2  │        152.79 / 154.78 ±1.30 / 156.62 ms │                     59.66 / 61.09 ±0.94 / 62.26 ms │ +2.53x faster │
│ QQuery 3  │        119.02 / 119.42 ±0.29 / 119.86 ms │                     27.61 / 27.79 ±0.15 / 28.02 ms │ +4.30x faster │
│ QQuery 4  │    1359.71 / 1391.54 ±16.45 / 1407.18 ms │                  298.02 / 304.41 ±4.20 / 309.77 ms │ +4.57x faster │
│ QQuery 5  │        186.01 / 187.97 ±1.47 / 189.92 ms │                     75.32 / 76.81 ±0.92 / 77.98 ms │ +2.45x faster │
│ QQuery 6  │     996.39 / 1022.99 ±40.65 / 1103.92 ms │                  863.92 / 871.63 ±7.43 / 881.67 ms │ +1.17x faster │
│ QQuery 7  │        354.10 / 360.61 ±4.15 / 366.01 ms │                     82.82 / 84.42 ±1.65 / 87.60 ms │ +4.27x faster │
│ QQuery 8  │        124.28 / 125.41 ±1.09 / 127.45 ms │                     35.82 / 37.59 ±1.17 / 39.22 ms │ +3.34x faster │
│ QQuery 9  │        113.41 / 121.84 ±8.87 / 139.04 ms │                  106.76 / 107.72 ±0.91 / 109.36 ms │ +1.13x faster │
│ QQuery 10 │        114.63 / 115.66 ±0.96 / 117.20 ms │                     78.40 / 78.92 ±0.38 / 79.43 ms │ +1.47x faster │
│ QQuery 11 │        970.78 / 979.03 ±5.72 / 986.09 ms │                  228.08 / 234.77 ±3.94 / 239.95 ms │ +4.17x faster │
│ QQuery 12 │           47.82 / 49.38 ±1.15 / 50.81 ms │                     26.81 / 28.17 ±1.57 / 31.04 ms │ +1.75x faster │
│ QQuery 13 │        403.07 / 405.90 ±2.91 / 411.35 ms │                  100.29 / 102.12 ±2.24 / 106.32 ms │ +3.97x faster │
│ QQuery 14 │     1101.36 / 1107.12 ±4.46 / 1112.22 ms │                 554.07 / 575.61 ±27.69 / 629.17 ms │ +1.92x faster │
│ QQuery 15 │           18.14 / 18.86 ±0.54 / 19.77 ms │                     19.02 / 19.94 ±0.97 / 21.77 ms │  1.06x slower │
│ QQuery 16 │           46.56 / 47.24 ±0.53 / 47.89 ms │                     46.92 / 48.48 ±0.97 / 49.45 ms │     no change │
│ QQuery 17 │        250.71 / 253.32 ±1.71 / 255.63 ms │                     81.05 / 83.16 ±1.20 / 84.61 ms │ +3.05x faster │
│ QQuery 18 │        133.12 / 134.10 ±0.79 / 134.98 ms │                   99.22 / 100.79 ±0.91 / 101.94 ms │ +1.33x faster │
│ QQuery 19 │        162.27 / 165.21 ±1.83 / 167.33 ms │                     45.88 / 46.41 ±0.45 / 47.11 ms │ +3.56x faster │
│ QQuery 20 │           15.66 / 16.61 ±0.78 / 17.90 ms │                     15.73 / 16.26 ±0.67 / 17.56 ms │     no change │
│ QQuery 21 │           23.38 / 24.62 ±0.64 / 25.11 ms │                     24.80 / 25.36 ±0.47 / 26.23 ms │     no change │
│ QQuery 22 │        518.29 / 524.91 ±5.45 / 532.63 ms │                  140.36 / 142.14 ±1.39 / 143.94 ms │ +3.69x faster │
│ QQuery 23 │        952.24 / 958.78 ±4.90 / 965.35 ms │               1048.38 / 1052.07 ±3.15 / 1056.83 ms │  1.10x slower │
│ QQuery 24 │        439.83 / 440.85 ±0.77 / 441.83 ms │                  205.25 / 208.67 ±4.99 / 218.49 ms │ +2.11x faster │
│ QQuery 25 │        366.17 / 368.92 ±1.62 / 370.77 ms │                 102.34 / 132.89 ±33.36 / 176.28 ms │ +2.78x faster │
│ QQuery 26 │           86.22 / 88.43 ±2.50 / 93.14 ms │                     55.53 / 58.45 ±3.03 / 64.24 ms │ +1.51x faster │
│ QQuery 27 │        351.76 / 355.90 ±2.94 / 360.63 ms │                     80.95 / 82.73 ±1.19 / 83.96 ms │ +4.30x faster │
│ QQuery 28 │        154.87 / 156.88 ±1.68 / 159.77 ms │                  117.96 / 119.80 ±1.47 / 121.89 ms │ +1.31x faster │
│ QQuery 29 │        308.72 / 313.17 ±2.58 / 316.73 ms │                  92.31 / 125.08 ±24.89 / 146.33 ms │ +2.50x faster │
│ QQuery 30 │           49.76 / 51.26 ±1.16 / 52.92 ms │                     46.65 / 49.55 ±1.80 / 51.31 ms │     no change │
│ QQuery 31 │        184.73 / 188.68 ±2.36 / 191.83 ms │                  112.79 / 115.85 ±2.16 / 118.35 ms │ +1.63x faster │
│ QQuery 32 │           62.91 / 64.51 ±1.60 / 67.35 ms │                     25.13 / 26.05 ±0.78 / 27.13 ms │ +2.48x faster │
│ QQuery 33 │        152.20 / 152.93 ±0.51 / 153.63 ms │                     51.26 / 55.55 ±6.08 / 67.59 ms │ +2.75x faster │
│ QQuery 34 │        110.70 / 113.16 ±1.91 / 116.24 ms │                     44.78 / 45.57 ±0.80 / 47.06 ms │ +2.48x faster │
│ QQuery 35 │        117.93 / 118.76 ±0.51 / 119.38 ms │                     92.58 / 93.26 ±0.88 / 94.89 ms │ +1.27x faster │
│ QQuery 36 │        220.02 / 226.25 ±4.62 / 233.07 ms │                     66.33 / 70.64 ±2.93 / 74.50 ms │ +3.20x faster │
│ QQuery 37 │        192.63 / 195.32 ±1.85 / 197.34 ms │                     37.26 / 37.76 ±0.45 / 38.44 ms │ +5.17x faster │
│ QQuery 38 │          92.21 / 98.46 ±3.70 / 103.12 ms │                  114.48 / 116.88 ±1.84 / 119.54 ms │  1.19x slower │
│ QQuery 39 │        139.58 / 142.10 ±1.81 / 144.56 ms │                  139.77 / 142.27 ±1.50 / 143.77 ms │     no change │
│ QQuery 40 │        118.15 / 124.29 ±5.59 / 134.81 ms │                     59.25 / 66.01 ±9.25 / 84.13 ms │ +1.88x faster │
│ QQuery 41 │           18.11 / 19.11 ±0.92 / 20.78 ms │                     20.00 / 20.55 ±0.62 / 21.69 ms │  1.08x slower │
│ QQuery 42 │        112.06 / 113.93 ±1.86 / 117.09 ms │                     26.06 / 26.50 ±0.36 / 26.98 ms │ +4.30x faster │
│ QQuery 43 │           89.60 / 90.88 ±1.64 / 94.04 ms │                     23.73 / 23.97 ±0.18 / 24.18 ms │ +3.79x faster │
│ QQuery 44 │           17.35 / 17.94 ±0.53 / 18.89 ms │                     17.13 / 17.84 ±0.68 / 19.12 ms │     no change │
│ QQuery 45 │           56.63 / 57.37 ±0.46 / 57.85 ms │                     37.43 / 39.08 ±1.28 / 40.91 ms │ +1.47x faster │
│ QQuery 46 │        238.99 / 242.60 ±2.22 / 244.80 ms │                     67.52 / 68.55 ±0.71 / 69.59 ms │ +3.54x faster │
│ QQuery 47 │        743.17 / 749.23 ±4.77 / 757.07 ms │                  143.59 / 144.69 ±0.95 / 146.30 ms │ +5.18x faster │
│ QQuery 48 │        285.66 / 292.60 ±5.56 / 298.50 ms │                     94.00 / 95.81 ±1.86 / 99.36 ms │ +3.05x faster │
│ QQuery 49 │        265.89 / 268.26 ±2.69 / 272.44 ms │                     88.01 / 88.95 ±1.14 / 91.12 ms │ +3.02x faster │
│ QQuery 50 │        246.47 / 250.06 ±3.85 / 257.50 ms │                     89.81 / 95.16 ±3.38 / 99.16 ms │ +2.63x faster │
│ QQuery 51 │        195.45 / 199.05 ±2.99 / 203.31 ms │                     89.57 / 91.22 ±1.14 / 92.94 ms │ +2.18x faster │
│ QQuery 52 │        111.95 / 112.84 ±0.83 / 113.83 ms │                     26.67 / 27.47 ±0.92 / 29.24 ms │ +4.11x faster │
│ QQuery 53 │        108.80 / 110.94 ±1.90 / 113.66 ms │                     24.20 / 25.13 ±1.17 / 27.37 ms │ +4.41x faster │
│ QQuery 54 │        155.81 / 156.76 ±0.59 / 157.61 ms │                     66.04 / 67.15 ±0.98 / 68.90 ms │ +2.33x faster │
│ QQuery 55 │        110.03 / 112.44 ±2.15 / 116.21 ms │                     25.56 / 25.94 ±0.21 / 26.20 ms │ +4.34x faster │
│ QQuery 56 │        151.69 / 153.48 ±1.67 / 156.16 ms │                     52.44 / 52.72 ±0.31 / 53.17 ms │ +2.91x faster │
│ QQuery 57 │        193.08 / 195.65 ±1.77 / 198.28 ms │                  191.89 / 193.63 ±1.71 / 196.69 ms │     no change │
│ QQuery 58 │        306.54 / 314.68 ±5.37 / 321.03 ms │                  122.70 / 132.89 ±8.33 / 142.90 ms │ +2.37x faster │
│ QQuery 59 │        207.74 / 210.02 ±1.97 / 213.64 ms │                     60.48 / 61.48 ±0.88 / 63.11 ms │ +3.42x faster │
│ QQuery 60 │        153.36 / 154.77 ±1.10 / 156.22 ms │                     55.06 / 55.35 ±0.36 / 56.04 ms │ +2.80x faster │
│ QQuery 61 │        181.75 / 184.22 ±1.44 / 186.24 ms │                   63.81 / 84.81 ±40.68 / 166.17 ms │ +2.17x faster │
│ QQuery 62 │       906.62 / 935.31 ±28.14 / 987.85 ms │                 873.44 / 913.07 ±45.85 / 998.58 ms │     no change │
│ QQuery 63 │        110.11 / 112.07 ±1.56 / 113.83 ms │                     24.91 / 27.34 ±1.91 / 30.24 ms │ +4.10x faster │
│ QQuery 64 │        727.95 / 730.50 ±2.79 / 735.68 ms │                404.46 / 477.05 ±123.84 / 723.70 ms │ +1.53x faster │
│ QQuery 65 │        260.82 / 264.03 ±2.94 / 269.48 ms │                     50.80 / 52.80 ±1.24 / 54.46 ms │ +5.00x faster │
│ QQuery 66 │        255.45 / 261.09 ±3.25 / 264.01 ms │                  170.69 / 176.01 ±3.56 / 181.23 ms │ +1.48x faster │
│ QQuery 67 │        342.81 / 345.17 ±3.14 / 351.14 ms │                  414.61 / 422.32 ±6.99 / 433.76 ms │  1.22x slower │
│ QQuery 68 │        285.38 / 288.57 ±2.63 / 292.88 ms │                     61.85 / 64.51 ±3.18 / 69.87 ms │ +4.47x faster │
│ QQuery 69 │        113.75 / 114.77 ±0.80 / 115.71 ms │                     73.11 / 74.44 ±0.86 / 75.42 ms │ +1.54x faster │
│ QQuery 70 │        345.76 / 361.41 ±8.30 / 369.87 ms │                  102.86 / 111.77 ±9.93 / 128.57 ms │ +3.23x faster │
│ QQuery 71 │        139.60 / 143.67 ±2.12 / 145.40 ms │                     37.70 / 38.63 ±0.61 / 39.51 ms │ +3.72x faster │
│ QQuery 72 │       731.84 / 749.79 ±10.51 / 761.28 ms │                 477.03 / 493.81 ±12.95 / 515.88 ms │ +1.52x faster │
│ QQuery 73 │        108.32 / 110.28 ±1.50 / 112.34 ms │                     43.20 / 45.37 ±1.69 / 47.53 ms │ +2.43x faster │
│ QQuery 74 │        601.53 / 604.76 ±1.74 / 606.54 ms │                  138.84 / 141.48 ±1.69 / 144.08 ms │ +4.27x faster │
│ QQuery 75 │        295.58 / 297.49 ±1.37 / 299.57 ms │                  165.95 / 172.81 ±3.87 / 177.41 ms │ +1.72x faster │
│ QQuery 76 │        138.03 / 138.73 ±0.41 / 139.20 ms │                     40.78 / 41.86 ±0.87 / 43.23 ms │ +3.31x faster │
│ QQuery 77 │        204.15 / 205.14 ±0.57 / 205.70 ms │                     66.65 / 67.90 ±0.76 / 68.80 ms │ +3.02x faster │
│ QQuery 78 │        363.51 / 366.75 ±2.47 / 371.16 ms │                  133.54 / 138.74 ±6.29 / 150.57 ms │ +2.64x faster │
│ QQuery 79 │        239.44 / 242.58 ±2.51 / 245.80 ms │                     66.05 / 67.27 ±0.90 / 68.57 ms │ +3.61x faster │
│ QQuery 80 │        342.28 / 344.26 ±1.70 / 347.14 ms │                  106.50 / 111.20 ±3.50 / 116.90 ms │ +3.10x faster │
│ QQuery 81 │           31.52 / 33.14 ±1.40 / 35.30 ms │                     37.10 / 38.59 ±0.92 / 39.48 ms │  1.16x slower │
│ QQuery 82 │        205.71 / 208.11 ±1.66 / 210.19 ms │                     36.48 / 37.26 ±0.42 / 37.67 ms │ +5.59x faster │
│ QQuery 83 │           47.26 / 48.90 ±1.74 / 52.20 ms │                     49.62 / 50.66 ±0.96 / 51.94 ms │     no change │
│ QQuery 84 │           53.16 / 53.85 ±0.40 / 54.29 ms │                     45.45 / 46.85 ±0.91 / 48.26 ms │ +1.15x faster │
│ QQuery 85 │        153.21 / 156.43 ±2.14 / 159.81 ms │                  101.83 / 103.16 ±0.87 / 104.32 ms │ +1.52x faster │
│ QQuery 86 │           41.94 / 42.71 ±1.09 / 44.87 ms │                     31.07 / 32.01 ±0.66 / 33.00 ms │ +1.33x faster │
│ QQuery 87 │          93.18 / 96.98 ±4.47 / 104.79 ms │                  112.85 / 116.55 ±1.94 / 118.60 ms │  1.20x slower │
│ QQuery 88 │        114.49 / 115.78 ±1.13 / 117.68 ms │                 108.00 / 118.63 ±11.66 / 140.38 ms │     no change │
│ QQuery 89 │        125.34 / 127.14 ±1.34 / 129.25 ms │                   26.63 / 47.28 ±40.14 / 127.56 ms │ +2.69x faster │
│ QQuery 90 │           29.16 / 29.36 ±0.17 / 29.55 ms │                     29.79 / 30.56 ±0.58 / 31.37 ms │     no change │
│ QQuery 91 │           66.71 / 68.62 ±1.68 / 71.43 ms │                     60.10 / 63.47 ±1.85 / 65.27 ms │ +1.08x faster │
│ QQuery 92 │           63.12 / 63.82 ±0.60 / 64.64 ms │                     31.90 / 32.46 ±0.53 / 33.44 ms │ +1.97x faster │
│ QQuery 93 │        199.98 / 204.00 ±2.36 / 206.65 ms │                   65.13 / 92.44 ±53.74 / 199.92 ms │ +2.21x faster │
│ QQuery 94 │           66.77 / 68.07 ±1.27 / 70.43 ms │                     34.20 / 34.95 ±0.51 / 35.52 ms │ +1.95x faster │
│ QQuery 95 │        141.97 / 146.28 ±3.09 / 149.70 ms │                  112.15 / 113.50 ±1.38 / 116.01 ms │ +1.29x faster │
│ QQuery 96 │           78.57 / 79.89 ±1.09 / 81.06 ms │                    49.86 / 56.78 ±12.08 / 80.90 ms │ +1.41x faster │
│ QQuery 97 │        134.65 / 137.69 ±2.38 / 141.67 ms │                     59.56 / 61.01 ±1.54 / 63.70 ms │ +2.26x faster │
│ QQuery 98 │        153.27 / 157.73 ±3.76 / 164.19 ms │                     37.48 / 38.26 ±0.47 / 38.72 ms │ +4.12x faster │
│ QQuery 99 │ 10781.30 / 10843.94 ±49.67 / 10934.18 ms │           10776.19 / 10837.85 ±40.24 / 10891.31 ms │     no change │
└───────────┴──────────────────────────────────────────┴────────────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                                 ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                                 │ 34841.89ms │
│ Total Time (parquet-morsel-driven-execution-237164415184908839)   │ 22753.56ms │
│ Average Time (HEAD)                                               │   351.94ms │
│ Average Time (parquet-morsel-driven-execution-237164415184908839) │   229.83ms │
│ Queries Faster                                                    │         79 │
│ Queries Slower                                                    │          7 │
│ Queries with No Change                                            │         13 │
│ Queries with Failure                                              │          0 │
└───────────────────────────────────────────────────────────────────┴────────────┘

Resource Usage

tpcds — base (merge-base)

Metric	Value
Wall time	174.5s
Peak memory	5.1 GiB
Avg memory	4.5 GiB
CPU user	276.4s
CPU sys	21.4s
Disk read	0 B
Disk write	637.1 MiB

tpcds — branch

Metric	Value
Wall time	114.1s
Peak memory	5.5 GiB
Avg memory	4.9 GiB
CPU user	290.5s
CPU sys	20.9s
Disk read	0 B
Disk write	772.0 KiB

adriangbot · 2026-03-13T07:52:55Z

🤖 Benchmark completed (GKE) | trigger

Details

Comparing HEAD and parquet-morsel-driven-execution-237164415184908839
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                  HEAD ┃ parquet-morsel-driven-execution-237164415184908839 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │          1.60 / 4.79 ±6.31 / 17.42 ms │                       1.59 / 4.96 ±6.54 / 18.04 ms │     no change │
│ QQuery 1  │        15.58 / 15.72 ±0.13 / 15.97 ms │                     13.80 / 14.35 ±0.30 / 14.66 ms │ +1.10x faster │
│ QQuery 2  │        57.02 / 57.49 ±0.32 / 57.97 ms │                     48.11 / 48.95 ±0.53 / 49.68 ms │ +1.17x faster │
│ QQuery 3  │        46.22 / 49.59 ±2.18 / 51.64 ms │                     42.99 / 43.40 ±0.51 / 44.39 ms │ +1.14x faster │
│ QQuery 4  │     304.05 / 311.33 ±6.00 / 318.79 ms │                  267.96 / 277.26 ±6.18 / 283.74 ms │ +1.12x faster │
│ QQuery 5  │     351.60 / 354.43 ±2.47 / 357.55 ms │                  312.34 / 320.89 ±5.04 / 327.30 ms │ +1.10x faster │
│ QQuery 6  │           5.50 / 5.88 ±0.43 / 6.49 ms │                        5.87 / 7.53 ±0.99 / 8.71 ms │  1.28x slower │
│ QQuery 7  │        18.36 / 18.78 ±0.43 / 19.59 ms │                     15.41 / 16.13 ±0.66 / 17.10 ms │ +1.16x faster │
│ QQuery 8  │     441.67 / 454.28 ±8.80 / 468.69 ms │                  386.94 / 396.54 ±8.07 / 411.41 ms │ +1.15x faster │
│ QQuery 9  │    685.87 / 697.85 ±13.90 / 724.49 ms │                 511.32 / 531.43 ±12.34 / 545.40 ms │ +1.31x faster │
│ QQuery 10 │        92.11 / 96.05 ±2.26 / 98.91 ms │                     78.73 / 81.31 ±2.63 / 85.63 ms │ +1.18x faster │
│ QQuery 11 │     107.07 / 107.50 ±0.39 / 108.08 ms │                     90.53 / 91.39 ±0.52 / 92.06 ms │ +1.18x faster │
│ QQuery 12 │     346.89 / 352.84 ±5.70 / 363.50 ms │                  301.29 / 309.18 ±8.49 / 324.55 ms │ +1.14x faster │
│ QQuery 13 │     463.56 / 470.01 ±7.95 / 485.21 ms │                  431.40 / 439.22 ±5.68 / 445.77 ms │ +1.07x faster │
│ QQuery 14 │     361.02 / 366.96 ±4.68 / 374.53 ms │                  322.25 / 327.02 ±5.46 / 337.63 ms │ +1.12x faster │
│ QQuery 15 │     369.36 / 377.38 ±4.33 / 381.40 ms │                 339.66 / 353.99 ±20.10 / 392.71 ms │ +1.07x faster │
│ QQuery 16 │     745.71 / 760.84 ±8.37 / 771.30 ms │                  707.71 / 715.76 ±6.14 / 721.87 ms │ +1.06x faster │
│ QQuery 17 │    741.69 / 759.43 ±12.58 / 772.51 ms │                  697.33 / 698.73 ±1.63 / 701.24 ms │ +1.09x faster │
│ QQuery 18 │ 1502.92 / 1550.67 ±42.50 / 1621.19 ms │              1307.30 / 1364.18 ±55.51 / 1456.62 ms │ +1.14x faster │
│ QQuery 19 │        35.62 / 37.33 ±1.04 / 38.61 ms │                     30.81 / 32.16 ±1.09 / 33.55 ms │ +1.16x faster │
│ QQuery 20 │    708.56 / 726.80 ±17.36 / 749.24 ms │                  498.60 / 502.60 ±2.98 / 506.72 ms │ +1.45x faster │
│ QQuery 21 │     758.22 / 760.34 ±1.21 / 761.91 ms │                  567.11 / 579.61 ±9.54 / 596.29 ms │ +1.31x faster │
│ QQuery 22 │  1117.70 / 1127.11 ±7.63 / 1139.69 ms │                 988.53 / 995.99 ±6.26 / 1005.01 ms │ +1.13x faster │
│ QQuery 23 │ 3165.83 / 3198.37 ±22.49 / 3234.11 ms │              3146.40 / 3164.21 ±14.84 / 3186.88 ms │     no change │
│ QQuery 24 │     102.70 / 104.89 ±2.38 / 108.98 ms │                     73.23 / 77.37 ±2.17 / 79.42 ms │ +1.36x faster │
│ QQuery 25 │     140.58 / 142.34 ±1.61 / 144.92 ms │                  110.90 / 113.03 ±2.17 / 117.08 ms │ +1.26x faster │
│ QQuery 26 │     102.73 / 104.11 ±1.51 / 106.77 ms │                     72.31 / 75.29 ±2.56 / 79.07 ms │ +1.38x faster │
│ QQuery 27 │     847.45 / 850.73 ±3.13 / 856.60 ms │                  642.85 / 654.23 ±7.23 / 663.26 ms │ +1.30x faster │
│ QQuery 28 │ 7666.79 / 7733.69 ±37.60 / 7771.52 ms │              6529.89 / 6667.46 ±97.52 / 6782.23 ms │ +1.16x faster │
│ QQuery 29 │     332.25 / 335.98 ±6.73 / 349.42 ms │                  282.20 / 286.82 ±7.03 / 300.64 ms │ +1.17x faster │
│ QQuery 30 │     367.11 / 373.26 ±5.89 / 383.41 ms │                  343.84 / 347.32 ±3.81 / 354.18 ms │ +1.07x faster │
│ QQuery 31 │     368.87 / 378.96 ±5.25 / 383.98 ms │                 329.78 / 340.91 ±11.12 / 360.45 ms │ +1.11x faster │
│ QQuery 32 │ 1064.67 / 1105.27 ±36.23 / 1154.45 ms │              1039.84 / 1053.75 ±18.89 / 1091.05 ms │     no change │
│ QQuery 33 │ 1491.10 / 1503.62 ±13.33 / 1526.82 ms │              1485.64 / 1502.66 ±17.58 / 1533.77 ms │     no change │
│ QQuery 34 │  1497.51 / 1503.31 ±4.87 / 1511.11 ms │              1483.50 / 1521.32 ±22.51 / 1549.62 ms │     no change │
│ QQuery 35 │     402.78 / 409.49 ±6.60 / 422.14 ms │                  335.17 / 338.72 ±5.04 / 348.68 ms │ +1.21x faster │
│ QQuery 36 │     117.99 / 125.92 ±6.39 / 134.55 ms │                     68.38 / 72.12 ±3.24 / 76.52 ms │ +1.75x faster │
│ QQuery 37 │        49.93 / 52.24 ±1.40 / 53.94 ms │                     38.15 / 40.15 ±1.17 / 41.50 ms │ +1.30x faster │
│ QQuery 38 │        75.92 / 78.51 ±2.09 / 82.17 ms │                     39.92 / 41.80 ±1.48 / 44.03 ms │ +1.88x faster │
│ QQuery 39 │     219.12 / 228.41 ±8.06 / 238.55 ms │                  135.33 / 139.76 ±2.85 / 143.87 ms │ +1.63x faster │
│ QQuery 40 │        25.94 / 28.24 ±1.34 / 29.67 ms │                     15.07 / 17.32 ±2.43 / 21.91 ms │ +1.63x faster │
│ QQuery 41 │        20.98 / 21.80 ±0.85 / 23.26 ms │                     15.48 / 16.73 ±0.99 / 18.17 ms │ +1.30x faster │
│ QQuery 42 │        20.50 / 21.21 ±0.48 / 21.98 ms │                     14.54 / 15.03 ±0.28 / 15.40 ms │ +1.41x faster │
└───────────┴───────────────────────────────────────┴────────────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                                 ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                                 │ 27763.74ms │
│ Total Time (parquet-morsel-driven-execution-237164415184908839)   │ 24638.61ms │
│ Average Time (HEAD)                                               │   645.67ms │
│ Average Time (parquet-morsel-driven-execution-237164415184908839) │   572.99ms │
│ Queries Faster                                                    │         37 │
│ Queries Slower                                                    │          1 │
│ Queries with No Change                                            │          5 │
│ Queries with Failure                                              │          0 │
└───────────────────────────────────────────────────────────────────┴────────────┘

Resource Usage

clickbench_partitioned — base (merge-base)

Metric	Value
Wall time	140.0s
Peak memory	40.0 GiB
Avg memory	29.9 GiB
CPU user	1315.3s
CPU sys	89.5s
Disk read	0 B
Disk write	4.2 GiB

clickbench_partitioned — branch

Metric	Value
Wall time	124.3s
Peak memory	41.5 GiB
Avg memory	33.5 GiB
CPU user	1331.5s
CPU sys	84.3s
Disk read	0 B
Disk write	768.0 KiB

Dandandan · 2026-03-13T08:00:05Z

run benchmark clickbench_extended

adriangbot · 2026-03-13T08:00:09Z

Benchmark job started for this request (job bench-c4053431901-196). Results will be posted here when complete.

adriangbot · 2026-03-13T08:02:30Z

🤖 Benchmark running (GKE) | trigger
Linux bench-c4053431901-196-bq7lc 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing parquet-morsel-driven-execution-237164415184908839 (b02a9c0) to ed793f0 (merge-base) diff using: clickbench_extended
Results will be posted here when complete

adriangbot · 2026-03-13T08:18:34Z

🤖 Benchmark completed (GKE) | trigger

Details

Comparing HEAD and parquet-morsel-driven-execution-237164415184908839
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query    ┃                                      HEAD ┃ parquet-morsel-driven-execution-237164415184908839 ┃        Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │         819.27 / 824.23 ±5.26 / 833.83 ms │                 952.14 / 968.25 ±13.21 / 988.25 ms │  1.17x slower │
│ QQuery 1 │         225.08 / 225.74 ±0.73 / 226.68 ms │                  179.77 / 180.84 ±0.95 / 182.16 ms │ +1.25x faster │
│ QQuery 2 │         528.63 / 532.69 ±4.03 / 537.71 ms │                  416.88 / 420.78 ±2.31 / 423.43 ms │ +1.27x faster │
│ QQuery 3 │         333.08 / 334.27 ±0.94 / 335.46 ms │                  255.37 / 256.11 ±0.41 / 256.62 ms │ +1.31x faster │
│ QQuery 4 │        665.59 / 686.14 ±14.86 / 711.04 ms │                  640.72 / 648.91 ±4.71 / 655.41 ms │ +1.06x faster │
│ QQuery 5 │ 10046.59 / 10200.59 ±143.64 / 10421.45 ms │             9116.94 / 9237.59 ±153.88 / 9503.35 ms │ +1.10x faster │
│ QQuery 6 │       983.76 / 995.88 ±17.09 / 1029.36 ms │                  918.40 / 925.52 ±5.17 / 933.42 ms │ +1.08x faster │
│ QQuery 7 │        778.00 / 795.61 ±11.31 / 811.88 ms │                  769.46 / 776.09 ±4.89 / 781.16 ms │     no change │
└──────────┴───────────────────────────────────────────┴────────────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                                 ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                                 │ 14595.14ms │
│ Total Time (parquet-morsel-driven-execution-237164415184908839)   │ 13414.08ms │
│ Average Time (HEAD)                                               │  1824.39ms │
│ Average Time (parquet-morsel-driven-execution-237164415184908839) │  1676.76ms │
│ Queries Faster                                                    │          6 │
│ Queries Slower                                                    │          1 │
│ Queries with No Change                                            │          1 │
│ Queries with Failure                                              │          0 │
└───────────────────────────────────────────────────────────────────┴────────────┘

Resource Usage

clickbench_extended — base (merge-base)

Metric	Value
Wall time	73.6s
Peak memory	33.2 GiB
Avg memory	28.6 GiB
CPU user	723.9s
CPU sys	23.7s
Disk read	0 B
Disk write	21.8 MiB

clickbench_extended — branch

Metric	Value
Wall time	67.6s
Peak memory	33.8 GiB
Avg memory	28.9 GiB
CPU user	728.2s
CPU sys	22.9s
Disk read	0 B
Disk write	72.0 KiB

Dandandan · 2026-03-16T16:29:24Z

run benchmark clickbench_pushdown

adriangbot · 2026-03-16T16:32:20Z

🤖 Benchmark running (GKE) | trigger
Linux bench-c4068979599-333-npw74 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing parquet-morsel-driven-execution-237164415184908839 (b02a9c0) to ed793f0 (merge-base) diff using: clickbench_pushdown
Results will be posted here when complete

…en-execution-237164415184908839 # Conflicts: # datafusion/datasource-parquet/src/opener.rs # datafusion/sqllogictest/test_files/clickbench.slt

…am/main - Add missing import for reverse_row_selection in opener.rs - Update clickbench.slt plan expectations for ProjectionExec optimization changes - Update projection_pushdown.slt for predicate pushdown in DataSourceExec - Update encrypted_parquet.slt for query that now succeeds Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

adriangbot · 2026-03-16T16:48:23Z

🤖 Benchmark completed (GKE) | trigger

Details

Comparing HEAD and parquet-morsel-driven-execution-237164415184908839
--------------------
Benchmark clickbench_pushdown.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                  HEAD ┃ parquet-morsel-driven-execution-237164415184908839 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │          1.58 / 4.78 ±6.32 / 17.42 ms │                       1.62 / 4.84 ±6.36 / 17.55 ms │     no change │
│ QQuery 1  │        16.23 / 16.65 ±0.26 / 17.04 ms │                     14.62 / 15.09 ±0.39 / 15.78 ms │ +1.10x faster │
│ QQuery 2  │        58.07 / 59.30 ±1.22 / 61.58 ms │                     48.57 / 49.48 ±0.72 / 50.45 ms │ +1.20x faster │
│ QQuery 3  │        51.53 / 52.65 ±0.67 / 53.60 ms │                     42.42 / 44.32 ±1.66 / 47.10 ms │ +1.19x faster │
│ QQuery 4  │     306.49 / 310.55 ±2.75 / 315.05 ms │                  273.89 / 283.04 ±5.45 / 290.21 ms │ +1.10x faster │
│ QQuery 5  │     350.15 / 354.36 ±3.68 / 360.13 ms │                  318.56 / 325.01 ±4.35 / 332.08 ms │ +1.09x faster │
│ QQuery 6  │        11.41 / 11.87 ±0.45 / 12.54 ms │                      9.69 / 10.77 ±1.05 / 12.11 ms │ +1.10x faster │
│ QQuery 7  │        22.69 / 23.57 ±0.92 / 25.19 ms │                     19.77 / 20.22 ±0.34 / 20.82 ms │ +1.17x faster │
│ QQuery 8  │     434.81 / 440.78 ±4.10 / 445.21 ms │                 400.63 / 419.12 ±12.02 / 434.14 ms │     no change │
│ QQuery 9  │     667.83 / 679.13 ±6.51 / 685.78 ms │                  532.15 / 535.49 ±3.18 / 540.96 ms │ +1.27x faster │
│ QQuery 10 │     121.78 / 125.47 ±3.52 / 131.67 ms │                  102.99 / 106.41 ±4.33 / 114.31 ms │ +1.18x faster │
│ QQuery 11 │     135.72 / 136.35 ±0.60 / 137.48 ms │                  115.53 / 115.71 ±0.20 / 116.06 ms │ +1.18x faster │
│ QQuery 12 │     381.12 / 385.44 ±2.45 / 388.36 ms │                  345.06 / 346.97 ±1.33 / 348.62 ms │ +1.11x faster │
│ QQuery 13 │    496.26 / 511.63 ±18.20 / 545.60 ms │                 455.11 / 489.49 ±27.72 / 530.05 ms │     no change │
│ QQuery 14 │     394.37 / 397.51 ±1.66 / 399.08 ms │                  358.58 / 360.59 ±2.47 / 365.29 ms │ +1.10x faster │
│ QQuery 15 │    375.67 / 389.13 ±11.29 / 408.18 ms │                 338.54 / 353.16 ±15.73 / 383.08 ms │ +1.10x faster │
│ QQuery 16 │    749.54 / 762.51 ±16.00 / 793.46 ms │                  705.17 / 715.04 ±7.01 / 723.97 ms │ +1.07x faster │
│ QQuery 17 │    747.38 / 799.05 ±31.29 / 845.65 ms │                  701.42 / 707.28 ±4.82 / 715.36 ms │ +1.13x faster │
│ QQuery 18 │ 1478.83 / 1520.54 ±23.82 / 1550.83 ms │              1333.65 / 1450.60 ±59.56 / 1501.08 ms │     no change │
│ QQuery 19 │        37.61 / 39.24 ±1.12 / 40.57 ms │                    32.86 / 44.28 ±19.97 / 84.12 ms │  1.13x slower │
│ QQuery 20 │    706.92 / 723.81 ±16.43 / 745.82 ms │                 507.37 / 574.89 ±53.85 / 633.69 ms │ +1.26x faster │
│ QQuery 21 │     738.40 / 746.53 ±6.61 / 756.85 ms │                 586.31 / 611.26 ±18.18 / 638.22 ms │ +1.22x faster │
│ QQuery 22 │ 1180.00 / 1198.03 ±14.75 / 1216.08 ms │                971.77 / 990.53 ±17.11 / 1015.44 ms │ +1.21x faster │
│ QQuery 23 │    389.40 / 402.92 ±18.43 / 439.44 ms │                 252.89 / 294.66 ±48.45 / 388.95 ms │ +1.37x faster │
│ QQuery 24 │     116.50 / 118.25 ±1.65 / 120.95 ms │                     69.34 / 74.84 ±4.49 / 82.87 ms │ +1.58x faster │
│ QQuery 25 │     190.50 / 195.19 ±3.10 / 200.17 ms │                  156.92 / 159.83 ±2.44 / 163.97 ms │ +1.22x faster │
│ QQuery 26 │     155.50 / 161.37 ±4.52 / 167.58 ms │                  106.76 / 112.15 ±3.32 / 115.51 ms │ +1.44x faster │
│ QQuery 27 │   959.52 / 983.36 ±19.90 / 1019.90 ms │                 708.53 / 732.54 ±28.77 / 789.08 ms │ +1.34x faster │
│ QQuery 28 │ 7836.33 / 7912.82 ±47.38 / 7981.55 ms │             6689.84 / 6822.39 ±103.20 / 6978.24 ms │ +1.16x faster │
│ QQuery 29 │     331.71 / 336.14 ±6.67 / 349.38 ms │                 282.95 / 295.60 ±20.05 / 335.39 ms │ +1.14x faster │
│ QQuery 30 │     375.31 / 385.95 ±5.85 / 392.39 ms │                  342.51 / 350.79 ±7.75 / 364.77 ms │ +1.10x faster │
│ QQuery 31 │    369.28 / 386.85 ±16.44 / 414.10 ms │                  345.57 / 352.50 ±5.61 / 361.41 ms │ +1.10x faster │
│ QQuery 32 │ 1329.16 / 1366.43 ±43.61 / 1437.44 ms │              1047.60 / 1105.11 ±53.95 / 1184.53 ms │ +1.24x faster │
│ QQuery 33 │ 1610.71 / 1669.07 ±48.35 / 1742.99 ms │              1536.41 / 1553.61 ±16.93 / 1580.56 ms │ +1.07x faster │
│ QQuery 34 │ 1510.32 / 1569.52 ±36.83 / 1623.42 ms │              1555.47 / 1565.59 ±11.47 / 1585.47 ms │     no change │
│ QQuery 35 │     409.06 / 415.43 ±4.39 / 421.28 ms │                  342.24 / 346.76 ±6.65 / 359.96 ms │ +1.20x faster │
│ QQuery 36 │     102.17 / 111.12 ±6.62 / 119.99 ms │                     76.74 / 80.72 ±2.79 / 83.42 ms │ +1.38x faster │
│ QQuery 37 │        56.23 / 59.89 ±3.13 / 64.45 ms │                     44.50 / 48.87 ±3.02 / 53.46 ms │ +1.23x faster │
│ QQuery 38 │        54.39 / 58.83 ±3.37 / 62.07 ms │                     40.36 / 42.92 ±2.21 / 45.77 ms │ +1.37x faster │
│ QQuery 39 │     182.72 / 190.42 ±5.19 / 198.59 ms │                  121.77 / 129.32 ±6.70 / 141.68 ms │ +1.47x faster │
│ QQuery 40 │        34.33 / 36.94 ±2.20 / 40.86 ms │                     24.77 / 27.18 ±1.57 / 29.32 ms │ +1.36x faster │
│ QQuery 41 │        32.03 / 32.80 ±0.96 / 34.43 ms │                     22.05 / 24.20 ±1.90 / 27.30 ms │ +1.36x faster │
│ QQuery 42 │        24.75 / 25.23 ±0.30 / 25.63 ms │                     18.26 / 18.58 ±0.28 / 19.05 ms │ +1.36x faster │
└───────────┴───────────────────────────────────────┴────────────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                                                 ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                                                 │ 26107.40ms │
│ Total Time (parquet-morsel-driven-execution-237164415184908839)   │ 22711.75ms │
│ Average Time (HEAD)                                               │   607.15ms │
│ Average Time (parquet-morsel-driven-execution-237164415184908839) │   528.18ms │
│ Queries Faster                                                    │         37 │
│ Queries Slower                                                    │          1 │
│ Queries with No Change                                            │          5 │
│ Queries with Failure                                              │          0 │
└───────────────────────────────────────────────────────────────────┴────────────┘

Resource Usage

clickbench_pushdown — base (merge-base)

Metric	Value
Wall time	131.8s
Peak memory	41.4 GiB
Avg memory	29.9 GiB
CPU user	1180.9s
CPU sys	117.8s
Disk read	0 B
Disk write	2.0 GiB

clickbench_pushdown — branch

Metric	Value
Wall time	114.8s
Peak memory	43.2 GiB
Avg memory	31.5 GiB
CPU user	1185.5s
CPU sys	101.9s
Disk read	0 B
Disk write	116.0 KiB

Dandandan · 2026-03-16T16:51:04Z

run benchmark clickbench_pushdown

env:
   SIMULATE_LATENCY: true

adriangbot · 2026-03-16T16:54:02Z

🤖 Benchmark running (GKE) | trigger
Linux bench-c4069123125-335-ndnlq 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux
Comparing parquet-morsel-driven-execution-237164415184908839 (d987e42) to c6f7145 (merge-base) diff using: clickbench_pushdown
Results will be posted here when complete

adriangbot · 2026-03-16T17:31:33Z

🤖 Benchmark completed (GKE) | trigger

Details

Comparing HEAD and parquet-morsel-driven-execution-237164415184908839
--------------------
Benchmark clickbench_pushdown.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                      HEAD ┃ parquet-morsel-driven-execution-237164415184908839 ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │              1.85 / 5.20 ±6.55 / 18.29 ms │                       1.78 / 5.26 ±6.65 / 18.56 ms │     no change │
│ QQuery 1  │    2051.28 / 2227.71 ±177.13 / 2488.58 ms │              1822.77 / 1889.26 ±43.42 / 1944.11 ms │ +1.18x faster │
│ QQuery 2  │    2640.98 / 2939.94 ±233.78 / 3338.32 ms │              2112.42 / 2179.45 ±56.71 / 2252.02 ms │ +1.35x faster │
│ QQuery 3  │    2553.63 / 2760.70 ±181.32 / 2982.74 ms │              2135.32 / 2181.63 ±64.71 / 2308.45 ms │ +1.27x faster │
│ QQuery 4  │     2836.68 / 2963.97 ±85.52 / 3067.33 ms │              2284.40 / 2333.57 ±54.99 / 2440.11 ms │ +1.27x faster │
│ QQuery 5  │    2645.40 / 2915.30 ±176.31 / 3151.48 ms │              2372.43 / 2434.07 ±73.40 / 2566.37 ms │ +1.20x faster │
│ QQuery 6  │       634.01 / 749.82 ±110.53 / 937.89 ms │                 474.25 / 507.62 ±25.09 / 546.09 ms │ +1.48x faster │
│ QQuery 7  │    2226.45 / 2342.47 ±133.03 / 2538.35 ms │              1851.73 / 1958.87 ±83.32 / 2078.22 ms │ +1.20x faster │
│ QQuery 8  │    2822.39 / 3060.41 ±172.76 / 3263.01 ms │             2414.89 / 2522.32 ±120.42 / 2728.31 ms │ +1.21x faster │
│ QQuery 9  │    3051.46 / 3375.14 ±192.05 / 3579.74 ms │              2643.39 / 2697.18 ±36.22 / 2756.20 ms │ +1.25x faster │
│ QQuery 10 │    5175.38 / 5486.33 ±297.08 / 5967.08 ms │             4291.72 / 4451.46 ±148.62 / 4666.86 ms │ +1.23x faster │
│ QQuery 11 │    5022.21 / 5236.48 ±172.50 / 5428.36 ms │             4246.05 / 4465.36 ±183.80 / 4765.16 ms │ +1.17x faster │
│ QQuery 12 │    2726.59 / 2964.91 ±152.01 / 3195.75 ms │              2415.08 / 2492.50 ±85.26 / 2639.72 ms │ +1.19x faster │
│ QQuery 13 │     5602.56 / 5732.70 ±90.60 / 5836.79 ms │             4619.87 / 4766.15 ±107.41 / 4950.14 ms │ +1.20x faster │
│ QQuery 14 │    5239.97 / 5543.59 ±152.54 / 5646.53 ms │             4401.33 / 4558.88 ±114.18 / 4740.10 ms │ +1.22x faster │
│ QQuery 15 │    2903.57 / 3075.31 ±156.39 / 3290.06 ms │              2336.51 / 2415.51 ±77.52 / 2558.60 ms │ +1.27x faster │
│ QQuery 16 │    3108.36 / 3230.98 ±146.02 / 3508.16 ms │              2774.88 / 2818.70 ±27.66 / 2860.69 ms │ +1.15x faster │
│ QQuery 17 │    2996.86 / 3249.99 ±145.60 / 3414.53 ms │              2809.10 / 2828.96 ±22.49 / 2872.18 ms │ +1.15x faster │
│ QQuery 18 │    3455.81 / 3670.66 ±132.61 / 3845.83 ms │              3014.70 / 3140.77 ±66.49 / 3192.49 ms │ +1.17x faster │
│ QQuery 19 │    2178.64 / 2422.30 ±239.43 / 2836.05 ms │              1680.73 / 1711.87 ±22.73 / 1746.08 ms │ +1.42x faster │
│ QQuery 20 │    2879.17 / 3144.97 ±182.39 / 3314.36 ms │             2572.29 / 2727.25 ±108.62 / 2871.43 ms │ +1.15x faster │
│ QQuery 21 │    5149.42 / 5586.17 ±329.53 / 6009.03 ms │              4877.62 / 4977.41 ±97.88 / 5105.06 ms │ +1.12x faster │
│ QQuery 22 │   9707.46 / 9937.73 ±221.73 / 10315.45 ms │             8687.22 / 8888.12 ±150.87 / 9023.85 ms │ +1.12x faster │
│ QQuery 23 │    3713.41 / 3949.50 ±168.21 / 4190.39 ms │              1871.80 / 1959.45 ±66.26 / 2026.87 ms │ +2.02x faster │
│ QQuery 24 │    2464.74 / 2666.89 ±119.90 / 2825.64 ms │              1482.20 / 1585.01 ±86.48 / 1736.16 ms │ +1.68x faster │
│ QQuery 25 │    2522.57 / 2712.88 ±140.15 / 2889.83 ms │              2253.42 / 2329.99 ±60.54 / 2436.69 ms │ +1.16x faster │
│ QQuery 26 │    2419.11 / 2680.81 ±174.85 / 2895.44 ms │             1581.70 / 1730.56 ±111.73 / 1881.88 ms │ +1.55x faster │
│ QQuery 27 │    5230.25 / 5465.87 ±254.92 / 5942.69 ms │             4955.57 / 5105.89 ±131.11 / 5339.26 ms │ +1.07x faster │
│ QQuery 28 │ 10150.98 / 10312.83 ±102.55 / 10447.80 ms │              8958.66 / 9027.47 ±41.36 / 9071.48 ms │ +1.14x faster │
│ QQuery 29 │    2492.48 / 2649.88 ±143.14 / 2889.53 ms │              2132.06 / 2214.46 ±57.24 / 2285.01 ms │ +1.20x faster │
│ QQuery 30 │    5367.93 / 5454.02 ±135.39 / 5721.43 ms │             4427.04 / 4604.45 ±158.65 / 4823.13 ms │ +1.18x faster │
│ QQuery 31 │    4979.49 / 5384.49 ±282.74 / 5719.00 ms │              4568.33 / 4612.85 ±31.07 / 4664.02 ms │ +1.17x faster │
│ QQuery 32 │    2880.71 / 3153.22 ±178.50 / 3415.00 ms │              2614.76 / 2657.18 ±30.05 / 2701.67 ms │ +1.19x faster │
│ QQuery 33 │    3708.92 / 3885.38 ±106.32 / 3978.94 ms │              3672.47 / 3735.22 ±55.44 / 3810.69 ms │     no change │
│ QQuery 34 │    3820.11 / 4068.86 ±276.42 / 4592.44 ms │              3663.35 / 3793.80 ±98.75 / 3894.39 ms │ +1.07x faster │
│ QQuery 35 │     2818.39 / 2965.07 ±93.25 / 3092.19 ms │              2364.85 / 2478.73 ±98.30 / 2619.06 ms │ +1.20x faster │
│ QQuery 36 │     1077.00 / 1158.69 ±67.86 / 1280.83 ms │                486.87 / 626.51 ±126.51 / 818.18 ms │ +1.85x faster │
│ QQuery 37 │     1032.10 / 1148.39 ±95.86 / 1290.63 ms │                  709.43 / 710.98 ±0.97 / 712.06 ms │ +1.62x faster │
│ QQuery 38 │     1238.80 / 1345.21 ±65.85 / 1399.93 ms │                526.52 / 679.91 ±121.20 / 878.66 ms │ +1.98x faster │
│ QQuery 39 │       884.73 / 980.28 ±99.41 / 1155.26 ms │                 472.67 / 535.33 ±71.03 / 668.77 ms │ +1.83x faster │
│ QQuery 40 │    1126.11 / 1310.05 ±162.67 / 1517.45 ms │                 518.06 / 623.58 ±96.91 / 794.00 ms │ +2.10x faster │
│ QQuery 41 │     1311.22 / 1344.55 ±20.43 / 1370.10 ms │               718.64 / 936.94 ±140.15 / 1147.76 ms │ +1.44x faster │
│ QQuery 42 │      946.86 / 1017.07 ±53.65 / 1097.64 ms │                  519.52 / 523.61 ±2.17 / 525.47 ms │ +1.94x faster │
└───────────┴───────────────────────────────────────────┴────────────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary                                                 ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD)                                                 │ 146276.69ms │
│ Total Time (parquet-morsel-driven-execution-237164415184908839)   │ 119424.10ms │
│ Average Time (HEAD)                                               │   3401.78ms │
│ Average Time (parquet-morsel-driven-execution-237164415184908839) │   2777.30ms │
│ Queries Faster                                                    │          41 │
│ Queries Slower                                                    │           0 │
│ Queries with No Change                                            │           2 │
│ Queries with Failure                                              │           0 │
└───────────────────────────────────────────────────────────────────┴─────────────┘

Resource Usage

clickbench_pushdown — base (merge-base)

Metric	Value
Wall time	732.7s
Peak memory	35.5 GiB
Avg memory	27.4 GiB
CPU user	1211.5s
CPU sys	150.6s
Disk read	0 B
Disk write	4.0 GiB

clickbench_pushdown — branch

Metric	Value
Wall time	598.4s
Peak memory	36.8 GiB
Avg memory	28.1 GiB
CPU user	1213.4s
CPU sys	145.4s
Disk read	0 B
Disk write	828.0 KiB

adriangb · 2026-03-16T17:51:11Z

Seems like morselization is even better w/ added latency??

Dandandan · 2026-03-16T18:34:13Z

Seems like morselization is even better w/ added latency??

Yeah still seems to improve things greatly!

However I think the relative improvement is smaller, because of lower utilization, perhaps also a bit due to somewhat smaller request sizes / more requests (and having no prefetching yet in place).

Dandandan · 2026-03-16T18:38:51Z

Oh wait - the relative improvement seems bigger still! 15% to 22% 🤯

I think it is mainly because of the added skew (from different request latencies). Nice to see this confirmed as well using this bench @alamb

Omega359 · 2026-03-16T18:59:15Z

The results seem very promising! I'm curious at what point you think this will be ready for final review and inclusion? Right now it seems to be still in the development phase?

alamb · 2026-03-18T21:47:36Z

The results seem very promising! I'm curious at what point you think this will be ready for final review and inclusion? Right now it seems to be still in the development phase?

I am trying to make a "production ready" version here

Sketch out a Morselize API #20820

I am hoping it will be ready / show the same improvements this PR does "soon" (friday hopefully)

Dandandan · 2026-03-20T07:17:18Z

The results seem very promising! I'm curious at what point you think this will be ready for final review and inclusion? Right now it seems to be still in the development phase?

I am working on a proposal

google-labs-jules bot and others added 2 commits February 22, 2026 13:12

Proto

a67f9ac

github-actions bot added core Core DataFusion crate common Related to common crate proto Related to proto crate datasource Changes to the datasource crate labels Feb 22, 2026

Dandandan mentioned this pull request Feb 22, 2026

PoC: introduce morsel-driven parquet scan #20477

Closed

Dandandan changed the title ~~PoC: Parquet morsel driven execution~~ PoC: introduce morsel-driven parquet scan Feb 22, 2026

Dandandan added 9 commits February 22, 2026 18:55

Proto

d0da5da

Fmt

32eec3c

Merge remote-tracking branch 'upstream/main' into parquet-morsel-driv…

5dc895c

…en-execution-237164415184908839

Proto

cc73788

Fix

d517b5d

Fix

de1606d

Clippy

950f6db

Refactor

7f57317

WIP

fd6d7fd

Dandandan changed the title ~~PoC: introduce morsel-driven parquet scan~~ Introduce morsel-driven parquet scan Feb 23, 2026

WIP

37126bf

Dandandan changed the title ~~Introduce morsel-driven parquet scan~~ Introduce morsel-driven Parquet scan Feb 23, 2026

Dandandan added 5 commits February 23, 2026 09:31

WIP

2d3c33e

WIP

98f0ea9

WIP

a389b02

Update

4065448

Update

415315d

Config

13b4977

github-actions bot added the documentation Improvements or additions to documentation label Feb 23, 2026

Dandandan added 2 commits February 23, 2026 12:53

Test

a30c3f8

Refactor

8b32ca8

Revert "Add the exp. prefetching"

b02a9c0

This reverts commit 4fc8edd.

Dandandan and others added 2 commits March 16, 2026 17:34

Merge remote-tracking branch 'upstream/main' into parquet-morsel-driv…

18c9630

…en-execution-237164415184908839 # Conflicts: # datafusion/datasource-parquet/src/opener.rs # datafusion/sqllogictest/test_files/clickbench.slt

Conversation

Dandandan commented Feb 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

Morsel-Driven Execution Architecture (partly claude-generated)

The Problem It Solves

Key Types

FileOpener Trait Extension -- datafusion/datasource/src/file_stream.rs:498

FileStream State Machine -- datafusion/datasource/src/file_stream.rs:141

Configuration

Benchmark results

Acknowledgements

Uh oh!

Dandandan commented Feb 23, 2026

Uh oh!

alamb-ghbot commented Mar 12, 2026

Uh oh!

Dandandan commented Mar 13, 2026

Uh oh!

adriangbot commented Mar 13, 2026

Uh oh!

adriangbot commented Mar 13, 2026

Uh oh!

adriangbot commented Mar 13, 2026

Uh oh!

adriangbot commented Mar 13, 2026

Uh oh!

adriangbot commented Mar 13, 2026

Uh oh!

adriangbot commented Mar 13, 2026

Uh oh!

adriangbot commented Mar 13, 2026

Uh oh!

adriangbot commented Mar 13, 2026

Uh oh!

adriangbot commented Mar 13, 2026

Uh oh!

Dandandan commented Mar 13, 2026

Uh oh!

adriangbot commented Mar 13, 2026

Uh oh!

adriangbot commented Mar 13, 2026

Uh oh!

adriangbot commented Mar 13, 2026

Uh oh!

Dandandan commented Mar 16, 2026

Uh oh!

adriangbot commented Mar 16, 2026

Uh oh!

adriangbot commented Mar 16, 2026

Uh oh!

Dandandan commented Mar 16, 2026

Uh oh!

adriangbot commented Mar 16, 2026

Uh oh!

adriangbot commented Mar 16, 2026

Uh oh!

adriangb commented Mar 16, 2026

Uh oh!

Dandandan commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Dandandan commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Omega359 commented Mar 16, 2026

Uh oh!

alamb commented Mar 18, 2026

Uh oh!

Dandandan commented Mar 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Dandandan commented Feb 22, 2026 •

edited

Loading

`FileOpener` Trait Extension -- `datafusion/datasource/src/file_stream.rs:498`

`FileStream` State Machine -- `datafusion/datasource/src/file_stream.rs:141`

Dandandan commented Mar 16, 2026 •

edited

Loading

Dandandan commented Mar 16, 2026 •

edited

Loading