[branch-52] Fix Arrow Spill Underrun (#20159)

[hareshkh]

Which issue does this PR close?

• Related to Release DataFusion 52.3.0 (minor/) Release (Mar 2026) #20681
• Backport of Fix Arrow Spill Underrun #20159

Rationale for this change

This adjusts the way that the spill channel works. Currently we have a spill writer & reader pairing which uses a mutex to coordindate when a file is ready to be read.

What happens is, that because we were using a spawn_buffered call, the read task would race ahead trying to read a file which is yet to be written out completely.

Alongside this, we need to flush each write to the file, as there is a chance that another thread may see stale data.

What changes are included in this PR?

Adds a flush on write, and converts the read task to not buffer reads.

Are these changes tested?

I haven't written a test, but I have been running the example in the attached issue. While it now fails with allocation errors, the original error goes away.

Are there any user-facing changes?

Nope

[hareshkh]

@alamb @adriangb : Can you please take a look at this? (You reviewed the original PR)

[adriangb]

Could you link in the comment description (so it ends up in the commit) to the original PR, and how you created this diff (e.g. "cherry picked abc123")?

[alamb]

Looks good to me -- thank you @hareshkh and @adriangb

I linked the PR that is being backported and I verified the diff is the same, and removed the duplicated template in the PR description

I also ran the problem query and verified that that before this PR , it fails like this:

andrewlamb@Andrews-MacBook-Pro-3:~/Software/datafusion$ ~/Software/datafusion-cli/datafusion-cli-52.2.0  -m 1G -c "SELECT \"UserID\", extract(minute FROM to_timestamp_seconds(\"EventTime\")) AS m, \"SearchPhrase\", COUNT(*) FROM '/Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned' GROUP BY \"UserID\", m, \"SearchPhrase\" ORDER BY COUNT(*) DESC LIMIT 10;"
DataFusion CLI v52.2.0
Error: Arrow error: Io error: failed to fill whole buffer

And after this PR it succeeds

andrewlamb@Andrews-MacBook-Pro-3:~/Software/datafusion$ ./target/profiling/datafusion-cli  -m 1G -c "SELECT \"UserID\", extract(minute FROM to_timestamp_seconds(\"EventTime\")) AS m, \"SearchPhrase\", COUNT(*) FROM '/Users/andrewlamb/Software/datafusion/benchmarks/data/hits_partitioned' GROUP BY \"UserID\", m, \"SearchPhrase\" ORDER BY COUNT(*) DESC LIMIT 10;"
DataFusion CLI v52.2.0
+---------------------+----+--------------+----------+
| UserID              | m  | SearchPhrase | count(*) |
+---------------------+----+--------------+----------+
| 1313338681122956954 | 31 |              | 589      |
| 1313338681122956954 | 28 |              | 578      |
| 1313338681122956954 | 29 |              | 572      |
| 1313338681122956954 | 33 |              | 567      |
| 1313338681122956954 | 27 |              | 557      |
| 1313338681122956954 | 32 |              | 554      |
| 1313338681122956954 | 30 |              | 552      |
| 1313338681122956954 | 34 |              | 546      |
| 1313338681122956954 | 26 |              | 540      |
| 1313338681122956954 | 10 |              | 539      |
+---------------------+----+--------------+----------+
10 row(s) fetched.
Elapsed 184.374 seconds.

[alamb]

Thanks again @hareshkh