fix(topk): avoid overflow panic in interleave emission#20494
Open
aviralgarg05 wants to merge 4 commits intoapache:mainfrom
Open
fix(topk): avoid overflow panic in interleave emission#20494aviralgarg05 wants to merge 4 commits intoapache:mainfrom
aviralgarg05 wants to merge 4 commits intoapache:mainfrom
Conversation
kosiew
requested changes
Mar 17, 2026
Contributor
kosiew
left a comment
kosiew
left a comment
There was a problem hiding this comment.
@aviralgarg05
Thanks for working on this
Member
|
FYI, I encountered a similar case in sort merge: #20922, and there is a PR changing panic to error for interleavehttps://github.com/apache/arrow-rs/pull/9549 |
xudong963
reviewed
Mar 18, 2026
| } | ||
| } | ||
|
|
||
| enum VarWidthKind { |
Member
There was a problem hiding this comment.
how about Utf8View/BinaryView
Member
There was a problem hiding this comment.
what's more, a Struct containing a Utf8 field
| .ok_or_else(|| { | ||
| internal_datafusion_err!("Expected Utf8 array for TopK interleave") | ||
| })? | ||
| .value_length(row) as i64, |
Member
There was a problem hiding this comment.
how about checking total column data buffer sizes first and skip per-row accounting entirely when the total is well under the limit
Comment on lines
+1050
to
+1063
| VarWidthKind::List => array | ||
| .as_any() | ||
| .downcast_ref::<ListArray>() | ||
| .ok_or_else(|| { | ||
| internal_datafusion_err!("Expected List array for TopK interleave") | ||
| })? | ||
| .value_length(row) as i64, | ||
| VarWidthKind::Map => array | ||
| .as_any() | ||
| .downcast_ref::<MapArray>() | ||
| .ok_or_else(|| { | ||
| internal_datafusion_err!("Expected Map array for TopK interleave") | ||
| })? | ||
| .value_length(row) as i64, |
Member
There was a problem hiding this comment.
For List and Map, value_length(row) returns the number of child elements, not their byte size.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
While testing, I found that
ORDER BY ... LIMITcould panic in TopK when interleaving very largeUtf8values. The problem was that TopK tried to interleave all selected rows in a single call, which can overflow Arrow’s i32 string offsets in extreme cases. This PR removes that panic path so the operator behaves safely under large-string workloads.What changes are included in this PR?
TopK.batch_sizefield.test_topk_heap_emit_with_state_respects_batch_size.Are these changes tested?
Yes.
datafusion/physical-plan/src/topk/mod.rsfor chunked TopK emission behavior.cargo fmt --all --check.cargo clippy -p datafusion-physical-plan --lib --tests -- -D warnings.cargo test -p datafusion-physical-plan topk::tests:: -- --nocapturethree times after final changes.Are there any user-facing changes?
Yes. For extreme large-string cases in
ORDER BY ... LIMIT, TopK no longer panics with overflow; it now follows normal error/execution handling.