Added support of ApacheArrow batches for BulkUpsert#593
Merged
alex268 merged 9 commits intoydb-platform:release_v2.4.0from Feb 20, 2026
Merged
Added support of ApacheArrow batches for BulkUpsert#593alex268 merged 9 commits intoydb-platform:release_v2.4.0from
alex268 merged 9 commits intoydb-platform:release_v2.4.0from
Conversation
There was a problem hiding this comment.
Pull request overview
Adds Apache Arrow batch support for executeBulkUpsert by introducing an extensible BulkUpsertData abstraction and an ApacheArrowWriter to build Arrow-encoded batches, with unit/integration tests to validate schema mapping and end-to-end upserts.
Changes:
- Introduce
BulkUpsertDataas an interface with concrete implementations for Proto rows, CSV, and Arrow batches. - Add
ApacheArrowWriterfor building Arrow schema + record batches for bulk upsert. - Add/adjust unit + integration tests and Maven dependencies/profiles for Apache Arrow.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| table/src/main/java/tech/ydb/table/query/ApacheArrowWriter.java | New Arrow writer for encoding YDB types into Arrow vectors + IPC schema/batch serialization. |
| table/src/main/java/tech/ydb/table/query/BulkUpsertData.java | Refactors bulk upsert payload into an interface with factories. |
| table/src/main/java/tech/ydb/table/query/BulkUpsertProtoData.java | New implementation for protobuf TypedValue rows payload. |
| table/src/main/java/tech/ydb/table/query/BulkUpsertArrowData.java | New implementation for Arrow batch payload (schema + data). |
| table/src/main/java/tech/ydb/table/query/BulkUpsertCsvData.java | New implementation for CSV payload. |
| table/src/main/java/tech/ydb/table/utils/LittleEndian.java | Small Javadoc enhancement for bswap(long). |
| table/src/main/java/tech/ydb/table/Session.java | Updates bulk upsert default method to use new BulkUpsertData factories. |
| table/src/test/java/tech/ydb/table/query/ApacheArrowWriterTest.java | New unit tests for Arrow schema/type mapping and validation behavior. |
| table/src/test/java/tech/ydb/table/integration/BulkUpsertTest.java | New integration tests covering Proto and Arrow bulk upserts against YDB. |
| table/src/test/java/tech/ydb/table/integration/AllTypesRecord.java | New test helper generating random records and writing/asserting rows for multiple types. |
| table/src/test/java/tech/ydb/table/integration/ReadTableTest.java | Adapts bulk upsert test setup to new BulkUpsertData API. |
| table/src/test/java/tech/ydb/table/integration/ValuesReadTest.java | Removes a stray debug println. |
| table/pom.xml | Adds optional Arrow deps and a JDK profile for Arrow-related JVM args. |
| pom.xml | Adds Arrow version/deps in dependencyManagement and tweaks JDK profile properties. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
table/src/test/java/tech/ydb/table/integration/AllTypesRecord.java
Outdated
Show resolved
Hide resolved
table/src/test/java/tech/ydb/table/integration/AllTypesRecord.java
Outdated
Show resolved
Hide resolved
table/src/test/java/tech/ydb/table/integration/AllTypesRecord.java
Outdated
Show resolved
Hide resolved
table/src/main/java/tech/ydb/table/query/ApacheArrowWriter.java
Outdated
Show resolved
Hide resolved
table/src/main/java/tech/ydb/table/query/ApacheArrowWriter.java
Outdated
Show resolved
Hide resolved
ee1c197 to
e2397f7
Compare
e2397f7 to
96c01e8
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## release_v2.4.0 #593 +/- ##
====================================================
+ Coverage 66.36% 67.14% +0.77%
- Complexity 2865 2907 +42
====================================================
Files 345 348 +3
Lines 14430 14751 +321
Branches 1492 1523 +31
====================================================
+ Hits 9577 9905 +328
+ Misses 4189 4181 -8
- Partials 664 665 +1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
f5ca5d5 to
ca0d8cb
Compare
pnv1
approved these changes
Feb 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.