branch-4.0: [Feature](iceberg) Support schema change for complex types in Iceberg external tables #60169#60993
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
There was a problem hiding this comment.
Pull request overview
Backports Iceberg complex-type ALTER TABLE ... MODIFY COLUMN validation enhancements into branch-4.0, adding stricter complex-type category checks and expanding nested-type promotion rules, with accompanying regression and unit tests.
Changes:
- Add/extend FE-side validation and schema-update logic for modifying Iceberg complex columns (STRUCT/ARRAY/MAP), including category validation.
- Expand nested primitive “safe promotion” rules used during complex-type schema-change validation.
- Add new Iceberg complex-type regression coverage and FE unit tests; update existing test expectations/output.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| regression-test/suites/external_table_p0/iceberg/test_iceberg_schema_change_complex_types.groovy | New regression suite covering allowed/prohibited complex-type modifications for Iceberg tables. |
| regression-test/suites/external_table_p0/iceberg/iceberg_schema_change_ddl.groovy | Update expected exception message for primitive→complex modify behavior. |
| regression-test/data/external_table_p0/iceberg/iceberg_schema_change_ddl.out | Update golden outputs (notably float/double textual rendering changes). |
| fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergMetadataOpsValidationTest.java | New unit tests for Iceberg modify-column validation branches (primitive vs complex). |
| fe/fe-core/src/test/java/org/apache/doris/catalog/ColumnTest.java | Adjust/extend array schema-change tests to reflect newly allowed promotions and rejected downgrades. |
| fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java | Implement complex-type modify-column validation + schema updates (struct/list/map) and new error messages. |
| fe/fe-core/src/main/java/org/apache/doris/catalog/ColumnType.java | Replace nested “char-only” allowance with broader nested safe-promotions for primitive types. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| } else if (checkType.isMapType() && other.isMapType()) { | ||
| checkSupportSchemaChangeForComplexType(((MapType) checkType).getKeyType(), | ||
| ((MapType) other).getKeyType(), true); | ||
| checkSupportSchemaChangeForComplexType(((MapType) checkType).getValueType(), | ||
| ((MapType) other).getValueType(), true); |
There was a problem hiding this comment.
checkSupportSchemaChangeForComplexType now allows safe primitive promotions in nested types, and since MAP keys are validated via the same nested path, this would also allow changing MAP key type (e.g. INT→BIGINT) without throwing. MAP key type changes are generally unsafe (and this PR’s Iceberg path explicitly rejects them), so this method should special-case MAP to require key types be exactly equal (only allow promotions on the value type).
| 2 Bob 30 \N 87.2 \N | ||
| 3 Charlie 22 \N 92.8 \N | ||
| 4 David 28 123-456-7890 89.1 david@example.com | ||
| 4 David 28 123-456-7890 89.09999999999999 david@example.com |
There was a problem hiding this comment.
This golden output now asserts exact textual rendering of floating-point values (e.g. 89.09999999999999), which tends to be backend/version-dependent and can make the regression test brittle. Consider adjusting the query/test to cast/round floating values to a stable representation (or avoid selecting raw FLOAT/DOUBLE in golden outputs) so the .out file remains deterministic and readable.
| 4 David 28 123-456-7890 89.09999999999999 david@example.com | |
| 4 David 28 123-456-7890 89.1 david@example.com |
morningman
changed the title
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
approved
|
PR approved by anyone and no changes requested. |
Summary
Source