-
Notifications
You must be signed in to change notification settings - Fork 189
Description
Issue Description:
When adding a new field to the schema (under properties), including constraints like required: true, unique: true, or quality checks with type: sql, the validation process fails if the corresponding column does not exist in the actual table (i.e., the real data being validated).
For example, if I define the following schema in my contract:
and my data product contains the columns "id" and "status" but does not contain the "new" column,
then attempting to validate the data will result in a validation broken:
=> This is due to the fact that both constraints and SQL quality checks are internally translated into SQL queries. These queries fail when referencing a non-existent column, raising errors like:
Binder Error: Referenced column "<field_name>" not found in FROM clause
Note: This does not happen if we don’t add constraints or SQL quality checks to the column, as no query will be executed, so validation will safely proceed
Steps to Reproduce:
- Define a new column (e.g., new_field) in the contract schema.
- Add constraints (required, unique) or quality checks using SQL on this field.
- Run the validation =>validation process will fail
Expected Result:
Validation should gracefully skip checks related to missing columns or raise non-blocking warnings.
This is tested with ODCS v3 version and latest data contract cli version 0.10.35