Skip to content

Data Contract Validation breaks for non-existent columns with constraints or SQL quality checks #888

@Aya-OURAG-Speedykom

Description

@Aya-OURAG-Speedykom

Issue Description:
When adding a new field to the schema (under properties), including constraints like required: true, unique: true, or quality checks with type: sql, the validation process fails if the corresponding column does not exist in the actual table (i.e., the real data being validated).

For example, if I define the following schema in my contract:

Image

and my data product contains the columns "id" and "status" but does not contain the "new" column,

then attempting to validate the data will result in a validation broken:

Image

=> This is due to the fact that both constraints and SQL quality checks are internally translated into SQL queries. These queries fail when referencing a non-existent column, raising errors like:

Binder Error: Referenced column "<field_name>" not found in FROM clause

Note: This does not happen if we don’t add constraints or SQL quality checks to the column, as no query will be executed, so validation will safely proceed

Steps to Reproduce:

  • Define a new column (e.g., new_field) in the contract schema.
  • Add constraints (required, unique) or quality checks using SQL on this field.
  • Run the validation =>validation process will fail

Expected Result:
Validation should gracefully skip checks related to missing columns or raise non-blocking warnings.

This is tested with ODCS v3 version and latest data contract cli version 0.10.35

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions