Simplify bundle JSON schema annotation system#4670
Draft
shreyas-goenka wants to merge 12 commits intomainfrom
Draft
Simplify bundle JSON schema annotation system#4670shreyas-goenka wants to merge 12 commits intomainfrom
shreyas-goenka wants to merge 12 commits intomainfrom
Conversation
shreyas-goenka
requested review from
andrewnester,
anton-107,
denik,
pietern and
simonfaltum
as code owners
shreyas-goenka
force-pushed
the
simplify-schema-annotations
branch
from
680bea6 to
f79f5b0
Compare
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Collaborator
|
Commit: 2d318c0
16 interesting tests: 7 SKIP, 6 RECOVERED, 3 flaky
Top 20 slowest tests (at least 2 minutes):
|
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
force-pushed
the
simplify-schema-annotations
branch
from
2f2a2d7 to
6225f44
Compare
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
force-pushed
the
simplify-schema-annotations
branch
from
e58258b to
08a6413
Compare
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
force-pushed
the
simplify-schema-annotations
branch
from
08a6413 to
963126c
Compare
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
force-pushed
the
simplify-schema-annotations
branch
from
963126c to
a508ab4
Compare
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Read OpenAPI descriptions directly from the spec at runtime instead of pre-extracting them into annotation YAML files. This makes the OpenAPI spec a requirement for schema generation and eliminates annotation files that flip-flopped on every SDK bump. Changes: - Remove annotations_openapi.yml and annotations_openapi_overrides.yml - Merge manual overrides into a single annotations.yml - Use bundle paths (e.g. resources.jobs.*) as annotation keys instead of Go type paths - Add path_mapping.go to resolve bundle paths to Go types via reflection - Remove bundle/docsgen (was the only other consumer of annotations) - Makefile schema targets now require DATABRICKS_OPENAPI_SPEC Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix copyloopvar lint warnings in annotations.go - Fix gofmt issue in path_mapping_test.go - Use maps.Copy instead of manual loop - Restore permission level enums, lifecycle descriptions, markdownDescription, and SDK type overrides from the deleted annotations_openapi_overrides.yml into annotations.yml - Update getOpenApiAnnotations to check embedded types for promoted field descriptions from OpenAPI spec - Update TestNoDetachedAnnotations to handle Go type path keys Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…qlWarehouse, and SDK types Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ter rebase Add a "Download OpenAPI spec" step to the validate-generated-is-up-to-date CI job so make schema can find .codegen/openapi.json. Also regenerate schema and python codegen to sync with latest main (removed grant types, updated SDK descriptions). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When DATABRICKS_OPENAPI_SPEC is set, OpenAPI-derived descriptions, deprecation info, and output-only markers are now persisted into annotations.yml. This makes the schema fully reproducible from annotations.yml alone, so CI can run `make schema` without downloading the OpenAPI spec. - Change OutputOnly from *bool to string for clean YAML serialization - Add schema-openapi Makefile target for generating with the spec - Revert CI download step; make schema now works without the spec - Merge annotations from both Go type paths and bundle paths Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…tions The Permission type is shared across alerts, dashboards, database_instances, and postgres_projects. Only one canonical annotation entry is needed since the path mapping selects the shortest path. Also adds nolint:forbidigo comment and uses slices.Contains for modernize linter. Co-authored-by: Isaac
shreyas-goenka
force-pushed
the
simplify-schema-annotations
branch
from
9a16182 to
fe3b5b7
Compare
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Permission level descriptions are now derived from the OpenAPI spec at runtime rather than being baked into annotations. Enum values are sorted alphabetically. Co-authored-by: Isaac
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
When annotations.yml has entries for the same type under both a Go type path and a bundle path, the old maps.Copy would overwrite entire descriptors depending on random map iteration order. This caused the whl description in PipelineLibrary to appear or disappear non-deterministically. Use mergeDescriptor to properly combine descriptor fields when merging duplicate annotation entries. Co-authored-by: Isaac
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
The docsgen package is no longer used anywhere. Remove it along with its .wsignore exclusion entries. Co-authored-by: Isaac
shreyas-goenka
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
simonfaltum
reviewed
Mar 12, 2026
Member
simonfaltum
left a comment
simonfaltum
left a comment
There was a problem hiding this comment.
Review (automated, 2 agents)
Verdict: Not ready yet | 0 Critical | 3 Major | 3 Gap(Major) | 3 Nit | 2 Suggestion
[Major] syncAnnotations overwrites hand-written annotations with OpenAPI
bundle/internal/schema/annotations.go (syncAnnotations)
mergeDescriptor(existing, desc) treats the OpenAPI annotation as override, but the intent (stated in the comment: "preserving hand-written fields") is the opposite. Compare with addAnnotations which correctly uses OpenAPI as base.
Suggestion: Swap arguments: mergeDescriptor(desc, existing).
[Major] OpenAPI metadata not persisted for fields with existing descriptions
bundle/internal/schema/annotations.go:86-101
If annotations.yml has a description for a field, the entire OpenAPI annotation is skipped during persistence, losing enum values, deprecation messages, preview status, and output-only markers. This breaks the "reproducible without the spec" goal.
Suggestion: Always persist the OpenAPI descriptor as base, merge annotations.yml on top.
[Major] PLACEHOLDER insertion for new CLI-only fields removed
The old missingAnnotations flow inserted PLACEHOLDER entries for fields missing descriptions. This refactor removes that behavior. New fields defined only in CLI types won't get added to annotations.yml.
Suggestion: Reintroduce a pass that records missing non-OpenAPI fields with PLACEHOLDER.
[Gap (Major)] TestRequiredAnnotationsForNewFields now skipped by default
bundle/internal/schema/main_test.go:43-57
Gated on DATABRICKS_OPENAPI_SPEC env var not set in standard CI. Missing annotations go undetected.
Suggestion: Split into always-on CLI check + spec-backed SDK check, or ensure CI sets the env var.
[Gap (Major)] No unit tests for getOpenApiAnnotations, mergeDescriptor, or syncAnnotations
bundle/internal/schema/annotations.go
Three significant logic functions with no direct tests. The merge direction bug (finding 1) would be caught by a simple unit test.
[Gap (Nit)] AGENTS.md still references deleted make docs target
AGENTS.md:29 - Dead instruction after docsgen removal.
[Nit] Mixed key formats in annotations.yml
11 entries still use Go type paths instead of bundle paths.
[Nit] bundlePathToType stale entries on re-mapping
path_mapping.go:130-140 - Old reverse mapping not cleaned up when shorter path found. Harmless but messy.
[Nit] OutputOnly type change (*bool to string) has asymmetric semantics
"false" is truthy in mergeDescriptor (non-empty) but falsy in assignAnnotation (not "true").
[Suggestion] recordMapping heuristic: use depth instead of string length
path_mapping.go - strings.Count(path, ".") is more robust than len(path) for canonical path selection.
[Suggestion] getOpenApiAnnotations processes embedded types before parent
Works correctly but counterintuitive ordering. Consider parent-first with embedded fallback.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Simplify how bundle JSON schema annotations are managed:
annotations.ymlso the schema is reproducible without the spec.annotations_openapi.ymlandannotations_openapi_overrides.ymlwhich flip-flopped on every SDK bump. Merge everything into a singleannotations.yml.resources.jobs.*) as annotation keys instead of Go type paths.bundle/docsgen/package.Test plan
make schemais idempotent)🤖 Generated with Claude Code