direct: Fix permissions state path to match input config schema

NEXT_CHANGELOG.md

@@ -7,6 +7,7 @@
 ### Bundles
 * Modify grants to use SDK types ([#4666](https://github.com/databricks/cli/pull/4666))
 * Modify permissions to use SDK types where available. This makes DABs validate permission levels, producing a warning on the unknown ones ([#4686](https://github.com/databricks/cli/pull/4686))
+* direct: Fix permissions state path to match input config schema ([#4703](https://github.com/databricks/cli/pull/4703))
 
 ### Dependency updates
 * Bump databricks-sdk-go from v0.112.0 to v0.119.0 ([#4631](https://github.com/databricks/cli/pull/4631), [#4695](https://github.com/databricks/cli/pull/4695))

TASKS/001.PLAN.md

@@ -0,0 +1,91 @@
+# Implementation Plan: `__EMBED__` Convention for Struct Walkers
+
+## Problem
+
+In bundle config, permissions are a direct slice on the resource:
+```
+resources.jobs.foo.permissions[0].user_name
+```
+
+In direct engine state, permissions are wrapped in `PermissionsState`:
+```go
+type PermissionsState struct {
+    ObjectID    string                     `json:"object_id"`
+    Permissions []iam.AccessControlRequest `json:"permissions,omitempty"`
+}
+```
+
+This creates an extra level in the path:
+```
+resources.jobs.foo.permissions.permissions[0].user_name
+```
+
+When reference resolution strips the prefix `resources.jobs.foo.permissions` and tries to navigate `[0].service_principal_name` within PermissionsState, it fails with "cannot index struct" because the struct expects `permissions[0].service_principal_name`.
+
+## Solution
+
+Introduce `__EMBED__` json tag convention: when a struct field has `json:"__EMBED__"`, all struct walkers in `libs/structs/` treat it as transparent - they don't add the field name to the path. The field's contents appear directly at the parent level.
+
+With `__EMBED__`, PermissionsState becomes:
+```go
+type PermissionsState struct {
+    ObjectID    string                     `json:"object_id"`
+    Permissions []iam.AccessControlRequest `json:"__EMBED__,omitempty"`
+}
+```
+
+Now `[0].user_name` navigates correctly through the embedded slice.
+
+## Implementation Steps
+
+### 1. `libs/structs/structtag/jsontag.go`
+- Add `const EmbedTagName = "__EMBED__"`.
+- Add `IsEmbed()` method on `JSONTag`.
+- Add test.
+
+### 2. `libs/structs/structwalk/walk.go` — `walkStruct`
+- After parsing json tag, check if `jsonTag.IsEmbed()`.
+- If so, walk the field value at the parent path (don't add field name to path), like anonymous embedding.
+- Still respect omitempty.
+- Add test case to existing table tests.
+
+### 3. `libs/structs/structwalk/walktype.go` — `walkTypeStruct`
+- Same logic: if json tag name is `__EMBED__`, walk at parent path level.
+- Add test case to existing table tests.
+
+### 4. `libs/structs/structaccess/get.go`
+- In `getValue`: when `cur` is a struct and the path node is an index or key-value selector, find the `__EMBED__` field (a slice) and navigate into it.
+- In `findFieldInStruct`: skip fields with `__EMBED__` tag name (they're not accessible by name).
+- Add helper `findEmbedField(v reflect.Value) reflect.Value`.
+- Add test cases to existing tests.
+
+### 5. `libs/structs/structaccess/set.go`
+- In `setValueAtNode`: when parent is a struct and the node is an index, resolve through `__EMBED__` field.
+- Add test cases.
+
+### 6. `libs/structs/structaccess/typecheck.go`
+- In `validateNodeSlice`: when type is struct and path expects index/bracket-star/key-value, find `__EMBED__` field type.
+- In `FindStructFieldByKeyType`: skip `__EMBED__` fields from string key lookups.
+- Add helper `findEmbedFieldType(t reflect.Type) reflect.Type`.
+- Add test cases.
+
+### 7. `libs/structs/structdiff/diff.go`
+- In `diffStruct`: when field has `__EMBED__` tag, use parent path (don't add field name).
+- Still handle omitempty/zero/forced as normal.
+- Add test cases.
+
+### 8. `bundle/direct/dresources/permissions.go`
+- Change `Permissions` tag to `json:"__EMBED__,omitempty"`.
+- Update `KeyedSlices()` to use `""` as key instead of `"permissions"` (since the slice is now at root level).
+
+### 9. Acceptance Tests
+- Enable direct engine in `acceptance/bundle/apps/job_permissions/test.toml`.
+- Add new acceptance tests for:
+  - Referencing a permission field from another resource.
+  - Referencing another resource from a permission field.
+  - Permission indices from remote backend (index not in local config).
+
+### 10. Validation
+- `make generate` runs clean.
+- `make build && make test-update` succeeds.
+- `make test-update-aws` for integration tests.

TASKS/001.RESPONSE.md

@@ -0,0 +1 @@
+This request is done, all tests pass. There should be summary but it was lost.

TASKS/001.md

@@ -0,0 +1,51 @@
+Permissions schema in direct engine.
+
+We have this state type in bundle/direct/dresources/permissions.go
+
+type PermissionsState struct {
+        ObjectID    string                     `json:"object_id"`
+        Permissions []iam.AccessControlRequest `json:"permissions,omitempty"`
+}
+
+
+however, in bundle config there is schema:
+
+bundle/config/resources/job.go-type Job struct {
+bundle/config/resources/job.go- BaseResource
+bundle/config/resources/job.go- jobs.JobSettings
+bundle/config/resources/job.go-
+bundle/config/resources/job.go: Permissions []JobPermission `json:"permissions,omitempty"`
+bundle/config/resources/job.go-}
+
+
+There is a mismatch, you access permission in input schema as resources.jobs.foo.permissions[0] but you access
+state via resources.jobs.foo.permissions.permissions[0] due to extra struct wrapping the slice.
+
+
+This mismatch is problematic and it means we cannot declare dependencies based on input schema.
+
+What we need to achieve is to have permissions slice in state available directly as resources.jobs.foo.permissions[0]
+
+For that I propose to introduce a new convention: if json tag name is __EMBED__ then this is a sign for all struct walkers in libs/struct to not add this to path
+but to consider it directly added. We only need to support single __EMBED__ per struct and it may only work on slices.
+
+Make sure to commit early and often on this branch. You should probably have commits for each item in libs/structs/
+
+Investigate how tests are done there and prefer to extend existing table tests rather than creating new one.
+
+For each change follow style of the surrounding package religiously, unless you can do things simpler, than do simpler.
+
+After __EMBED__ is supported, you will use it on PermissionState.
+
+Note, we new a few kinds of acceptance tests: one where were reference a permission and another where we reference from permission.
+
+We also need tests for cases where slice index is coming from remote backend rather than from local, e.g. resources.jobs.foo.permissions[2] but [2] is not in the config but it is in the remote.
+
+
+Once everything is done, validate your work by enabling this test on direct engine: acceptance/bundle/apps/job_permissions/test.toml
+
+Do necessary fixes.
+
+Make sure "make generate" runs clean. Make sure make && make test-update succeeds cleanly. Run integration tests last, after everything else appears in order, since they are the slowest. Use make test-update-aws for those.
+
+As a first step, validate the above problem description and write detailed implementation PLAN.md. Commit that first.

TASKS/002.PLAN.md

@@ -0,0 +1,40 @@
+# Plan for Task 002: Refine __EMBED__ convention
+
+## Changes Required
+
+### 1. Replace `__EMBED__` json tag convention with `EmbeddedSlice` field name convention
+
+Instead of checking json tag for `__EMBED__`, struct walkers will check if field name is `EmbeddedSlice`.
+The json tag becomes normal (e.g., `json:"permissions,omitempty"`), so no custom JSON marshaling is needed.
+
+**PermissionsState becomes:**
+```go
+type PermissionsState struct {
+    ObjectID      string                     `json:"object_id"`
+    EmbeddedSlice []iam.AccessControlRequest `json:"permissions,omitempty"`
+}
+```
+
+**Files to update:**
+- `libs/structs/structtag/jsontag.go` — remove `EmbedTagName` constant and `IsEmbed()` method
+- `libs/structs/structtag/jsontag_test.go` — remove embed test cases
+- `libs/structs/structaccess/embed.go` — check field name `EmbeddedSlice` instead of json tag `__EMBED__`
+- `libs/structs/structaccess/get.go` — update embed field detection
+- `libs/structs/structaccess/set.go` — update embed field detection
+- `libs/structs/structaccess/typecheck.go` — update embed field detection
+- `libs/structs/structwalk/walk.go` — check field name instead of json tag
+- `libs/structs/structwalk/walktype.go` — check field name instead of json tag
+- `libs/structs/structdiff/diff.go` — check field name instead of json tag
+- `bundle/direct/dresources/permissions.go` — rename field, remove custom marshaling
+- All test files with `__EMBED__` test structs — rename field to `EmbeddedSlice`, use normal json tag
+
+### 2. Fix refschema: remove dot before `[*]`
+
+Current output: `resources.clusters.*.permissions.[*]`
+Expected output: `resources.clusters.*.permissions[*]`
+
+The dot is likely inserted by the walktype or refschema path building code when appending `[*]` to a path that already ends with a field name. Need to find where paths are joined and fix.
+
+### 3. Remove dual-type for PermissionsState
+
+Remove `permissionsStateJSON`, `MarshalJSON()`, `UnmarshalJSON()` since the json tag will be normal `"permissions"`.

TASKS/002.STATUS.md

@@ -0,0 +1,12 @@
+## Status: DONE
+
+### Completed
+- Replaced `__EMBED__` json tag convention with `EmbeddedSlice` field name convention
+- Struct walkers detect embedded slices by Go field name instead of json tag
+- Removed custom MarshalJSON/UnmarshalJSON and dual type (permissionsStateJSON)
+- Renamed PermissionsState.Permissions to PermissionsState.EmbeddedSlice
+- Fixed refschema output: `permissions.[*]` → `permissions[*]` (removed dot before bracket)
+- Fixed refschema output: `permissions.permissions[*]` → `permissions[*]` (no duplicate prefix)
+- All unit tests pass (4579 tests)
+- All acceptance tests pass including job_permissions for both engines
+- Rebased on latest main

TASKS/002.md

@@ -0,0 +1,35 @@
+things to change:
+
+1.
+
+in refschema you have:
+  +resources.clusters.*.permissions.[*]   iam.AccessControlRequest        ALL
+
+  I don't like the dot after permissions, it should be resources.clusters.*.permissions[*]
+
+
+2.
+
+ type PermissionsState struct {
++       ObjectID    string                     `json:"object_id"`
++       Permissions []iam.AccessControlRequest `json:"__EMBED__,omitempty"`
++}
++
++// permissionsStateJSON is the JSON representation of PermissionsState.
++// The __EMBED__ tag is a convention for struct walkers, but JSON serialization
++// uses "permissions" as the field name.
++type permissionsStateJSON struct {
+        ObjectID    string                     `json:"object_id"`
+        Permissions []iam.AccessControlRequest `json:"permissions,omitempty"`
+ }
+
+ I dont like that we have 2 types now. Stick with one. Furthermore, let's use empty string instread of __EMBED__. And let's use EmbeddedSlice instead of Permissions as field name
+ So EmbeddedSlice field name is the one that will giev it special meaning when traversing. The empty string will appear in plan/state as key.
+
+
+
+Things that look good, should be kept:
+
+       "changes": {
+-        "permissions[group_name='admin-team']": {
++        "[group_name='admin-team']": {

TASKS/004.PLAN.md

@@ -0,0 +1,21 @@
+## Plan for Task 004: Acceptance tests for permission references
+
+### Goal
+Create acceptance tests that demonstrate cross-resource references between permissions on two jobs:
+1. One reference using integer index: `${resources.jobs.job_b.permissions[0].level}`
+2. One reference using key-value syntax: `${resources.jobs.job_b.permissions[group_name='admin-team'].level}`
+
+### Test Design
+Create a new test directory: `acceptance/bundle/resource_deps/permission_ref/`
+
+**databricks.yml**: Two jobs with permissions:
+- `job_a`: permissions reference `job_b`'s permission level (both by index and by key-value)
+- `job_b`: has explicit permission entries that `job_a` references
+
+**script**: Deploy, verify that the resolved permission level appears in the job creation request for `job_a`.
+
+### Files to Create
+- `acceptance/bundle/resource_deps/permission_ref/databricks.yml`
+- `acceptance/bundle/resource_deps/permission_ref/script`
+
+Then run with `-update` to generate output files.

TASKS/004.STATUS.md

@@ -0,0 +1,13 @@
+## Status: DONE
+
+### Completed
+- Fixed reference resolution for permission sub-resources (splitResourcePath routes permissions[0].level to the permissions sub-resource node)
+- Added level→permission_level field remapping for both reference lookup and ref storage (input config uses "level" but state uses "permission_level")
+- Created acceptance test `acceptance/bundle/resource_deps/permission_ref/` demonstrating cross-resource permission level references
+- Test verifies: job_a's permission levels reference job_b's permission levels via `${resources.jobs.job_b.permissions[0].level}`
+- All unit tests pass (4595 tests)
+- All acceptance tests pass
+
+### Notes
+- Key-value syntax for references (e.g., `permissions[group_name='viewers'].level`) is not supported by the variable reference regex, which only allows integer indices. Only integer index references work.
+- Test is direct-engine only since terraform handles permissions as separate resources and doesn't support cross-resource permission references in the same way.

TASKS/004.md

@@ -0,0 +1,3 @@
+Create acceptance tests that show that references work. Two permissions on two jobs, one permission's level references another permission's level.
+
+One time do reference by integer index. Another time do reference by key-value syntaxx.

TASKS/005.PLAN.md

@@ -0,0 +1,25 @@
+## Plan for Task 005: Use bundle config Permission type in PermissionsState
+
+### Goal
+Replace `[]iam.AccessControlRequest` with `[]resources.Permission` in `PermissionsState`, eliminating the `level` → `permission_level` field remapping.
+
+### Changes
+
+1. **`bundle/direct/dresources/permissions.go`**:
+   - Change `EmbeddedSlice` type from `[]iam.AccessControlRequest` to `[]resources.Permission`
+   - Remove `toAccessControlRequests` — `PreparePermissionsInputConfig` can directly use `[]resources.Permission` from input config
+   - Update `accessControlRequestKey` to work with `resources.Permission`
+   - Update `DoRead` to convert API response → `[]resources.Permission`
+   - Update `DoUpdate` to convert `[]resources.Permission` → `[]iam.AccessControlRequest` for API call
+   - Update `PreparePermissionsInputConfig` to accept permissions directly without reflection conversion
+
+2. **`bundle/direct/bundle_plan.go`**:
+   - Remove `remapPermissionFieldPath` function
+   - Remove remapping call in `splitResourcePath`
+   - Remove remapping block around line 886-894
+
+3. **Update acceptance test outputs** if any paths change (they shouldn't since field is now `level` matching config)
+
+### Risks
+- Need to ensure `PreparePermissionsInputConfig` still works with various permission types (JobPermission, PipelinePermission, etc.) since they all have different Level types but share the same struct layout
+- The reflection-based `toAccessControlRequests` needs to be replaced with type-safe conversion

TASKS/005.STATUS.md

@@ -0,0 +1,12 @@
+## Status: DONE
+
+### Completed
+- Replaced `[]iam.AccessControlRequest` with `[]resources.Permission` in `PermissionsState`
+- Removed `toAccessControlRequests` reflection-based conversion
+- Updated `DoRead` to convert API response → `[]resources.Permission`
+- Updated `DoUpdate` to convert `[]resources.Permission` → `[]iam.AccessControlRequest`
+- Removed `remapPermissionFieldPath` function from `bundle_plan.go` (no longer needed since config and state both use `level`)
+- Updated refschema and acceptance test outputs for `permission_level` → `level` change
+- Fixed exhaustruct lint for Permission type
+- All unit tests pass
+- All acceptance tests pass

TASKS/005.md

@@ -0,0 +1,8 @@
+type PermissionsState struct {
+-       ObjectID    string                     `json:"object_id"`
+-       Permissions []iam.AccessControlRequest `json:"permissions,omitempty"`
++       ObjectID      string                     `json:"object_id"`
++       EmbeddedSlice []iam.AccessControlRequest `json:"_,omitempty"`
+ }
+
+instead of iam.AccessControlRequest use Permission type from bundle config so that we don't need to remap permission_level to level

TASKS/006.PLAN.md

@@ -0,0 +1,41 @@
+## Plan for Task 006: Backward-compatible migration of permission_level → level in state
+
+### Problem
+Task 005 changed `PermissionsState` from `[]iam.AccessControlRequest` (with `json:"permission_level"`) to `[]resources.Permission` (with `json:"level"`). Existing state files on disk still have `"permission_level"` as the JSON key. When deserializing old state, the `Level` field will be empty because `json.Unmarshal` doesn't match `"permission_level"` to `json:"level"`.
+
+### Approach
+Per task 006 instructions: "in our state struct (but not in config or remote), we should have both level and permission_level; when we see level is empty, we copy it from permission_level; then we reset permission_level to nil."
+
+1. **`bundle/direct/dresources/permissions.go`**:
+   - Add `PermissionLevel` field to `PermissionsState`'s embedded permission type (or use a state-specific permission struct with both `level` and `permission_level` JSON fields)
+   - Actually simplest: add a `PermissionLevel` field to `resources.Permission`... but that would affect config too. Instead, create a state-only type `StatePermission` that embeds `resources.Permission` and adds `PermissionLevel`.
+   - Alternatively: just add `PermissionLevel iam.PermissionLevel` with `json:"permission_level,omitempty"` directly to `PermissionsState` item type. Use a local struct.
+   - In `PrepareState`, migrate: for each permission, if `Level` is empty and `PermissionLevel` is not, copy → `Level`, clear `PermissionLevel`.
+
+2. **Acceptance test**: Create a test with old state format as fixture, deploy to trigger migration, verify new state format.
+
+### Implementation Detail
+Define a state-specific permission struct in permissions.go:
+```go
+type statePermission struct {
+    Level                iam.PermissionLevel `json:"level,omitempty"`
+    PermissionLevel      iam.PermissionLevel `json:"permission_level,omitempty"`
+    UserName             string              `json:"user_name,omitempty"`
+    ServicePrincipalName string              `json:"service_principal_name,omitempty"`
+    GroupName            string              `json:"group_name,omitempty"`
+}
+```
+
+Change `PermissionsState.EmbeddedSlice` to `[]statePermission`.
+
+In `PrepareState`:
+```go
+for i := range s.EmbeddedSlice {
+    if s.EmbeddedSlice[i].Level == "" && s.EmbeddedSlice[i].PermissionLevel != "" {
+        s.EmbeddedSlice[i].Level = s.EmbeddedSlice[i].PermissionLevel
+        s.EmbeddedSlice[i].PermissionLevel = ""
+    }
+}
+```
+
+Update `PreparePermissionsInputConfig` and `DoRead`/`DoUpdate` to use `statePermission` instead of `resources.Permission`.

TASKS/006.STATUS.md

@@ -0,0 +1,11 @@
+## Status: DONE
+
+### Completed
+- Added `StatePermission` type with both `level` and `permission_level` JSON fields for backward compatibility
+- Custom `UnmarshalJSON` on `PermissionsState` migrates old `permission_level` to `level` during deserialization
+- Migration clears `permission_level` after copying to `level`, so re-saved state uses only `level`
+- Updated refschema output to reflect new `StatePermission` type
+- Added acceptance test `acceptance/bundle/state/permission_level_migration/` verifying old state is migrated
+- All unit tests pass (4595 tests)
+- All acceptance tests pass
+- Lint, fmt, checks, generate all clean

TASKS/006.md

@@ -0,0 +1,3 @@
+we should not lose old permission_level setting. instead, in our state struct (but not in config or remote), we should have both level and permission_evel; when we see level is empty, we copy it  from permission_level; then we reset permission_level to nil ("" without forcesendfields) so that it's stored.
+
+you need to add an acceptance test with old state part of the fixture that is being migrated to new state.