Skip to content

Add sharded cloud acceptance test entry point#4743

Draft
shreyas-goenka wants to merge 1 commit intomainfrom
sharded-tests
Draft

Add sharded cloud acceptance test entry point#4743
shreyas-goenka wants to merge 1 commit intomainfrom
sharded-tests

Conversation

@shreyas-goenka
Copy link
Contributor

Summary

  • Add TestAcceptCloudSharded - a new sharded acceptance test entry point that reads TOTAL_SHARDS/SHARD_INDEX environment variables and uses FNV-1a hashing to deterministically assign tests to shards.
  • Each test (identified by its directory + env variant name) is assigned to exactly one shard, so all shards together cover the full suite with no overlap.
  • This is a separate code path from the existing TestAccept - the existing pipeline is untouched.

Test plan

  • Run TestAcceptCloudSharded locally with TOTAL_SHARDS=3 SHARD_INDEX=0 and verify only a subset of tests execute.
  • Run with all shard indices (0, 1, 2) and verify every test is covered exactly once across the three runs.
  • Verify TestAccept continues to work unchanged without the sharding env vars.

Add `TestAcceptCloudSharded` which reads `TOTAL_SHARDS` and
`SHARD_INDEX` environment variables to deterministically assign
acceptance tests to shards using FNV-1a hashing. This enables
parallel CI execution of cloud tests across multiple runners
without overlap or gaps.

The existing `TestAccept` code path is unchanged.

Co-authored-by: Isaac
@eng-dev-ecosystem-bot
Copy link
Collaborator

eng-dev-ecosystem-bot commented Mar 14, 2026

Commit: 704fcee

Run: 23092105663

Env 🪲​BUG 🟨​KNOWN 🔄​flaky 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
🪲​ aws linux 1 7 1 7 268 787 7:52
🪲​ aws windows 1 7 1 7 270 785 6:17
🪲​ aws-ucws linux 1 2 7 7 364 702 6:32
🪲​ aws-ucws windows 1 8 7 367 700 5:44
🪲​ azure linux 1 4 2 9 267 785 10:02
🪲​ azure windows 1 2 9 273 783 5:10
🪲​ azure-ucws linux 1 2 1 9 369 698 8:10
🪲​ azure-ucws windows 1 2 1 9 371 696 5:52
🪲​ gcp linux 1 2 9 267 788 6:12
🪲​ gcp windows 1 2 9 269 786 4:26
21 interesting tests: 7 KNOWN, 7 SKIP, 5 flaky, 1 RECOVERED, 1 BUG
Test Name aws linux aws windows aws-ucws linux aws-ucws windows azure linux azure windows azure-ucws linux azure-ucws windows gcp linux gcp windows
🟨​ TestAccept 🟨​K 🟨​K 🔄​f 💚​R 💚​R 💚​R 🔄​f 🔄​f 💚​R 💚​R
🔄​ TestAccept/bundle/deploy/files/no-snapshot-sync ✅​p ✅​p ✅​p ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p
🔄​ TestAccept/bundle/deploy/files/no-snapshot-sync/DATABRICKS_BUNDLE_ENGINE=terraform ✅​p ✅​p ✅​p ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p
🔄​ TestAccept/bundle/resources/apps/inline_config ✅​p ✅​p ✅​p ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p
🔄​ TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct ✅​p ✅​p ✅​p ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p
🙈​ TestAccept/bundle/resources/permissions 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions 🟨​K 🟨​K 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions 🟨​K 🟨​K 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K 🟨​K 💚​R 💚​R
🙈​ TestAccept/bundle/resources/postgres_branches/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/recreate 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/update_protected 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/without_branch_id 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_endpoints/recreate 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/synced_database_tables/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🔄​ TestAccept/ssh/connect-serverless-gpu 🙈​s 🙈​s 🔄​f ✅​p 🙈​s 🙈​s 🔄​f 🔄​f 🙈​s 🙈​s
💚​ TestAccept/ssh/connection 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🪲​ TestAcceptCloudSharded 🪲​B 🪲​B 🪲​B 🪲​B 🪲​B 🪲​B 🪲​B 🪲​B 🪲​B 🪲​B
Top 25 slowest tests (at least 2 minutes):
duration env testname
3:39 azure windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:21 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:10 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:07 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:05 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:57 gcp linux TestAccept/ssh/connection
2:55 aws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:44 aws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:44 aws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:42 aws-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:39 aws-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:39 azure linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:37 azure-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:36 aws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:35 azure-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:35 aws-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:34 aws-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:25 aws linux TestAccept/ssh/connection
2:17 azure linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:10 gcp linux TestSecretsPutSecretStringValue
2:10 azure-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:09 aws-ucws windows TestAccept/ssh/connection
2:07 azure-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:05 aws linux TestSecretsPutSecretStringValue
2:00 azure windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants