Reduce flaky idempotency test timeout on Windows
#581
+8
−6
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The `idempotency` test was running **110 concurrent steps** (10 + 100), with each step requiring multiple filesystem operations in the local world: 1. **Step creation** - writes step JSON file 2. **Step update** - reads and overwrites step JSON file 3. **Event creation** - writes event JSON file That's ~3-4 file operations per step × 110 steps = **330-440 total file operations**. On Windows, this is especially slow because: - **NTFS** is slower than ext4/APFS for small file writes - **Windows Defender** real-time scanning adds latency to new file creation - The **atomic write pattern** (write temp file → rename) is slower on Windows Reduced the number of steps from **110 to 20** (5 + 15 instead of 10 + 100): **`workflows/noop.ts`**: - First batch: 10 → 5 steps - Second batch: 100 → 15 steps **`src/idempotency.mts`**: - Updated assertion to expect 20 numbers instead of 110 This is ~5.5x fewer file operations while still testing the same concurrent step execution and idempotency behavior. The test should now complete well within the 60-second timeout even on slow Windows CI machines.
🦋 Changeset detectedLatest commit: 0548aa4 The changes in this PR will be included in the next version bump. This PR includes changesets to release 0 packagesWhen changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
📊 Benchmark Results
workflow with no steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) workflow with 1 step💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) workflow with 10 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro workflow with 10 parallel steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) stress test: Promise.all with 100 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro stress test: Promise.race with 100 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro Stream Benchmarks (includes TTFB metrics)workflow with stream💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro SummaryFastest Framework by WorldWinner determined by most benchmark wins
Fastest World by FrameworkWinner determined by most benchmark wins
Column Definitions
Worlds:
❌ Some benchmark jobs failed:
Check the workflow run for details. |
Contributor
🧪 E2E Test Results❌ Some tests failed Summary
❌ Failed Tests🌍 Community Worlds (11 failed)mongodb (1 failed):
redis (1 failed):
starter (8 failed):
turso (1 failed):
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
❌ 🌍 Community Worlds
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR reduces the number of concurrent steps in the idempotency test from 110 to 20 to address timeout issues on Windows CI machines. The reduction accounts for Windows-specific file system performance characteristics (NTFS, Windows Defender scanning, atomic write patterns) while maintaining the same test coverage for concurrent step execution and idempotency behavior.
- Reduced first batch of steps from 10 to 5
- Reduced second batch of steps from 100 to 15
- Updated test assertion to expect 20 numbers instead of 110
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
packages/world-testing/workflows/noop.ts |
Reduces concurrent steps from 10+100 to 5+15, updates console logs and number ranges accordingly |
packages/world-testing/src/idempotency.mts |
Updates assertion to expect 20 numbers instead of 110 to match reduced step count |
.changeset/light-ducks-fly.md |
Adds empty changeset file that needs package declaration and description |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+1
to
+2
| --- | ||
| --- |
There was a problem hiding this comment.
The changeset file is empty and missing the required package declaration and description. Based on the changeset format in this repository, it should include:
- A package name with version bump type (e.g.,
"@workflow/world-testing": patch) - A description of the change
For example:
---
"@workflow/world-testing": patch
---
Reduce idempotency test timeout on Windows by decreasing concurrent steps from 110 to 20
Suggested change
| --- | |
| --- | |
| --- | |
| "@workflow/world-testing": patch | |
| --- | |
| Reduce idempotency test timeout on Windows by decreasing concurrent steps from 110 to 20 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
idempotencytest was running 110 concurrent steps (10 + 100), with each step requiring multiple filesystem operations in the local world:That's ~3-4 file operations per step × 110 steps = 330-440 total file operations.
On Windows, this is especially slow because:
Reduced the number of steps from 110 to 20 (5 + 15 instead of 10 + 100):
workflows/noop.ts:src/idempotency.mts:This is ~5.5x fewer file operations while still testing the same concurrent step execution and idempotency behavior. The test should now complete well within the 60-second timeout even on slow Windows CI machines.