fix: retry tests on transient GCP errors instead of failing

[cfsmp3]

Summary

• Make tests retry automatically when hitting transient GCP errors (QUOTA_EXCEEDED, ZONE_RESOURCE_POOL_EXHAUSTED)
• Only permanent errors mark tests as failed

Problem

When GCP quota is exceeded, tests were marked as failed with an error on GitHub:

GCP quota limit reached. Please wait for other tests to complete or contact the administrator.

This was confusing because:

1. The message said "wait" but the test was already permanently failed
2. Users saw a red X on their PR for infrastructure issues, not code problems
3. Tests wouldn't automatically retry when resources became available

Solution

Transient GCP errors now leave the test pending instead of failed:

• Test remains in queue without a GcpInstance record
• Next cron run will attempt to start it again
• When resources are available, the test runs normally

Permanent errors (like RESOURCE_NOT_FOUND) still immediately fail.

Changes

• Added GCP_RETRYABLE_ERRORS set defining transient error codes
• Added get_gcp_error_code() and is_retryable_gcp_error() helpers
• Modified start_test() to skip mark_test_failed() for retryable errors
• Updated error message for QUOTA_EXCEEDED to match actual behavior
• Updated tests to verify new behavior

Test plan

• Updated existing test to verify QUOTA_EXCEEDED doesn't mark test as failed
• Added new test to verify RESOURCE_NOT_FOUND still marks test as failed
• CI passes

🤖 Generated with Claude Code

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud