Skip to content

Add cache size limit for S3 uploads#23

Open
dmonroy wants to merge 1 commit intomainfrom
darwin/dont-push-large-cache-directories
Open

Add cache size limit for S3 uploads#23
dmonroy wants to merge 1 commit intomainfrom
darwin/dont-push-large-cache-directories

Conversation

@dmonroy
Copy link

@dmonroy dmonroy commented Jan 7, 2026

Problem: Build caches can grow very large. Uploading caches over 7GB to S3 wastes time and storage.

Solution: Check cache directory size before uploading. Skip the upload if it exceeds the limit.

  • Default limit: 7GB
  • Set APPPACK_MAX_CACHE_SIZE_GB=0 to disable the check
  • Set APPPACK_MAX_CACHE_SIZE_GB=10 for a 10GB limit
  • Invalid values (negative, non-numeric) fall back to 7GB with a warning

Changes:

  • build/build.go: Added dirSize(), getMaxCacheSizeGB(), updated archiveCache()
  • build/build_test.go: Tests for env var parsing and directory size calculation

Testing:

go test ./build/...

All tests pass.

Skip uploading build cache to S3 when it exceeds the size limit.
Configurable via APPPACK_MAX_CACHE_SIZE_GB environment variable
(default: 7GB). Set to 0 to disable the limit entirely.

Invalid or negative values fall back to the default with a warning.
if val := os.Getenv(MaxCacheSizeEnvVar); val != "" {
parsed, err := strconv.Atoi(val)
if err != nil {
fmt.Printf("WARNING: Invalid %s value '%s', using default %dGB\n", MaxCacheSizeEnvVar, val, DefaultMaxCacheSizeGB)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use the logging system here instead of print?

b.Log().Warn().Err(err).Msg("failed to calculate cache directory size")
} else if size > maxSize {
sizeMB := size / (1024 * 1024)
fmt.Printf("WARNING: Cache directory is %dMB, exceeding %dGB limit. Skipping cache upload.\n", sizeMB, maxGB)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need for print + log. Just log

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants