Skip to content

Warm page cache before save to eliminate cold-NVMe IOPS bottleneck #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
joshfriend merged 1 commit into from
Apr 2, 2026
Merged

Warm page cache before save to eliminate cold-NVMe IOPS bottleneck #4

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions gradlecache/save.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import (
"context"
"fmt"
"io"
"io/fs"
"log/slog"
"os"
"os/exec"
Expand Down Expand Up @@ -124,6 +125,7 @@ type SaveConfig struct {
GitDir string
GradleUserHome string
IncludedBuilds []string
SkipWarm bool // skip page cache warming (for benchmarking cold baseline)
Metrics MetricsClient
Logger *slog.Logger
}
Expand Down Expand Up @@ -194,6 +196,16 @@ func Save(ctx context.Context, cfg SaveConfig) error {
pr, pw := io.Pipe()

log.Info("saving bundle", "commit", cfg.Commit[:min(8, len(cfg.Commit))], "cache-key", cfg.CacheKey)

if !cfg.SkipWarm {
log.Debug("warming page cache")
warmStart := time.Now()
warmPageCache(sources)
log.Debug("page cache warm", "duration", time.Since(warmStart).Round(time.Millisecond))
Comment on lines +200 to +204
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Honor context cancellation before warming files

Save now performs a full warmPageCache(sources) pass before starting CreateTarZstd, but this warm step does not observe ctx.Done(). In timeout/cancel scenarios (common in CI), Save will continue scanning and reading the entire cache tree before returning, which can add minutes of uninterruptible work after cancellation and delay job teardown. Please gate or short-circuit warming when the context is canceled.

Useful? React with 👍 / 👎.

} else {
log.Debug("skipping page cache warm (SkipWarm=true)")
}

saveStart := time.Now()

// Wrap the archive→upload boundary to measure upload wait time.
Expand Down Expand Up @@ -509,6 +521,40 @@ func matchesAny(name string, patterns []string) bool {
return false
}

// warmPageCache reads every regular file under each TarSource in parallel,
// faulting pages into the OS page cache before tar reads them sequentially.
// On cold NVMe storage with many small files (e.g. 200K Gradle cache entries),
// tar is limited to ~80 MB/s by per-file IOPS overhead. Warming the cache with
// parallel readers saturates IOPS up front so that tar subsequently reads at
// memory speed (~1300 MB/s).
func warmPageCache(sources []TarSource) {
concurrency := min(runtime.GOMAXPROCS(0)*2, 32)
sem := make(chan struct{}, concurrency)
var wg sync.WaitGroup

for _, src := range sources {
root := filepath.Join(src.BaseDir, src.Path)
_ = filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
if err != nil || !d.Type().IsRegular() {
return nil
Comment on lines +537 to +539
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Skip warming files excluded from the archive

warmPageCache reads every regular file under each source, but CreateTarZstd later excludes many paths (including wrapper/dists/*/*/*.zip and CacheExclusions). This means we spend I/O warming data that will never be archived, and large excluded files can evict useful cache pages and reduce the intended speedup. The warm walk should apply the same exclusion rules as tar input selection.

Useful? React with 👍 / 👎.

}
sem <- struct{}{}
wg.Add(1)
go func() {
defer func() { <-sem; wg.Done() }()
f, err := os.Open(path)
if err != nil {
return
}
_, _ = io.Copy(io.Discard, f)
_ = f.Close()
}()
return nil
})
}
wg.Wait()
}

// CreateTarZstd creates a zstd-compressed tar archive from the given sources.
// If pzstd is available it is used to produce a multi-frame archive that can
// be decompressed in parallel on restore. Otherwise klauspost is used.
Expand Down
Loading