feat(coverage): rivet coverage --aggregate — file-based cross-repo V&V matrix (#188 sub-issue 3)#270
Draft
avrabe wants to merge 1 commit into
Draft
feat(coverage): rivet coverage --aggregate — file-based cross-repo V&V matrix (#188 sub-issue 3)#270avrabe wants to merge 1 commit into
avrabe wants to merge 1 commit into
Conversation
…V matrix Implements sub-issue 3 of the V&V coverage matrix decomposition on #188: the cross-repo aggregator. Rather than reaching into the GitHub org API, each repo's CI emits `rivet coverage --matrix --format json` and a top-level job merges the files with `rivet coverage --aggregate a.json b.json ... --format {text,markdown,html,json}`. - Duplicate `(repo, id)` rows are coalesced (first wins) so re-running the aggregator over overlapping inputs is idempotent. - The merged JSON uses the same envelope as the per-repo command, so it re-feeds the aggregator unchanged. - Bad / wrong-shaped inputs fail with a diagnostic that names the file. 3 integration tests: markdown merge + column union, JSON round-trip + dedup, bad-input diagnostics. Existing `coverage --matrix` tests, clippy, fmt, and `rivet validate` baseline all unchanged. Implements: REQ-007 Refs: FEAT-001
7 tasks
📐 Rivet artifact deltaNo artifact changes in this PR. Code-only changes (renderer, CLI wiring, tests) don't touch the artifact graph. |
There was a problem hiding this comment.
⚠️ Performance Alert ⚠️
Possible performance regression was detected for benchmark 'Rivet Criterion Benchmarks'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.
| Benchmark suite | Current: d311e5c | Previous: 660a5b0 | Ratio |
|---|---|---|---|
store_insert/100 |
83378 ns/iter (± 7193) |
64712 ns/iter (± 272) |
1.29 |
store_insert/1000 |
895976 ns/iter (± 11989) |
691021 ns/iter (± 20651) |
1.30 |
store_lookup/100 |
1948 ns/iter (± 6) |
1510 ns/iter (± 4) |
1.29 |
store_lookup/1000 |
24586 ns/iter (± 1246) |
18519 ns/iter (± 479) |
1.33 |
store_lookup/10000 |
369381 ns/iter (± 2449) |
273285 ns/iter (± 8003) |
1.35 |
store_by_type/100 |
97 ns/iter (± 0) |
75 ns/iter (± 1) |
1.29 |
store_by_type/1000 |
97 ns/iter (± 0) |
75 ns/iter (± 0) |
1.29 |
store_by_type/10000 |
97 ns/iter (± 0) |
75 ns/iter (± 1) |
1.29 |
schema_load_and_merge |
1174024 ns/iter (± 34717) |
916366 ns/iter (± 3140) |
1.28 |
link_graph_build/100 |
166628 ns/iter (± 3941) |
125502 ns/iter (± 590) |
1.33 |
link_graph_build/1000 |
1926533 ns/iter (± 42617) |
1484186 ns/iter (± 19358) |
1.30 |
validate/100 |
136550 ns/iter (± 463) |
104408 ns/iter (± 955) |
1.31 |
validate/1000 |
1267897 ns/iter (± 31943) |
946010 ns/iter (± 4777) |
1.34 |
traceability_matrix/100 |
4076 ns/iter (± 15) |
3193 ns/iter (± 8) |
1.28 |
traceability_matrix/1000 |
43501 ns/iter (± 682) |
34612 ns/iter (± 140) |
1.26 |
traceability_matrix/10000 |
762509 ns/iter (± 4673) |
566932 ns/iter (± 6065) |
1.34 |
diff/100 |
63049 ns/iter (± 880) |
43810 ns/iter (± 183) |
1.44 |
diff/1000 |
711265 ns/iter (± 3875) |
487443 ns/iter (± 2726) |
1.46 |
query/100 |
744 ns/iter (± 19) |
570 ns/iter (± 1) |
1.31 |
query/1000 |
6967 ns/iter (± 63) |
4821 ns/iter (± 67) |
1.45 |
query/10000 |
89850 ns/iter (± 413) |
69315 ns/iter (± 421) |
1.30 |
document_parse/10 |
22104 ns/iter (± 104) |
16677 ns/iter (± 250) |
1.33 |
document_parse/100 |
150587 ns/iter (± 9472) |
115679 ns/iter (± 619) |
1.30 |
document_parse/1000 |
1395775 ns/iter (± 8515) |
1058032 ns/iter (± 19974) |
1.32 |
This comment was automatically generated by workflow using github-action-benchmark.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Picks up sub-issue 3 of the 2026-04-26 decomposition of #188 — the cross-repo aggregator — in the form that triage comment already endorsed:
This PR is the file-based form: each repo's CI runs
rivet coverage --matrix --format jsonand uploads the result; a top-level job merges those files:No GitHub API access is needed, so it works inside the org-scoped triage agent and as an ordinary CI artifact-collection job.
Sub-issues 1 (
vv-coverageschema, #232) and 2 (rivet coverage --matrix, #243) are already merged; this is the next in-scope chunk.Behaviour
--aggregate <FILE>...(repeatable / multi-value) onrivet coverage. Implies matrix mode; the local project is not read. Conflicts with--testsat the clap layer.render_matrix_jsonemits ({command, columns, repos:[{id, repo, techniques_applied, techniques_gated_in_ci, notes, cells}]}).text(default),markdown,html,json— the same renderers as--matrix.·absent,○applied,●applied + CI-gated).(repo, id)rows are coalesced (first occurrence wins), so re-running the aggregator over overlapping inputs is idempotent.reading matrix JSON <path>,parsing matrix JSON <path>,<path>: ... missing 'repos' array).#188 acceptance status
rivetschema to capture technique-adoption status per repo —schemas/vv-coverage.yaml/repo-status(feat(schemas): vv-coverage — repo-status type for V&V technique tracking (#188) #232, merged).rivet coverage --matrixemits markdown / HTML / JSON — feat(coverage): rivet coverage --matrix — V&V matrix from repo-status artifacts #243, merged.--aggregate). The GitHub-org-API form, if still wanted, is a follow-up; the triage comment explicitly scoped the file-based version first.So this PR does not close #188; sub-issue 4 and the two stretch goals remain. Filed as draft for the maintainer's call on whether the file-based aggregator fully satisfies the "CI-friendly aggregator" bullet or whether a GitHub-API variant is also wanted.
Test plan
cargo test -p rivet-cli— green (all suites, incl. 3 newcoverage_aggregate_*tests and the existingcoverage_matrix_*tests).cargo clippy -p rivet-cli --tests— clean.cargo fmt -p rivet-cli --check— clean.cargo run -p rivet-cli -- validate— 6 errors / 140 warnings, unchanged fromorigin/main(the 6 live in thespar:external fixture).Note on process
Consulted https://pulseengine.eu/blog/ at the start of this run and again before opening this PR — no posts there describe branch/PR/test conventions beyond the spec-driven / oracle-gated workflow already followed (acceptance-criteria-as-oracle, one PR per issue,
rivet validategate, artifact trailers).🤖 Generated with Claude Code — issue-triage agent run 2026-05-12.
Generated by Claude Code