Skip to content

feat(coverage): rivet coverage --aggregate — file-based cross-repo V&V matrix (#188 sub-issue 3)#270

Draft
avrabe wants to merge 1 commit into
mainfrom
feat/issue-188-vv-matrix-aggregator
Draft

feat(coverage): rivet coverage --aggregate — file-based cross-repo V&V matrix (#188 sub-issue 3)#270
avrabe wants to merge 1 commit into
mainfrom
feat/issue-188-vv-matrix-aggregator

Conversation

@avrabe
Copy link
Copy Markdown
Contributor

@avrabe avrabe commented May 12, 2026

Summary

Picks up sub-issue 3 of the 2026-04-26 decomposition of #188 — the cross-repo aggregator — in the form that triage comment already endorsed:

walks an org (probably via a config file listing repos rather than GitHub API at first), composes the matrix.

This PR is the file-based form: each repo's CI runs rivet coverage --matrix --format json and uploads the result; a top-level job merges those files:

rivet coverage --aggregate rivet.json loom.json sigil.json ... --format markdown

No GitHub API access is needed, so it works inside the org-scoped triage agent and as an ordinary CI artifact-collection job.

Sub-issues 1 (vv-coverage schema, #232) and 2 (rivet coverage --matrix, #243) are already merged; this is the next in-scope chunk.

Behaviour

  • --aggregate <FILE>... (repeatable / multi-value) on rivet coverage. Implies matrix mode; the local project is not read. Conflicts with --tests at the clap layer.
  • Inputs are the JSON envelope render_matrix_json emits ({command, columns, repos:[{id, repo, techniques_applied, techniques_gated_in_ci, notes, cells}]}).
  • Output formats: text (default), markdown, html, json — the same renderers as --matrix.
  • Columns = sorted union of every input's techniques. Cell legend unchanged (· absent, applied, applied + CI-gated).
  • Duplicate (repo, id) rows are coalesced (first occurrence wins), so re-running the aggregator over overlapping inputs is idempotent.
  • The merged JSON re-feeds the aggregator unchanged (round-trips).
  • Bad / wrong-shaped input fails with a diagnostic naming the file (reading matrix JSON <path>, parsing matrix JSON <path>, <path>: ... missing 'repos' array).

#188 acceptance status

  • Extend rivet schema to capture technique-adoption status per repo — schemas/vv-coverage.yaml / repo-status (feat(schemas): vv-coverage — repo-status type for V&V technique tracking (#188) #232, merged).
  • New subcommand rivet coverage --matrix emits markdown / HTML / JSON — feat(coverage): rivet coverage --matrix — V&V matrix from repo-status artifacts #243, merged.
  • Aggregator that walks the pulseengine org and composes a cross-repo matrix (CI-friendly) — this PR, file-based form (--aggregate). The GitHub-org-API form, if still wanted, is a follow-up; the triage comment explicitly scoped the file-based version first.
  • Published dashboard (GitHub Pages / rivet-web component) — sub-issue 4, separate PR.
  • Integration: pulseengine.eu surfaces the live matrix — out of scope for a repo-scoped run.
  • Per-technique trend line — stretch.
  • Per-standard mapping view — stretch.

So this PR does not close #188; sub-issue 4 and the two stretch goals remain. Filed as draft for the maintainer's call on whether the file-based aggregator fully satisfies the "CI-friendly aggregator" bullet or whether a GitHub-API variant is also wanted.

Test plan

  • cargo test -p rivet-cli — green (all suites, incl. 3 new coverage_aggregate_* tests and the existing coverage_matrix_* tests).
  • cargo clippy -p rivet-cli --tests — clean.
  • cargo fmt -p rivet-cli --check — clean.
  • cargo run -p rivet-cli -- validate — 6 errors / 140 warnings, unchanged from origin/main (the 6 live in the spar: external fixture).
  • Manual smoke: merge two single-repo JSONs across all four formats; dedup on repeated input; re-feed merged JSON; bad-input diagnostics.
  • CI on this PR.

Note on process

Consulted https://pulseengine.eu/blog/ at the start of this run and again before opening this PR — no posts there describe branch/PR/test conventions beyond the spec-driven / oracle-gated workflow already followed (acceptance-criteria-as-oracle, one PR per issue, rivet validate gate, artifact trailers).

🤖 Generated with Claude Code — issue-triage agent run 2026-05-12.


Generated by Claude Code

…V matrix

Implements sub-issue 3 of the V&V coverage matrix decomposition on #188:
the cross-repo aggregator. Rather than reaching into the GitHub org API,
each repo's CI emits `rivet coverage --matrix --format json` and a
top-level job merges the files with `rivet coverage --aggregate a.json
b.json ... --format {text,markdown,html,json}`.

- Duplicate `(repo, id)` rows are coalesced (first wins) so re-running
  the aggregator over overlapping inputs is idempotent.
- The merged JSON uses the same envelope as the per-repo command, so it
  re-feeds the aggregator unchanged.
- Bad / wrong-shaped inputs fail with a diagnostic that names the file.

3 integration tests: markdown merge + column union, JSON round-trip +
dedup, bad-input diagnostics. Existing `coverage --matrix` tests, clippy,
fmt, and `rivet validate` baseline all unchanged.

Implements: REQ-007
Refs: FEAT-001
@github-actions
Copy link
Copy Markdown

📐 Rivet artifact delta

No artifact changes in this PR. Code-only changes (renderer, CLI wiring, tests) don't touch the artifact graph.

Copy link
Copy Markdown

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Rivet Criterion Benchmarks'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.

Benchmark suite Current: d311e5c Previous: 660a5b0 Ratio
store_insert/100 83378 ns/iter (± 7193) 64712 ns/iter (± 272) 1.29
store_insert/1000 895976 ns/iter (± 11989) 691021 ns/iter (± 20651) 1.30
store_lookup/100 1948 ns/iter (± 6) 1510 ns/iter (± 4) 1.29
store_lookup/1000 24586 ns/iter (± 1246) 18519 ns/iter (± 479) 1.33
store_lookup/10000 369381 ns/iter (± 2449) 273285 ns/iter (± 8003) 1.35
store_by_type/100 97 ns/iter (± 0) 75 ns/iter (± 1) 1.29
store_by_type/1000 97 ns/iter (± 0) 75 ns/iter (± 0) 1.29
store_by_type/10000 97 ns/iter (± 0) 75 ns/iter (± 1) 1.29
schema_load_and_merge 1174024 ns/iter (± 34717) 916366 ns/iter (± 3140) 1.28
link_graph_build/100 166628 ns/iter (± 3941) 125502 ns/iter (± 590) 1.33
link_graph_build/1000 1926533 ns/iter (± 42617) 1484186 ns/iter (± 19358) 1.30
validate/100 136550 ns/iter (± 463) 104408 ns/iter (± 955) 1.31
validate/1000 1267897 ns/iter (± 31943) 946010 ns/iter (± 4777) 1.34
traceability_matrix/100 4076 ns/iter (± 15) 3193 ns/iter (± 8) 1.28
traceability_matrix/1000 43501 ns/iter (± 682) 34612 ns/iter (± 140) 1.26
traceability_matrix/10000 762509 ns/iter (± 4673) 566932 ns/iter (± 6065) 1.34
diff/100 63049 ns/iter (± 880) 43810 ns/iter (± 183) 1.44
diff/1000 711265 ns/iter (± 3875) 487443 ns/iter (± 2726) 1.46
query/100 744 ns/iter (± 19) 570 ns/iter (± 1) 1.31
query/1000 6967 ns/iter (± 63) 4821 ns/iter (± 67) 1.45
query/10000 89850 ns/iter (± 413) 69315 ns/iter (± 421) 1.30
document_parse/10 22104 ns/iter (± 104) 16677 ns/iter (± 250) 1.33
document_parse/100 150587 ns/iter (± 9472) 115679 ns/iter (± 619) 1.30
document_parse/1000 1395775 ns/iter (± 8515) 1058032 ns/iter (± 19974) 1.32

This comment was automatically generated by workflow using github-action-benchmark.

@codecov
Copy link
Copy Markdown

codecov Bot commented May 12, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

V&V coverage matrix dashboard via rivet coverage (per-commit, per-repo, per-technique)

2 participants