Skip to content

feat: VCOutcome uses unknown instead of sat for unvalidated SMT results#722

Merged
MikaelMayer merged 40 commits intomainfrom
issue-721-vcoutcome-should-use-unknown-instead-of
Apr 3, 2026
Merged

feat: VCOutcome uses unknown instead of sat for unvalidated SMT results#722
MikaelMayer merged 40 commits intomainfrom
issue-721-vcoutcome-should-use-unknown-instead-of

Conversation

@MikaelMayer
Copy link
Copy Markdown
Contributor

@MikaelMayer MikaelMayer commented Mar 31, 2026

When the SMT solver returns "sat", it does not necessarily mean the property is truly satisfiable — uninterpreted functions may admit spurious models. This PR makes VCOutcome sound by treating unvalidated SMT "sat" results as "unknown" until counter-example validation is implemented.

Changes:

  • Result.unknown now carries an optional candidate model (candidateModel : Option (CounterEx Ident)), distinguishing between an empty model (some []) and no model available (none). This allows unknown results to preserve counterexample data for future validation while clarifying that the model is unverified.
  • SolverPhaseLog is a structure with phase : String and result : SMT.Result fields, so each log entry is labeled with the pipeline phase that produced it. VCOutcome gains a solverLog : List SolverPhaseLog field — an ordered list recording both the raw solver results (labeled solver.sat / solver.val) and per-phase adjusted results, for future diagnostic and traceability tooling.
  • In the verification pipeline, every SMT "sat" result that we can't reliably validate is converted to "unknown" (with the model attached). The ways we know a SMT "sat" is preserved is that
    • There was no procedure eliminated from the entry of the procedure to the assertion point
    • There was no loop eliminated from the entry of the procedure to the assertion point.
    • For front-end, by default "sat" is not preserved until they implement a validation procedure, as front-ends might encode functions with uninterpreted functions.
  • PipelinePhase structure couples each program-to-program transformation with its AbstractedPhase (model validation), ensuring transforms and their soundness annotations stay in sync. Each AbstractedPhase carries a name field that flows into the solver log. corePipelinePhases is the single parameterized function that drives both the pipeline and the validation — it accepts the target procedure list and factory, and returns the full pipeline including filtering, call/loop elimination, precondition elimination, and the final keep-set filter. coreAbstractedPhases is derived from it via .map (.phase).
  • All filter phases (FilterProcedures, keep-set filter) and PrecondElim are explicit pipeline phases marked as model-preserving, since they only remove procedures or add new assertions without introducing over-approximations.
  • Pipeline phase types (ModelValidation, AbstractedPhase, PipelinePhase, obligationHasLabelPrefix, modelPreservingPipelinePhase) are defined in Strata/Languages/Core/PipelinePhase.lean. Each transform pass defines its own pipeline phase next to its implementation: callElimPipelinePhase in CallElim.lean, loopElimPipelinePhase in LoopElim.lean, filterProceduresPipelinePhase in FilterProcedures.lean, and precondElimPipelinePhase in PrecondElim.lean.
  • Label prefixes used by call elimination (callElimAssertPrefix, callElimAssumePrefix) are defined in CallElim.lean. createAsserts/createAssumes accept a labelPrefix parameter, decoupling the generic helpers from the call-elimination-specific prefixes.
  • loopElimPipelinePhase is placed last in the pipeline because loop elimination happens during evaluation (not as a program-to-program pass), making it the closest phase to SMT.
  • Phase validation is non-cascading: each phase with modelToValidate independently validates the model, regardless of whether the current result is .sat or .unknown. A phase can demote .sat m to .unknown (some m) (when the model fails validation) or promote .unknown (some m) back to .sat m (when the model passes validation against the pre-phase semantics). Phases without a validator (modelPreserving) pass results through unchanged. When no candidate model is available (.unknown none), validation is skipped.
  • In bug-finding modes (bugFinding and bugFindingAssumingCompleteSpec), unreachable assertions are now errors instead of warnings. Dead code in bug-finding contexts signals a specification or implementation problem. Added docs/VerificationModes.md documenting the three verification modes and their error classification, referenced from the README (in the "Running Analyses" section, after lake exe strata verify is introduced).
  • Extracted unreachableMsg helper to deduplicate the check-mode/property match pattern in VCOutcome.label and VCOutcome.emoji.
  • Per-pass obligation tests live next to their transforms: StrataTest/Transform/CallElim.lean and StrataTest/Transform/LoopElim.lean.

Test expectations updated to reflect the new behavior. Note that in many cases in Core, we keep "sat" unchanged. It's only changing what the front-end receive.

Fixes #721

…ts (#721)

- Add optional model parameter to Result.unknown for carrying
  counterexample data from inconclusive solver results
- Add SolverPhaseLog structure and solverLog field to VCOutcome to
  preserve raw solver output before soundness adjustments
- Convert SMT sat results to unknown (with model) in the verification
  pipeline since uninterpreted functions may admit spurious models
- PE-determined results remain unchanged (they are sound)
- Update test expectations to reflect the new behavior
Comment thread Strata/Languages/Core/Verifier.lean
Comment thread Strata/Languages/Core/Verifier.lean Outdated
Comment thread StrataTest/Languages/Core/Examples/Cover.lean Outdated
Comment thread StrataTest/Languages/Core/Examples/Cover.lean Outdated
- Change SolverPhaseLog to store one SMT.Result per entry
- Add Program.hasOverApproximation to detect uninterpreted functions
- Conditionally apply satToUnknown only when over-approximations exist
- Revert test expectations for programs without uninterpreted functions
- Keep sat→unknown for programs with uninterpreted functions (Map, consts, etc.)
@MikaelMayer

This comment was marked as resolved.

BooleanQuantification.bpl and Unique.bpl have uninterpreted functions/types,
so hasOverApproximation is true and sat results correctly become unknown.
Comment thread Strata/Languages/Core/Verifier.lean Outdated
@MikaelMayer

This comment was marked as resolved.

@MikaelMayer

This comment was marked as outdated.

Comment thread Strata/Languages/Core/Verifier.lean Outdated
Comment thread Strata/Languages/Core/Verifier.lean Outdated
MikaelMayer

This comment was marked as resolved.

…ix PE path

- Remove unused 'name' field from AbstractedPhase
- Move needsValidation to test file (only used there)
- Add docstring note about solverLog's intended consumer
- Extract buildSolverLog helper to deduplicate SMT/PE paths
- Skip adjustForPhases in PE path (PE results are sound by construction)
Comment thread Strata/Languages/Core/Verifier.lean Outdated
…front-end phases

AbstractedPhase now determines model validation per-obligation by inspecting
path condition labels. Core pipeline phases (callElimPhase, loopElimPhase)
check whether the obligation's path includes labels from their respective
transforms. Bodiless functions in Core are no longer treated as Core
over-approximations — that responsibility belongs to the front-end phase.

Front-ends (Boole, C_Simp, Laurel, Python) pass a blanket frontEndPhase
that converts all SAT to UNKNOWN until model validation is available.
Core.verify and Strata.verify accept an externalPhases parameter.

Tests updated for the new per-obligation precision: obligations unaffected
by abstractions now show definitive results instead of unknown.
…rt test expectations

The previous commit added frontEndPhase to all front-end languages, but
this is too aggressive — it converts all sat results to unknown, breaking
tests that expect definite failures. The frontEndPhase should only be
used by Python (via StrataMain) where tests already expect this behavior.

Also reverts Map.lean, T14_Quantifiers, and T19_InvokeOn expected outputs
back to main's values, since the per-obligation phase system does not
convert sat→unknown for const declarations or uninterpreted functions.
… label prefixes

- Move Verification Modes reference in README to after 'lake exe strata verify'
- Move callElimAssumePrefix to CallElim.lean, add callElimAssertPrefix
- Parameterize createAsserts/createAssumes with labelPrefix argument
- Create PipelinePhase.lean with ModelValidation, AbstractedPhase, PipelinePhase,
  obligationHasLabelPrefix, and modelPreservingPipelinePhase
- Move callElimPipelinePhase to CallElim.lean
- Move loopElimPipelinePhase to LoopElim.lean
- Verifier.lean imports these from the pass files
Comment thread Strata/Languages/Core/PipelinePhase.lean
Comment thread Strata/Transform/LoopElim.lean Outdated
Comment thread StrataTest/Languages/Core/VCOutcomeTests.lean
Comment thread Strata/Languages/Core/Verifier.lean Outdated
Comment thread Strata/Languages/Core/Verifier.lean Outdated
Comment thread Strata/Languages/Core/Verifier.lean Outdated
Comment thread Strata/Languages/Core/Verifier.lean
Per review feedback from aqjune-aws and shigoel: rename the model field
in Result.unknown from 'model' to 'candidateModel' and change its type
from 'CounterEx Ident' to 'Option (CounterEx Ident)'. This distinguishes
between an empty model [] and no model available (none).
… their files

- Combine redundant end/public section in LoopElim.lean
- Move per-pass obligation tests to StrataTest/Transform/{CallElim,LoopElim}.lean
- Extract unreachableMsg helper for repeated match pattern in VCOutcome
- Move filterProceduresPipelinePhase to FilterProcedures.lean
- Move precondElimPipelinePhase to PrecondElim.lean
Use loopElimInvariantPrefix and loopElimGuardPrefix constants in
the restructured guard-specific loop elimination code from main.
SolverPhaseLog is now a structure with phase (name) and result fields
instead of a plain SMT.Result alias. Each AbstractedPhase carries a
name field that flows into the log, so consumers can identify which
pipeline phase produced each entry.

Phase names: FilterProcedures, CallElim, PrecondElim, KeepSetFilter,
LoopElim, FrontEnd. Raw SMT results use SMT.sat / SMT.val.
@MikaelMayer MikaelMayer enabled auto-merge April 2, 2026 21:11
aqjune-aws
aqjune-aws previously approved these changes Apr 2, 2026
Copy link
Copy Markdown
Contributor

@aqjune-aws aqjune-aws left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Adding the name field to AbstractedPhase seems to make sense to me. :)

atomb
atomb previously approved these changes Apr 2, 2026
Comment thread Strata/Transform/CallElim.lean
Comment thread Strata/Transform/LoopElim.lean
Comment thread Strata/Languages/Core/Verifier.lean
shigoel
shigoel previously approved these changes Apr 3, 2026
Use main's Core.verifyProgram API with externalPhases parameter from this branch.
@MikaelMayer MikaelMayer dismissed stale reviews from shigoel, atomb, and aqjune-aws via 23617b7 April 3, 2026 02:51
Comment thread Strata/Languages/Core/Verifier.lean
Comment thread Strata/Languages/Core/Verifier.lean
Comment thread Strata/Languages/Core/Verifier.lean Outdated
Comment thread Strata/Languages/Core/Verifier.lean
Comment thread Strata/Languages/Core/Verifier.lean
Comment thread Strata/Languages/Core/PipelinePhase.lean
@MikaelMayer MikaelMayer added this pull request to the merge queue Apr 3, 2026
Merged via the queue into main with commit 2a64a51 Apr 3, 2026
15 checks passed
@MikaelMayer MikaelMayer deleted the issue-721-vcoutcome-should-use-unknown-instead-of branch April 3, 2026 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

VCOutcome should use "unknown" instead of "sat" until we have proper counter-example validation

5 participants