feat: VCOutcome uses unknown instead of sat for unvalidated SMT results#722
Merged
MikaelMayer merged 40 commits intomainfrom Apr 3, 2026
Merged
Conversation
…ts (#721) - Add optional model parameter to Result.unknown for carrying counterexample data from inconclusive solver results - Add SolverPhaseLog structure and solverLog field to VCOutcome to preserve raw solver output before soundness adjustments - Convert SMT sat results to unknown (with model) in the verification pipeline since uninterpreted functions may admit spurious models - PE-determined results remain unchanged (they are sound) - Update test expectations to reflect the new behavior
MikaelMayer
commented
Mar 31, 2026
MikaelMayer
commented
Mar 31, 2026
MikaelMayer
commented
Mar 31, 2026
MikaelMayer
commented
Mar 31, 2026
- Change SolverPhaseLog to store one SMT.Result per entry - Add Program.hasOverApproximation to detect uninterpreted functions - Conditionally apply satToUnknown only when over-approximations exist - Revert test expectations for programs without uninterpreted functions - Keep sat→unknown for programs with uninterpreted functions (Map, consts, etc.)
This comment was marked as resolved.
This comment was marked as resolved.
BooleanQuantification.bpl and Unique.bpl have uninterpreted functions/types, so hasOverApproximation is true and sat results correctly become unknown.
MikaelMayer
commented
Mar 31, 2026
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as outdated.
This comment was marked as outdated.
MikaelMayer
commented
Apr 1, 2026
MikaelMayer
commented
Apr 1, 2026
…ix PE path - Remove unused 'name' field from AbstractedPhase - Move needsValidation to test file (only used there) - Add docstring note about solverLog's intended consumer - Extract buildSolverLog helper to deduplicate SMT/PE paths - Skip adjustForPhases in PE path (PE results are sound by construction)
MikaelMayer
commented
Apr 1, 2026
…front-end phases AbstractedPhase now determines model validation per-obligation by inspecting path condition labels. Core pipeline phases (callElimPhase, loopElimPhase) check whether the obligation's path includes labels from their respective transforms. Bodiless functions in Core are no longer treated as Core over-approximations — that responsibility belongs to the front-end phase. Front-ends (Boole, C_Simp, Laurel, Python) pass a blanket frontEndPhase that converts all SAT to UNKNOWN until model validation is available. Core.verify and Strata.verify accept an externalPhases parameter. Tests updated for the new per-obligation precision: obligations unaffected by abstractions now show definitive results instead of unknown.
…rt test expectations The previous commit added frontEndPhase to all front-end languages, but this is too aggressive — it converts all sat results to unknown, breaking tests that expect definite failures. The frontEndPhase should only be used by Python (via StrataMain) where tests already expect this behavior. Also reverts Map.lean, T14_Quantifiers, and T19_InvokeOn expected outputs back to main's values, since the per-obligation phase system does not convert sat→unknown for const declarations or uninterpreted functions.
… label prefixes - Move Verification Modes reference in README to after 'lake exe strata verify' - Move callElimAssumePrefix to CallElim.lean, add callElimAssertPrefix - Parameterize createAsserts/createAssumes with labelPrefix argument - Create PipelinePhase.lean with ModelValidation, AbstractedPhase, PipelinePhase, obligationHasLabelPrefix, and modelPreservingPipelinePhase - Move callElimPipelinePhase to CallElim.lean - Move loopElimPipelinePhase to LoopElim.lean - Verifier.lean imports these from the pass files
shigoel
reviewed
Apr 2, 2026
aqjune-aws
reviewed
Apr 2, 2026
Per review feedback from aqjune-aws and shigoel: rename the model field in Result.unknown from 'model' to 'candidateModel' and change its type from 'CounterEx Ident' to 'Option (CounterEx Ident)'. This distinguishes between an empty model [] and no model available (none).
… their files
- Combine redundant end/public section in LoopElim.lean
- Move per-pass obligation tests to StrataTest/Transform/{CallElim,LoopElim}.lean
- Extract unreachableMsg helper for repeated match pattern in VCOutcome
- Move filterProceduresPipelinePhase to FilterProcedures.lean
- Move precondElimPipelinePhase to PrecondElim.lean
Use loopElimInvariantPrefix and loopElimGuardPrefix constants in the restructured guard-specific loop elimination code from main.
SolverPhaseLog is now a structure with phase (name) and result fields instead of a plain SMT.Result alias. Each AbstractedPhase carries a name field that flows into the log, so consumers can identify which pipeline phase produced each entry. Phase names: FilterProcedures, CallElim, PrecondElim, KeepSetFilter, LoopElim, FrontEnd. Raw SMT results use SMT.sat / SMT.val.
aqjune-aws
previously approved these changes
Apr 2, 2026
Contributor
aqjune-aws
left a comment
There was a problem hiding this comment.
Thanks. Adding the name field to AbstractedPhase seems to make sense to me. :)
atomb
previously approved these changes
Apr 2, 2026
shigoel
previously approved these changes
Apr 3, 2026
Use main's Core.verifyProgram API with externalPhases parameter from this branch.
23617b7
tautschnig
reviewed
Apr 3, 2026
…rename phase labels
tautschnig
approved these changes
Apr 3, 2026
This was referenced Apr 3, 2026
shigoel
reviewed
Apr 3, 2026
shigoel
approved these changes
Apr 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When the SMT solver returns "sat", it does not necessarily mean the property is truly satisfiable — uninterpreted functions may admit spurious models. This PR makes VCOutcome sound by treating unvalidated SMT "sat" results as "unknown" until counter-example validation is implemented.
Changes:
Result.unknownnow carries an optional candidate model (candidateModel : Option (CounterEx Ident)), distinguishing between an empty model (some []) and no model available (none). This allows unknown results to preserve counterexample data for future validation while clarifying that the model is unverified.SolverPhaseLogis a structure withphase : Stringandresult : SMT.Resultfields, so each log entry is labeled with the pipeline phase that produced it.VCOutcomegains asolverLog : List SolverPhaseLogfield — an ordered list recording both the raw solver results (labeledsolver.sat/solver.val) and per-phase adjusted results, for future diagnostic and traceability tooling.PipelinePhasestructure couples each program-to-program transformation with itsAbstractedPhase(model validation), ensuring transforms and their soundness annotations stay in sync. EachAbstractedPhasecarries anamefield that flows into the solver log.corePipelinePhasesis the single parameterized function that drives both the pipeline and the validation — it accepts the target procedure list and factory, and returns the full pipeline including filtering, call/loop elimination, precondition elimination, and the final keep-set filter.coreAbstractedPhasesis derived from it via.map (.phase).ModelValidation,AbstractedPhase,PipelinePhase,obligationHasLabelPrefix,modelPreservingPipelinePhase) are defined inStrata/Languages/Core/PipelinePhase.lean. Each transform pass defines its own pipeline phase next to its implementation:callElimPipelinePhaseinCallElim.lean,loopElimPipelinePhaseinLoopElim.lean,filterProceduresPipelinePhaseinFilterProcedures.lean, andprecondElimPipelinePhaseinPrecondElim.lean.callElimAssertPrefix,callElimAssumePrefix) are defined inCallElim.lean.createAsserts/createAssumesaccept alabelPrefixparameter, decoupling the generic helpers from the call-elimination-specific prefixes.loopElimPipelinePhaseis placed last in the pipeline because loop elimination happens during evaluation (not as a program-to-program pass), making it the closest phase to SMT.modelToValidateindependently validates the model, regardless of whether the current result is.sator.unknown. A phase can demote.sat mto.unknown (some m)(when the model fails validation) or promote.unknown (some m)back to.sat m(when the model passes validation against the pre-phase semantics). Phases without a validator (modelPreserving) pass results through unchanged. When no candidate model is available (.unknown none), validation is skipped.bugFindingandbugFindingAssumingCompleteSpec), unreachable assertions are now errors instead of warnings. Dead code in bug-finding contexts signals a specification or implementation problem. Addeddocs/VerificationModes.mddocumenting the three verification modes and their error classification, referenced from the README (in the "Running Analyses" section, afterlake exe strata verifyis introduced).unreachableMsghelper to deduplicate the check-mode/property match pattern inVCOutcome.labelandVCOutcome.emoji.StrataTest/Transform/CallElim.leanandStrataTest/Transform/LoopElim.lean.Test expectations updated to reflect the new behavior. Note that in many cases in Core, we keep "sat" unchanged. It's only changing what the front-end receive.
Fixes #721