test(cli): add non-interactive smoke test and CI workflow#131
Open
mmurrs wants to merge 2 commits into
Open
Conversation
… and --force Two regressions from #126 surface as opaque "Cannot prompt in non-interactive mode" errors in CI and scripted flows. Both are fixed here. 1. `getEnvironmentInteractive` wrapped environment validation in a silent try/catch, so any error from `getEnvironmentConfig` (unknown env, env unavailable in current build type) fell through to `ensureInteractive` and surfaced as a generic "Cannot prompt in non-interactive mode. Provide --environment or ECLOUD_ENV via CLI flags or environment variables" message -- even when the user had provided ECLOUD_ENV or --environment. The real errors ("Unknown environment: X", "Environment X is not available in this build type") are now surfaced directly, and the interactive guard only fires when nothing was provided at all. 2. `promptUseVerifiableBuild` was called unconditionally from `app deploy` and `app upgrade` in the image-ref-only path (no --dockerfile). It routes through `confirmWithDefault`, which throws in non-TTY mode with the message "Use --force to skip confirmation prompts." But the function had no --force awareness, so passing --force did nothing. Now `promptUseVerifiableBuild(force)` short-circuits to the default answer (false = non-verifiable) when force is true, matching the pattern already used in `auth/login.ts`. Also adds vitest coverage for both regressions in `src/utils/__tests__/prompts.test.ts`. Verified manually: - `ECLOUD_ENV=mainnet-alpha ecloud compute app info <id>` returns app info - `ECLOUD_ENV=bogusenv ecloud compute app info <id>` returns "Unknown environment: bogusenv" - `ecloud compute app deploy --image-ref <ref> ... --force` proceeds past the "Build from verifiable source?" prompt
Black-box smoke test that invokes the built CLI binary with stdin closed and asserts every command path with a non-interactive escape hatch either succeeds or fails with a specific, actionable error message -- never the generic "Cannot prompt in non-interactive mode" fallback. This would have caught both regressions fixed in #130: - Bug 1 (getEnvironmentInteractive swallowing the real error): 6 of 8 assertions in the "unknown environment" and "env unavailable in build type" sections fail against unpatched master, with the generic prompt error instead of the expected specific message. - Bug 2 (promptUseVerifiableBuild ignoring --force): 2 assertions for image-ref-only deploy and upgrade with --force fail against unpatched master, matching the "Cannot confirm Build from verifiable source" message. Verified by temporarily checking out packages/cli/src from origin/master and running the script -- 8 passes, 8 failures, matching the exact regressions #130 fixes. On the patched branch: 16/16 pass. Layout: - .github/workflows/cli-smoke.yml -- runs on every PR + master push that touches packages/cli, packages/sdk, or the workflow itself - packages/cli/test/e2e/non-interactive.sh -- portable assertion script (~10s runtime, no secrets, no on-chain writes) - packages/cli/test/e2e/fixtures/minimal.env -- trivial env fixture - packages/cli/test/e2e/README.md -- how to run locally and how to extend. Also sketches the two follow-up tiers (full testnet deploy, agent-skill runbook). The workflow also runs pnpm -C packages/cli run test so the existing vitest suite gains a PR-level gate it didn't have before.
mmurrs
requested review from
Chris-Moller,
anupsv,
solimander,
taekyunggg and
vineetguptadev
as code owners
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a black-box CLI smoke test that invokes the built binary with
stdin </dev/nulland asserts every command path with a non-interactive escape hatch (--force, explicit flag, env var) either succeeds or fails with a specific, actionable error — never the genericCannot prompt in non-interactive modefallback.This would have caught both regressions fixed in #130. Verified by temporarily checking out
packages/cli/srcfromorigin/masterand running the script — 8 passes, 8 failures, exactly matching the two bugs #130 fixes. On the patched branch: 16/16 pass.Why this belongs in CI
The two bugs in #130 (
getEnvironmentInteractiveswallowing the real error,promptUseVerifiableBuildignoring--force) shipped inv0.5.0-dev.2and silently broke everyECLOUD_ENV=mainnet-alphascript and every image-ref-only deploy in non-TTY contexts. Neither has unit tests that would catch the regression at the binary-invocation level — the unit tests in #130 cover the functions directly but not how they compose through the full oclif command tree.This script closes that gap. It runs in ~10 seconds, requires no secrets, does no on-chain writes.
What it covers
--help/--version/ subcommand help all load without prompt errorsECLOUD_ENV=bogusenvemitsUnknown environment: bogusenv--environment bogusenvemitsUnknown environment: bogusenvECLOUD_ENV=sepolia-devon a prod build emitsnot available in this build type--image-refonly +--forcedoes NOT emitCannot confirm "Build from verifiable source?"--image-refonly +--forcedoes NOT emitCannot confirm "Build from verifiable source?"auth whoamiruns cleanlyLayout
The workflow also runs
pnpm -C packages/cli run testso the existing vitest suite finally has a PR-level gate it didn't have before.Follow-ups the README sketches
This is Tier 1 of a three-tier validation stack I'd like to build out over time:
Tier 2 and Tier 3 are blocked on a couple of upstream fixes (
billing subscribehas no non-interactive flag,auth gen --storecrashes when replacing an existing keyring key). Happy to file issues for those.Dependencies
Stacked on #130. The smoke script asserts the fixes that #130 makes. Against unpatched
masterthe script correctly fails (proving the assertions work). Merge #130 first; this will rebase cleanly onto master after that.Test plan
fix/non-interactive-regressions): 16/16 passorigin/master(unpatched): 8 pass, 8 fail — the 8 failures match Bug 1 and Bug 2 exactlyshellcheck packages/cli/test/e2e/non-interactive.shclean