self-hosted managed indexer treats env var lookup failure as fatal instead of degrading gracefully

[flowq-C]

Summary

The build-stage indexer aborts the entire deployment if environment variables cannot be fetched. This creates a full deployment outage from a secondary metadata/indexing dependency.

Proven behavior

• Indexer failure prevents image build completion
• No fallback path
• No warning-only mode

Why this matters

• This turns a metadata/bootstrap problem into a full deployment outage
• In self-hosted environments this is especially brittle because auth/storage/config drift is more likely than in cloud

Repro

1. Cause envvar fetch to fail during local build indexing
2. Run deployment
3. Observe full build failure

Expected

• Indexer should continue with:
  • env = {}
• or mark metadata incomplete but still allow deployment image build

Actual

• Full Docker build fails
• Deployment status ends as FAILED

Evidence

• managed-index-controller throws immediately on failed envvar fetch:
  • throw new Error(\Failed to fetch environment variables: ${$env.error}`)`

Suggested fix

• Downgrade envvar fetch failure to warning
• Continue indexing with empty env variables
• Only fail hard if task graph itself cannot be resolved

[flowq-C]

Additional confirmation from production-like self-hosted validation:

We were able to get the deployment through, but only after working around the build-stage API/auth path.

That reinforced the main point of this issue:

• the indexer is currently too eager to make secondary metadata/bootstrap calls fatal
• in self-hosted setups, auth, routing, object storage, and local registry topology are all more fragile than in cloud
• turning those failures into full deployment outages makes the platform operationally brittle

Even after the broader deployment path was repaired, the original recommendation still stands:

• env/bootstrap/indexing failures should degrade gracefully where possible
• the deployable image should not be blocked by non-essential metadata steps

[flowq-C]

Confirmed and extended — root cause of both build-stage failures documented in #3258

Full working fix with reproduction, root cause, and patch code posted on the related issue:

→ #3258 covers both the /envvars 401 and the /background-workers connectivity failure with complete working code

The summary of what we found:

1. /envvars → runtime key gets 401 → indexer aborts → graceful degradation (empty vars, log warning) is the right upstream fix
2. /background-workers → localhost:8030 unreachable from BuildKit bridge-network container → URL needs to use Docker gateway IP (172.24.1.1:8030) → original auth key works correctly once routing is fixed

Both are confirmed reproducible on self-hosted v4.4.3 with --local-build.