fix: retry redis conn in the bg and fail open otherwise

[Flo4604]

What does this PR do?

Adds resilient middleware engine handling to prevent service failures when Redis is unavailable. The middleware engine now fails closed with a 503 Service Unavailable response when Redis connectivity is lost, and automatically retries connection in the background with exponential backoff.

Introduces a new ResilientEvaluator wrapper that atomically swaps between unavailable and working engine states. When Redis is configured but fails to connect, the service returns 503 errors instead of crashing, and continues attempting to reconnect until successful.

Adds a new error code EngineUnavailable with appropriate HTTP status mapping and Prometheus metrics for monitoring engine unavailability events.

Fixes #5365

Type of change

• Enhancement (small improvements)
• Bug fix (non-breaking change which fixes an issue)
• Chore (refactoring code, technical debt, workflow improvements)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How should this be tested?

• Start sentinel service with Redis URL configured but Redis server unavailable
• Verify requests return 503 with "middleware engine temporarily unavailable" message
• Start Redis server and verify engine automatically recovers
• Monitor sentinel_engine_unavailable_total Prometheus metric during unavailability
• Test with empty Redis URL to ensure pass-through mode still works

Checklist

Required

• Filled out the "How to test" section in this PR
• Read Contributing Guide
• Self-reviewed my own code
• Commented on my code in hard-to-understand areas
• Ran pnpm build
• Ran pnpm fmt
• Ran make fmt on /go directory
• Checked for warnings, there are none
• Removed all console.logs
• Merged the latest changes from main onto my branch with git pull origin main
• My changes don't cause any responsiveness issues

Appreciated

• If a UI change was made: Added a screen recording or screenshots to this PR
• Updated the Unkey Docs if changes were necessary

[Flo4604]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• fix: use correct health handlers #5376
• fix: retry redis conn in the bg and fail open otherwise #5377 👈 (View in Graphite)
• feat: add sentinel prom metrics #5283
• feat: add prometheus metrics for frontline routing and proxy #5268
• fix: instance-aware frontline routing to prevent 503s in regions without instances #5267
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
dashboard	Ready	Preview, Comment	Mar 24, 2026 1:23pm

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 3ae8b072-77bd-416d-aaa0-8524da8d3f62

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch 03-18-fix_retry_redis_conn_in_the_bg_and_fail_open_otherwise

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[chronark]

Aren't we over complicating this?
Redos being down should be fine, our api handles it gracefully too and falls back to local evaluation

Why do we do it differently here and introduce redos as potential failure point?

[Flo4604]

the api doesnt tho? the api also requires redis to be up when starting otherwise it will error

Its just handles redis being down in some moment in time that is not the startup
pkg/counter/redis.go L78