feat: implement exponential backoff retry strategy for transient errors

[noahpodgurski]

What's new

• Added exponential backoff retry strategy for transient errors coming from LiteLLM

AIPLAT-452

[ti3x]

I noticed we have two identical retry wrappers—_call_litellm_with_retry() in completions.py and litellm_request_with_retry() in utils.py. Just want to check: was that intentional, or would it make sense to use one shared function instead? Either way, the PR looks great!

[noahpodgurski]

I noticed we have two identical retry wrappers—_call_litellm_with_retry() in completions.py and litellm_request_with_retry() in utils.py. Just want to check: was that intentional, or would it make sense to use one shared function instead? Either way, the PR looks great!

Yeah they're separate functions - they do basically the same thing but I kept the one in completion.py unique to make the tests easier to handle

[ti3x]

@noahpodgurski let's hold off merging for now as I am sorting out PT + Paygo fallback

[subpath]

@noahpodgurski also please run make docs

we can even make a pre-commit hook maybe

[ti3x]

@noahpodgurski feel free to merge this PR since it's requried for the new error code