-
Notifications
You must be signed in to change notification settings - Fork 1
[BOT ISSUE] OpenAI: Responses API retrieve(), cancel(), and background mode not instrumented #187
Description
Summary
The OpenAI Responses API supports background execution (background=True) and stored responses (store=True), but the wrapper only instruments create() and parse(). The retrieve() and cancel() methods fall through via NamedWrapper.__getattr__ to the unwrapped SDK, producing no Braintrust spans.
This is most impactful for background mode: when background=True, the create() call returns immediately with status: "queued" or "in_progress" and no output or token usage. The user must then call client.responses.retrieve(response_id) to poll for the completed result — but that call is completely untraced. The final output, token usage, and cost data are invisible to Braintrust.
What is missing
ResponsesV1Wrapper / AsyncResponsesV1Wrapper only wrap:
| Method | Instrumented? |
|---|---|
responses.create() |
Yes |
responses.parse() |
Yes |
responses.stream() |
No (tracked in #170) |
responses.retrieve() |
No |
responses.cancel() |
No |
Background mode workflow (entirely untraced after create()):
client = wrap_openai(OpenAI())
# Step 1: create() IS traced, but span has no output/usage (response is queued)
response = client.responses.create(
model="gpt-4o",
input="Analyze this dataset...",
tools=[{"type": "code_interpreter"}],
background=True, # returns immediately
)
# span logged: input ✓, output ✗ (empty), metrics ✗ (no usage yet)
# Step 2: retrieve() is NOT traced — falls through to unwrapped SDK
while response.status != "completed":
response = client.responses.retrieve(response.id)
# final output, token usage, and cost: invisible to BraintrustImpact
- Users of background responses (common with long-running built-in tool tasks like
code_interpreterorfile_search) get a span with request metadata but no output, no token usage, and no cost data cancel()for aborting background responses is also untraced — no visibility into cancelled work
Braintrust docs status
not_found — The OpenAI integration page does not mention retrieve(), background responses, or stored responses.
Upstream sources
- OpenAI Python SDK
Responsesclass:openai/resources/responses/responses.py— definescreate(),retrieve(),cancel(),delete(),stream(),parse(),connect() create()acceptsbackground(bool) andstore(bool) parametersretrieve(response_id)fetches a stored or background response by ID
Local files inspected
py/src/braintrust/oai.py:ResponsesV1Wrapper(line 873) — only definescreate()andparse()AsyncResponsesV1Wrapper(line 892) — only defines asynccreate()andparse()NamedWrapper.__getattr__(line 24) — delegates unwrapped attrs to original client
py/src/braintrust/wrappers/test_openai.py— no tests forretrieve(),cancel(), orbackground=True