Skip to content

[BOT ISSUE] OpenAI: Responses API retrieve(), cancel(), and background mode not instrumented #187

@braintrust-bot

Description

@braintrust-bot

Summary

The OpenAI Responses API supports background execution (background=True) and stored responses (store=True), but the wrapper only instruments create() and parse(). The retrieve() and cancel() methods fall through via NamedWrapper.__getattr__ to the unwrapped SDK, producing no Braintrust spans.

This is most impactful for background mode: when background=True, the create() call returns immediately with status: "queued" or "in_progress" and no output or token usage. The user must then call client.responses.retrieve(response_id) to poll for the completed result — but that call is completely untraced. The final output, token usage, and cost data are invisible to Braintrust.

What is missing

ResponsesV1Wrapper / AsyncResponsesV1Wrapper only wrap:

Method Instrumented?
responses.create() Yes
responses.parse() Yes
responses.stream() No (tracked in #170)
responses.retrieve() No
responses.cancel() No

Background mode workflow (entirely untraced after create()):

client = wrap_openai(OpenAI())

# Step 1: create() IS traced, but span has no output/usage (response is queued)
response = client.responses.create(
    model="gpt-4o",
    input="Analyze this dataset...",
    tools=[{"type": "code_interpreter"}],
    background=True,  # returns immediately
)
# span logged: input ✓, output ✗ (empty), metrics ✗ (no usage yet)

# Step 2: retrieve() is NOT traced — falls through to unwrapped SDK
while response.status != "completed":
    response = client.responses.retrieve(response.id)
# final output, token usage, and cost: invisible to Braintrust

Impact

  • Users of background responses (common with long-running built-in tool tasks like code_interpreter or file_search) get a span with request metadata but no output, no token usage, and no cost data
  • cancel() for aborting background responses is also untraced — no visibility into cancelled work

Braintrust docs status

not_found — The OpenAI integration page does not mention retrieve(), background responses, or stored responses.

Upstream sources

  • OpenAI Python SDK Responses class: openai/resources/responses/responses.py — defines create(), retrieve(), cancel(), delete(), stream(), parse(), connect()
  • create() accepts background (bool) and store (bool) parameters
  • retrieve(response_id) fetches a stored or background response by ID

Local files inspected

  • py/src/braintrust/oai.py:
    • ResponsesV1Wrapper (line 873) — only defines create() and parse()
    • AsyncResponsesV1Wrapper (line 892) — only defines async create() and parse()
    • NamedWrapper.__getattr__ (line 24) — delegates unwrapped attrs to original client
  • py/src/braintrust/wrappers/test_openai.py — no tests for retrieve(), cancel(), or background=True

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions