-
Notifications
You must be signed in to change notification settings - Fork 1
Anthropic: stop_reason and stop_sequence not captured in span output #175
Copy link
Copy link
Labels
Description
Summary
The Anthropic integration's _log_message_to_span function extracts only role and content from the response message, dropping stop_reason and stop_sequence. Every other comparable wrapper in this repo captures the model's stop/finish reason in the span output.
Inconsistency across wrappers
| Wrapper | Captures stop/finish reason? | Location |
|---|---|---|
| OpenAI (chat completions) | Yes — finish_reason in output choices |
oai.py ~line 354 (streaming), ~line 204 (non-streaming) |
| OpenAI (responses) | Yes — status in response object |
oai.py response wrapper |
| Google GenAI | Yes — finish_reason per candidate |
google_genai/tracing.py ~line 313 |
| Agno | Yes — finish_reason in aggregated output |
agno/tracing.py ~line 293 |
| Anthropic | No | anthropic/tracing.py ~line 445 |
What is dropped
The Anthropic Message response object includes:
stop_reason: One of"end_turn","max_tokens","stop_sequence","tool_use"— tells users why the model stopped generatingstop_sequence: The specific stop sequence matched (whenstop_reasonis"stop_sequence")model: The resolved model name (e.g.,claude-sonnet-4-5-20250929when the request usedclaude-sonnet-4-5-latest)
The current code at _log_message_to_span (line ~445 of tracing.py) only extracts:
output = {
k: v
for k, v in {"role": getattr(message, "role", None), "content": getattr(message, "content", None)}.items()
if v
} or NoneWhy this matters
stop_reason: "max_tokens"signals output truncation — a common debugging needstop_reason: "tool_use"signals the model wants tool execution — critical for agent workflows- The resolved
modelname is needed for accurate cost tracking when using model aliases - Other wrappers in this repo already capture the equivalent field, so users may expect it from Anthropic too
Braintrust docs status
The Anthropic integration docs describe streaming and metric collection but do not mention stop_reason capture: not_found.
Upstream source
Anthropic Messages API reference documents stop_reason as a top-level field on every Message response.
Local files inspected
py/src/braintrust/integrations/anthropic/tracing.py—_log_message_to_span(line ~445) only extractsroleandcontentpy/src/braintrust/integrations/anthropic/_utils.py—extract_anthropic_usageextracts token metrics but not response-level fieldspy/src/braintrust/oai.py— OpenAI wrapper capturesfinish_reasonfor comparisonpy/src/braintrust/integrations/google_genai/tracing.py— Google GenAI capturesfinish_reasonfor comparison
Reactions are currently unavailable