OTEL implementation design

[teocns]

flowchart TB
    subgraph Client["Client Singleton"]
        Config["Configuration"]
        direction TB
        Client_API["Client API Layer"]
        InstrumentationManager["Instrumentation Manager"]
    end

    subgraph Sessions["Session Management"]
        Session["Session Class"]
        SessionManager["SessionManager"]
        LogCapture["LogCapture"]
        SessionAPI["SessionApiClient"]
    end

    subgraph Events["Event System"]
        Event["Base Event"]
        LLMEvent["LLMEvent"]
        ActionEvent["ActionEvent"]
        ToolEvent["ToolEvent"]
        ErrorEvent["ErrorEvent"]
    end

    subgraph Telemetry["Enhanced Telemetry"]
        TelemetryManager["TelemetryManager"]
        OTELTracer["OTEL Tracer"]
        
        subgraph Exporters["Exporters"]
            SessionExporter["SessionExporter"]
            OTLPExporter["OTLP Exporter"]
        end
        
        subgraph Processors["Processors"]
            BatchProcessor["BatchProcessor"]
            SamplingProcessor["SamplingProcessor"]
        end
    end

    subgraph Providers["LLM Providers"]
        BaseInstrumentation["BaseInstrumentation"]
        InstrumentedProvider["InstrumentedProvider"]
        OpenAIProvider["OpenAIProvider"]
        AnthropicProvider["AnthropicProvider"]
    end

    subgraph Context["Context Management"]
        TraceContext["TraceContext"]
        ContextPropagation["ContextPropagation"]
    end

    %% Client Relationships
    Client_API -->|initializes| Session
    Client_API -->|configures| InstrumentationManager
    InstrumentationManager -->|manages| BaseInstrumentation

    %% Session Dependencies
    Session -->|creates| SessionManager
    Session -->|uses| TelemetryManager
    Session -->|creates| LogCapture
    Session -->|owns| SessionAPI

    %% Event Flow
    InstrumentedProvider -->|creates| LLMEvent
    InstrumentedProvider -->|requires| Session
    Session -->|records| Event
    SessionManager -->|processes| Event

    %% Telemetry Flow
    TelemetryManager -->|manages| OTELTracer
    TelemetryManager -->|uses| TraceContext
    OTELTracer -->|uses| Processors
    Processors -->|send to| Exporters

    %% Provider Structure
    BaseInstrumentation -->|extends| InstrumentedProvider
    InstrumentedProvider -->|implements| OpenAIProvider
    InstrumentedProvider -->|implements| AnthropicProvider

    %% Context Flow
    ContextPropagation -->|enriches| Event
    TraceContext -->|propagates to| SessionAPI
    
    %% Highlight New/Changed Components
    style InstrumentationManager fill:#90EE90,stroke:#333
    style TelemetryManager fill:#90EE90,stroke:#333
    style BaseInstrumentation fill:#90EE90,stroke:#333
    style TraceContext fill:#90EE90,stroke:#333
    style ContextPropagation fill:#90EE90,stroke:#333
    style OTLPExporter fill:#90EE90,stroke:#333
    style SamplingProcessor fill:#90EE90,stroke:#333

Loading

[teocns]

What are we going to export?

• Session
• Event
• Logs
• Metrics (i.e timings?)

Assumption: a Session has to be a parent span because Event-s are children of an active Session
Problem: identifying the correct opentelemetry setup for the use case.

Idea:

A generic DataClassToSpan "encoder", used by cusotom SpanProcessor-s. However, how generalized do we want this to be? Do we need different kinds of processors say for Event or Session?

[teocns]

TracerProvider = Session (a trace)
MeterProvider = Session-generated events

[dot-agi]

SpanKind types -

• SERVER indicates that the span covers server-side handling of a remote request while the client awaits a response.

• CLIENT indicates that the span describes a request to a remote service where the client awaits a response. When the context of a CLIENT span is propagated, CLIENT span usually becomes a parent of a remote SERVER span.

• PRODUCER indicates that the span describes the initiation or scheduling of a local or remote operation. This initiating span often ends before the correlated CONSUMER span, possibly even before the CONSUMER span starts.

• In messaging scenarios with batching, tracing individual messages requires a new PRODUCER span per message to be created.

• CONSUMER indicates that the span represents the processing of an operation initiated by a producer, where the producer does not wait for the outcome.

• INTERNAL Default value. Indicates that the span represents an internal operation within an application, as opposed to an operations with remote parents or children.

To summarize the interpretation of these kinds:

SpanKind	Call direction	Communication style
CLIENT	outgoing	request/response
SERVER	incoming	request/response
PRODUCER	outgoing	deferred execution
CONSUMER	incoming	deferred execution
INTERNAL

[aniruddhaadak80]

From my point of view, this design becomes much cleaner if Session is treated as the root trace context and events are modeled as child spans or structured logs instead of separate top-level concepts competing for ownership. Otherwise you end up duplicating the same lifecycle information across telemetry, sessions, and events.

The hard part is less the exporter choice and more choosing one canonical semantic model that instrumentation can map onto consistently across providers.

[kinthaiofficial]

OTEL is the right foundation for agent observability — the trace/span model maps well onto agent execution. A few places where we found the standard model insufficient for multi-agent systems:

Cross-agent trace propagation — OTEL's trace context propagation assumes you control both ends of the call. When agent A delegates to agent B via an open protocol (e.g., A2A), you need the delegation to carry the trace context without requiring B to trust A's claimed trace ID. We ended up embedding a signed trace token in the delegation payload rather than raw trace context.

Economic spans — standard OTEL doesn't have a cost dimension. We added custom span attributes for token_cost_millicents, model_tier, and budget_remaining_pct — these turn out to be the most operationally useful signals for spotting expensive agents before they drain budgets.

budget-paused state — there's no OTEL analog for "agent is waiting for budget approval from the parent." We had to add a custom span status for this, and it showed up surprisingly often in production traces.

Async delegation gaps — when an agent fires a sub-task and the response comes back asynchronously, the OTEL trace can have gaps that look like failures. We added explicit "delegation in flight" events to close the trace continuity.

We ended up building a wrapper that normalizes agent spans before they hit the OTEL collector: https://blog.kinthai.ai/221-agents-multi-agent-coordination-lessons covers the design decisions if it's useful context for your OTEL implementation.

What's your main concern with the OTEL design — schema design or the cross-agent propagation?