Disconnect during job start causes the agent to hang

[Fox32]

Describe the bug

If I disconnect after received job request and before the session is started, the job hangs and is killed with job is unresponsive

Relevant log output

In the end the job is hard killed after a job is unresponsive log entry.

[15:56:05.303] DEBUG (88816): job exiting
[15:57:04.963] WARN (88816): job is unresponsive
[15:57:04.963] DEBUG (89120): SIGTERM received in job proc
    jobID: "AJ_67dQtGe5Ck2E"

Describe your environment

1.0.48

Minimal reproducible example

I have very simple agent with a greeting inside of onEnter:

class CustomAgent extends voice.Agent {
    constructor() {
        super({
            instructions:
                "You are a helpful assistant, you can hear the user's message and respond to it.",
        })
    }

    async onEnter(): Promise<void> {
        // Note that I have a greeting on enter
        this.session.say('Hello, how can I help you today?', { allowInterruptions: false })
    }
}

// …

const agent = new CustomAgent()

const session = new voice.AgentSession({
    stt: 'assemblyai/universal-streaming:en',
    llm: 'openai/gpt-4.1-mini',
    tts: 'cartesia/sonic-2:9626c31c-bec5-4cca-baa8-f8ba9e84c8bc',
    vad: jobContext.proc.userData.vad! as silero.VAD,
    turnDetection: new turnDetector.MultilingualModel(),
    voiceOptions: {
        preemptiveGeneration: true,
    },
    connOptions: {
        llmConnOptions: {
            maxRetry: 1,
            retryIntervalMs: 2000,
            timeoutMs: 60000,
        },
    },
})

// Inside the entrypoint I first sleep to make it easier to get to this error:
await new Promise((resolve) => setTimeout(resolve, 2000))

await session.start({
    agent,
    room: jobContext.room,
    inputOptions: {
        noiseCancellation: BackgroundVoiceCancellation(),
    },
})

I added the timeout to make it easier to reproduce the issue.

Additional information

I already tried to debug the issue:

If I connect and disconnect during the 2s timeout, I have the agent hanging, ultimately resulting in job is unresponsive.

My idea what the problem is from initial debugging:

1. Inside the shutdown code of the agent session at some point closeImplInner is called with drain = false (default value).
2. activity.interrupt is therefore called and awaited (here the shutdown code is hanging on the await!)
3. Inside AgentActivity.interrupt all speech handles are interrupted.
4. As there is a speech handle active (see my greeting in onEnter), the lower part with the done callback is executed
5. The done callback would resolve the returned future, but never does so

As far as I see, the doneCallbacks are executed by _markDone (via the doneFut). I guess _markDone is not called properly if the speech handle is interrupted. I haven't investigated further yet… Maybe you have an idea?

In general it waits for the speech to be completed before starting to shutdown, for testing I added a setImmediate(() => session.shutdown({ reason: 'test', drain: false })) after the session start to get to it quicker. The behavior is the same.

[tobiplancraft]

we've been testing with the fix from #1090 applied and while it's a good improvement, we're still seeing the hang. The root cause seems to be a step further down in ttsTask, here is what we found so far:

When a speech handle is interrupted during ttsTask, the cleanup runs:

audioOutput.clearBuffer();
await audioOutput.waitForPlayout();

waitForPlayout() resolves when playbackFinishedCount >= playbackSegmentsCount. Here's the flow we observed that leads to the hang:

1. ttsTask starts processing the speech handle. captureFrame() is called, which sets playbackSegmentsCount = 1 on audioOutput.
2. TTS generation begins, but the participant disconnects before any audio frames are produced.
3. The speech handle is interrupted, ttsTask enters its cleanup path.
4. clearBuffer() is called on audioOutput. This delegates down through the wrapper layers to ParticipantAudioOutput.clearBuffer().
5. ParticipantAudioOutput.clearBuffer() checks this.pushedDuration -- since no audio was ever pushed, it's 0, and the method returns immediately without doing anything.
6. Because clearBuffer() was effectively a no-op, onPlaybackFinished is never emitted.
7. waitForPlayout() sees playbackFinishedCount (0) < playbackSegmentsCount (1) and waits for an onPlaybackFinished event that will never come.

Our suspicion is that this can happen in production whenever a speech handle is interrupted before TTS has pushed audio frames to the underlying ParticipantAudioOutput.

As a workaround, we're patching the cleanup to reconcile the segment counters before awaiting playout:

audioOutput.clearBuffer();
while (audioOutput.playbackFinishedCount < audioOutput.playbackSegmentsCount) {
  audioOutput.onPlaybackFinished({ playbackPosition: 0, interrupted: true });
}
await audioOutput.waitForPlayout();

This is in agent_activity.ts, inside the if (speechHandle.interrupted) block of ttsTask (~line 1484). Not sure if this is the right place for the fix or if it should be handled deeper in the AudioOutput/clearBuffer implementation.