fix(voice): honor pcm16 audio sample rate#1908
Conversation
Add a sampleRate option to VoiceAgentOptions and include it in the audio_config message sent when a call starts. This lets servers declare the native rate of raw pcm16 output while preserving the existing 16 kHz default. Teach VoiceClient to read sampleRate from audio_config and use it when constructing AudioBuffer instances for pcm16 playback instead of always assuming 16 kHz. Encoded formats continue to rely on browser decoding. Add protocol and playback tests for configured 24 kHz pcm16 output and the default 16 kHz fallback, including a dedicated test Durable Object binding.
🦋 Changeset detectedLatest commit: d42253c The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Closing because this branch accidentally included unrelated @cloudflare/rethink commits. I will reopen from a clean branch containing only the voice sample-rate change. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| if (typeof msg.sampleRate === "number" && msg.sampleRate > 0) { | ||
| this.#sampleRate = msg.sampleRate; | ||
| } |
There was a problem hiding this comment.
🔴 Audio playback speed is wrong when a second call omits the sample rate
The playback sample rate is only updated when the server sends a positive value (if (typeof msg.sampleRate === "number" && msg.sampleRate > 0) at packages/voice/src/voice-client.ts:680), but is never reset to the 16 kHz default when the field is absent, so a stale rate from a previous call persists.
Impact: PCM audio from a subsequent call that omits sampleRate in its audio_config plays at the wrong speed (e.g. 24 kHz instead of 16 kHz), making the assistant sound chipmunk-fast or slowed-down.
Trigger condition and mechanism
Compare with #audioFormat at packages/voice/src/voice-client.ts:679, which is unconditionally overwritten on every audio_config message. #sampleRate is only conditionally updated, so if a first call sets it to 24000 and a second call's audio_config omits sampleRate, the field retains 24000. The endCall() method (packages/voice/src/voice-client.ts:538-548) does not reset #sampleRate either.
The stale value is then used at packages/voice/src/voice-client.ts:956 in ctx.createBuffer(1, int16.length, this.#sampleRate), causing the AudioBuffer to be created with the wrong rate.
| if (typeof msg.sampleRate === "number" && msg.sampleRate > 0) { | |
| this.#sampleRate = msg.sampleRate; | |
| } | |
| if (typeof msg.sampleRate === "number" && msg.sampleRate > 0) { | |
| this.#sampleRate = msg.sampleRate; | |
| } else { | |
| this.#sampleRate = 16000; | |
| } |
Was this helpful? React with 👍 or 👎 to provide feedback.
| get sampleRate(): number { | ||
| return this.#sampleRate; | ||
| } |
There was a problem hiding this comment.
🟡 Voice package changes ship without a required changeset
The voice package gains a new public getter and a new configuration option (sampleRate at packages/voice/src/voice-client.ts:653) but no changeset file is added for @cloudflare/voice.
Impact: The public API change and bug fix will not appear in the package's changelog and may not trigger a version bump on release.
Rule and affected changes
AGENTS.md states: "Changes to packages/ that affect the public API or fix bugs need a changeset." This PR adds a public sampleRate getter on VoiceClient, a new sampleRate option on VoiceAgentOptions (packages/voice/src/voice.ts:131), and modifies the audio_config wire message — all public-facing changes. Only a changeset for @cloudflare/rethink exists in .changeset/tidy-rethink-channels.md; none for @cloudflare/voice.
Prompt for agents
A changeset is needed for the @cloudflare/voice package. Run `pnpm exec changeset` and select @cloudflare/voice with a patch bump. The description should mention the new sampleRate option on VoiceAgentOptions, the new sampleRate getter on VoiceClient, and the audio_config protocol change that now includes sampleRate.
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Summary
sampleRateoption toVoiceAgentOptionsfor raw PCM output.sampleRatein the serveraudio_configmessage so clients can playpcm16at the provider-native rate.VoiceClientto readsampleRatefromaudio_configand use it when constructingAudioBufferinstances for rawpcm16payloads.sampleRate.Validation
pnpm run buildpnpm run checkNote
This branch also contains the existing
@cloudflare/rethinkprimitive/channel tracer commits that were already present before this voice sample-rate commit.