POST /v1/audio/speech.
When to use it
Use the WebSocket for live conversations: voice agents on the phone or in a browser, anything where a model writes the reply while someone waits for it. Compared with one HTTP request per sentence, it changes four things:- You set up the connection once per call. A new HTTPS connection needs a TCP and TLS handshake, which takes 0.4 to 1 second from India and sometimes longer when a connection attempt has to be retried.
- You can send tokens as they arrive. We find the sentence ends.
- The next sentence is synthesised while the current one plays, so there is no pause between sentences.
- Barge-in is one message. The sentence that was playing stops at once.
429 straight away.
For one-off audio, such as a file, a voicemail or a notification, the
HTTP endpoint is simpler.
The two are priced differently. A socket costs ₹0.90 a minute for as long
as it is open, whatever it says. An HTTP request costs ₹1.80 per 1,000
characters. See Billing.
Connect
From your server
Send your API key in theAuthorization header, as on every other request:
?api_key= returns 401), because URLs end
up in proxy and server logs.
From a browser
A browser cannot set headers on a WebSocket, and your API key must never reach a web page. Your backend mints a short-lived token withPOST /v2/tts/stream/tokens and hands the page only the token. See
Browser stream tokens for
the request, the response and the rules, and the
browser example below for a page that uses one.
When the connection is refused
We check everything we can before the WebSocket upgrade. A refused connection gets an ordinary HTTP response with the usual error envelope:
A successful upgrade carries
X-Session-Id: ttsws_…. Quote it when you ask us
about a socket.
How a call flows
- Connect. The first message you receive is
session.ready, with the settings in force and your limits. - Send
session.updateto pick a voice and an output format. You can skip it if the defaults suit you. - For each turn of the conversation, pick a new
context_idand send the reply’s text intextmessages as your model produces it. - When the reply is complete, send
flush. - Each sentence arrives as
audio.start, then its audio, thenaudio.done. After the last sentence of the turn you getcontext.done. - If the caller interrupts, send
cancelfor that context. See Barge-in. - Keep the socket for the next turn, and close it when the call ends.
Messages you send
Every message is a JSON text frame with atype. A message we cannot accept is
answered with an error event and changes
nothing. The socket stays open.
session.update
response_format without
sample_rate, you get that format’s default rate. If response_format
includes a rate, leave sample_rate out or send the same value. Audio for
sentences already cut keeps the settings it was cut with, and every
audio.start tells you what its audio is.
wav is refused because a WAV header has to state the length of audio that
does not exist yet. Ask for pcm at the rate you need. The compressed formats
(mp3, opus, aac, flac) are for HTTP requests
only and are refused on the socket.
speed, pitch and loudness use the same ranges as on
HTTP. They apply to sentences
cut after the update, so audio already queued keeps its old settings. A value
outside its range returns an invalid_request error and changes nothing.
text
context_id is
any string of 1 to 128 characters that you choose. Send the text exactly as
your model streams it, spaces included: we join the pieces, so a token without
its leading space runs into the word before it.
Events you receive
session.ready
audio.done
Audio
Binary frames carry raw audio with no header, always in whole samples:pcmis signed 16-bit little-endian mono (encoding: pcm_s16le), 2 bytes per sample.- μ-law and A-law are G.711, 1 byte per sample. At 8 kHz that is 8,000 bytes a second, so 160 bytes is 20 ms.
audio.start. Use
audio_transport: base64 only if your client cannot handle binary frames: the
same audio then arrives in audio.chunk events and takes about a third more
bytes.
How text becomes sentences
We hold each context’s text until one of these happens:- A sentence ends:
.,?,!,।,॥or…, optionally followed by a closing quote or bracket, and then whitespace. - A new line arrives.
- The text passes 250 characters (
limits.max_segment_chars) with no sentence end. We cut at the last,;:—–, or else at the last space. - You send
flush.
3.5 and example.com are never
split, and a stop at the very end of what you have sent waits for your next
token or a flush. A full stop after an abbreviation such as Rs., Dr.,
Mr. or e.g., or after a single initial, does not end a sentence. A sentence
shorter than two words joins the next one, so “Okay.” is not sent on its own.
Lengths are counted in Unicode code points.
Ordering
These hold on every socket:- Sentences are spoken one at a time, in the order they were cut, across all contexts.
- Each
audio.startis closed by exactly oneaudio.doneorerrorwith the samerequest_id, or bycontext.cancelledfor its context. The nextaudio.startcomes after that. - So every binary frame belongs to the most recent
audio.start. - A context’s
context.donecomes after its last sentence’saudio.doneorerror. - Nothing for a context is sent after its
context.cancelled.
Barge-in
When the caller starts talking over your voice agent:- Send
cancelwith thecontext_idthat is speaking. - Drop any audio you still receive for that context. Some may already be on
the wire. Track the speaking context from the latest
audio.start, and start playing again at the nextaudio.startfor a different context. - Clear the audio you have already handed to your player or phone provider.
On Twilio, send a
clearevent; on Plivo,clearAudio; on Exotel,clear. - Use a new
context_idfor the next reply.
cancel discards the context’s unsent text and its queued
sentences, and stops the sentence being synthesised straight away. You always
get context.cancelled, even if the context had already
finished, so you never need to know whether you were too late.
When your workspace is busy
Each sentence takes one synthesis slot from your workspace’s concurrency limit, the same pool HTTP requests use, while it is synthesised. An open socket that is not speaking holds no slot. When a sentence’s turn comes and the workspace is full, it waits for a slot in a queue shared by all your sockets, for up to 3 seconds. If no slot frees in that time, you get anerror with code at_capacity and retry_after_secs
for that sentence, and the socket moves on to the next one. Send that text
again if you still need it. pipecat-mirai does this once for you.
If you run many calls at once, ask us for a higher limit. Ask
for a little more than your peak number of simultaneous calls, because each
call starts its next sentence shortly before the current one finishes.
Billing
A socket is billed for the time it is open: ₹0.90 a minute, counted per second fromsession.ready until the socket closes, whoever closes it. Idle
time counts, because an open socket holds its place on our side whether or not
it is speaking. The sentences it speaks are not charged again.
- Opening. The socket, and the stream token
that promises one, need one minute’s price in your wallet. Otherwise you get
402 insufficient_balanceand nothing opens. - While it is open. We charge each full minute when it ends, as one row on
your wallet. If your wallet then
cannot cover the next minute, we send
errorandsession.closedwithinsufficient_balanceand close the socket (close code 1008). - Closing. The last part-minute is charged when the socket closes. The
whole socket costs its seconds × ₹0.90 / 60, rounded up once to the paisa:
a 95-second socket costs ₹1.43.
session.closedcarriesbilled_secondsandcost_paise.
session.ready states the terms the socket opened with, and they hold until it
closes. If your workspace has agreed a different arrangement with us, such as a
per-character price, billing says so: {"mode": "chars", "paise_per_1k_chars": 180} means each sentence is charged on its own at ₹1.80
per 1,000 characters, and audio.done reports its chars_billed and
cost_paise.
Request log
Every sentence has its own row in your request log, withtransport set to
websocket. Its request_id is the row’s id. If your wallet balance runs
out, a sentence fails with insufficient_balance, or the socket closes with
that reason (see Closing).
Limits
We send a WebSocket ping every 20 seconds and drop a client that has not
answered for 60 seconds. Most WebSocket libraries, and every browser, answer
pings for you. Pings do not count as activity for the idle limit: to keep a
quiet socket open, send
{"type": "session.update"} with no fields every 30
seconds or so. It changes nothing.
If your client stops reading, we stop reading your messages until it catches
up, and a client that stays stuck is disconnected. Read the socket all the
time, and do slow work such as pacing audio to a phone line on another task.
Errors on an open socket
Anerror event affects one message or one sentence. The socket stays open and
the next sentence goes ahead.
Closing
Close the socket yourself when the call ends. If you sendclose first, we
finish speaking what you have sent before closing. When we close a socket, we
send session.closed with a reason, then close it:
Text that was not delivered when a socket closed is dropped. Send it again on a
new socket.
Python example
This script streams a model’s reply into the socket token by token and saves 8 kHz μ-law, the format a phone line takes. Replacellm_tokens with your
model’s stream.
stream_tts.py
websockets older than 14, pass extra_headers= instead of
additional_headers=.
Bridge to a phone call
To play the audio on a Twilio call, forward each binary frame to the Media Stream instead of writing it to a file. The class below does that for one call. It keeps Twilio at most 0.4 seconds ahead of what the caller hears, which absorbs short stalls on your server without making barge-in slow, and it handles barge-in. Plivo and Exotel work the same way with their own message names, and Exotel takespcm_8000 instead of μ-law (see
For phone calls).
phone_leg.py
on_mirai_message. The pacing then waits in that task while the rest of your
voice agent keeps running. The same pacing is what apply_output_lead does in
Pipecat.
Browser example
A page that speaks typed text, with a Stop button for barge-in. Your backend mints a stream token, so the API key stays on your server.requireSignedIn
stands in for your own sign-in check.
server.js
AudioWorklet and holds 150 ms of audio
before each sentence starts, so a short first chunk cannot start playback and
then stall.
index.html
connect mints a new one each time it needs to
reconnect, for example after the socket has been idle for two minutes.
Related
TTS quickstart
The HTTP endpoint, output formats and the request log.
Pipecat
MiraiTTSService and the recommended phone setup.Fixing choppy audio
Symptoms, causes and fixes for gaps and breaks.
Limits
Concurrency, sockets and rate limits.