pip install -U pipecat-mirai works too. This page describes version 0.5.0,
plus MiraiSTTService, which needs 0.6.0.
You need Pipecat 1.8.1 or newer and Python 3.11 or newer.
Add it to your pipeline
UseMiraiTTSService. It opens one WebSocket to Mirai when the
pipeline starts and carries the whole call over it:
ashu, neha, shruti, sameer, munshi or tanvi as the voice
(you can hear them here). Each sentence of the LLM’s reply goes to Mirai
as soon as Pipecat has it, and Mirai synthesises the next sentence while the
current one plays, so there is no pause between them. The service also:
- asks Mirai for audio at your pipeline’s sample rate, so a phone pipeline at 8 kHz downloads a sixth of the data that 48 kHz would need, and nothing is resampled on your side;
- cancels the reply on Mirai the moment the caller interrupts, drops any of its audio still arriving, and you are not charged for the cut sentence;
- waits and resends once if Mirai is briefly at capacity and says when to try again (up to 5 seconds);
- reopens a dropped socket at once and resends a reply the drop cut short, from the first sentence the caller had not started hearing;
- keeps a quiet socket open through long pauses in the call;
- holds 150 ms of audio before each sentence starts, so a short first chunk cannot start playback and then stall.
text_aggregation_mode=TextAggregationMode.TOKEN (from
pipecat.services.tts_service). Mirai knows the danda (।), abbreviations like
“Rs.” and “Dr.”, and numbers like 3.5.
Each socket is billed for the time it is open, ₹0.90 a minute, not per
character (see WebSocket billing), and each
sentence Mirai delivers shows up in your
request log.
The WebSocket page has the protocol underneath, if you
want to know what the service sends.
Streaming from the edge
The service streams from Mirai’s edge, a server next to the speech GPUs, instead of through the API region. You don’t change anything. The socket opens on the edge with your API key, the same way it would on the API, and the key is sent in the header, never in a URL. In our production test from India (10 calls at once on the default settings), the first audio of a reply came ~100 ms sooner at the median and up to ~500 ms sooner at p95. If the edge can’t be reached or doesn’t answer within 3 seconds, the socket opens on the API (url) as before and the edge is left alone for a minute. If that
keeps happening, the wait doubles, up to 16 minutes. A rejected key, an empty
wallet or a rate limit always comes back with the API’s own error message.
edge=False (or MIRAI_TTS_EDGE=off in the environment) keeps every socket on
the API. tts.connected_url shows where a call is streaming from.
The edge serves only sockets billed per character. With per-minute billing,
which is the default, the edge declines with 403 edge_not_available and the
service opens the socket on the API (url) instead.
Recommended phone setup
For a voice agent that answers phone calls through a websocket transport (Twilio, Plivo, Exotel or Telnyx), run the pipeline at 8 kHz and useMiraiTTSService with its defaults. That’s all: the service sets the output
lead on your transport itself, shares warm connections across every call in the
process, and races a second connect when one stalls.
- With the pipeline at 8 kHz, Mirai sends 128 kbit/s per call instead of 768 kbit/s at 48 kHz. At 48 kHz, six calls on an ordinary link already arrive slower than real time, and callers hear gaps.
- Pipecat normally sends audio to the phone line at exactly real time, so the
provider never holds more than about 40 ms of it. If your server stalls for
longer (a blocking call, a heavy parse, many calls on one CPU), the line
runs dry and the voice breaks up. The Mirai services let Pipecat send up to
0.4 seconds ahead (
output_lead_secs, applied automatically when the pipeline starts), so short stalls never reach the caller. When the caller interrupts, Pipecat still tells the provider to clear its queue, so at most 0.4 seconds of sent audio is thrown away. With another TTS vendor, callapply_output_lead(transport)yourself. - Calls in the same process share connections that are already open, so after the first call a greeting doesn’t wait for a TCP and TLS handshake. When many calls start at once, a connect that stalls (a lost packet costs a full second) is raced by a second attempt after 300 ms.
- No sentence waits for a new connection, and a sentence waits briefly for capacity instead of failing.
apply_output_lead.
The lead has one side effect: Pipecat’s “bot stopped speaking” event fires up
to 0.4 seconds before the caller actually stops hearing the bot.
If callers still hear gaps, see
Fixing choppy or broken audio.
Limits
Two limits apply per workspace, shared by every process that uses your key (the limits page has the full table):- Sentences in flight: 10. A call holds a slot while one of its sentences
is streaming, which is most of the time its bot is speaking, and none while
the caller talks. Ten slots carry ten bots speaking at the same moment. A
sentence that starts when all ten are taken waits for one (up to 3 seconds),
which shows up as a longer time to first byte; past that the service reports
at_capacity. - Open sockets. One per live call, plus the few each worker process keeps waiting (two, or its recent peak of calls plus two).
HTTP only
You shouldn’t need it: if WebSockets can’t get through to Mirai at all (a proxy that only allows plain HTTPS, say),MiraiTTSService speaks the call over
Mirai’s HTTP endpoint by itself and logs it once. If you want HTTP and nothing
else, MiraiHttpTTSService makes one HTTPS request per sentence over shared,
kept-warm connections, with the same voices and settings.
Settings
Code written for the HTTP service of 0.4 and earlier still runs:
base_url, http_client, warm_connection and keep_warm_secs are accepted.
Upgrading
Runuv add "pipecat-mirai>=0.5.0" or pip install -U pipecat-mirai. Your code
keeps working as it is:
MiraiTTSServicenow streams each call over a WebSocket from the edge (before 0.5.0 it made one HTTPS request per sentence). Nothing to change.MiraiWebsocketTTSServiceis another name forMiraiTTSService, the same class. Nothing to change.- Any
apply_output_lead(transport)call can go: the service applies the lead itself (calling it as well is harmless).
server_sample_rate=None brings back the old
behaviour), and the first audio of each sentence waits until 150 ms is in hand
(prebuffer_secs=0 plays from the first byte).
Check that it works
hello.wav. It uses a few paise of credit.
03-websocket-say-hello.py
does the same over the WebSocket, feeding the reply in a few words at a time
the way an LLM would.
Transcribe with MiraiSTTService
From version 0.6.0 the package can replace your STT service too, with Mira STT, our own speech recognition and the one our agents listen with. Upgrade withuv add "pipecat-mirai>=0.6.0" and add MiraiSTTService where your STT
service was:
InterimTranscriptionFrames, and each utterance ends with one
TranscriptionFrame. language takes any of the
ten language codes: hi-IN, en-IN, gu-IN and seven
more. It also:
- sends a phone pipeline’s 8 kHz audio as it is, and resamples any other rate to 16 kHz first;
- reconnects once if the socket drops mid-call and resends up to 30 seconds of audio that had no transcript yet, so the caller doesn’t have to repeat themselves;
- turns a fatal error from Mirai into an
ErrorFrame.
Endpointing
When your pipeline has a VAD, such as avad_analyzer on the user aggregator
in Pipecat’s examples, the service uses manual endpointing. Your
VAD’s “user started speaking” opens an utterance with half a second of the
audio from just before it, and “user stopped speaking” closes it. The final
transcript comes back straight away, with no silence wait on Mirai’s side.
Without a VAD, the service lets Mirai find the turns (vad endpointing),
which waits for a second of silence by default before the final. Pass
endpointing="manual" or endpointing="vad" to choose yourself.
The service’s TTFB metric is the time from the end of speech to the final
transcript, which is the pause your caller sits through before the LLM can
start. Its usage metric is seconds of audio, which is what you’re billed for
(see Billing).
Limits for STT
A workspace can have 20 sessions open at once (100 on a monthly plan): one per live call, plus the few sockets each process keeps open and waiting so a call’s first audio does not wait for a handshake. Before a load test with more calls than that, tell us the number and we raise the limit. The streaming page has the full table.Run the whole turn on Mirai
The services above replace only your TTS and STT. You can also hand us the whole turn: speech recognition, turn detection, the model and the voice. UseMiraiRealtimeLLMService from the same package. It connects to our
Realtime API and replaces your STT, LLM and TTS services:
agent_id (and variables for its
placeholders). Each reply’s timing on our side arrives through an event
handler:
base_url=PRODUCTION_REALTIME_URL (also from pipecat_mirai). On a phone line the pipeline can stay at
8 kHz: the service converts the caller’s audio for you.
Pipecat’s own OpenAIRealtimeLLMService works too, with
base_url="wss://sandbox.voice.miraiminds.co/v2/realtime". The package version
adds our settings (language, temperature, tool timeout), the per-turn timing and
the agent parameters that the stock service can’t send. The
Realtime API page has the protocol details.
Moving from OpenAITTSService
If you set Mirai up through Pipecat’sOpenAITTSService with the
VALID_VOICES workaround, replace it with MiraiTTSService. You can
delete the voice-list patch, and the 24 kHz warning goes away. Your key, voices
and billing stay the same.
pipecat-mirai on GitHub
Source, a Twilio phone-bot example and the phone-call benchmarks.
Streaming text to speech
The WebSocket protocol
MiraiTTSService speaks.Speech to text, live
The WebSocket protocol
MiraiSTTService speaks.