Pipecat

If your voice bot runs on Pipecat, install our integration and add one service to your pipeline.

uv add pipecat-mirai

pip install -U pipecat-mirai works too. This page describes version 0.5.0. You need Pipecat 1.8.1 or newer and Python 3.11 or newer.

Add it to your pipeline

Use MiraiTTSService. It opens one WebSocket to Mirai when the pipeline starts and carries the whole call over it:

import os
from pipecat_mirai import MiraiTTSService

tts = MiraiTTSService(
    api_key=os.environ["MIRA_API_KEY"],
    settings=MiraiTTSService.Settings(voice="shruti"),
)

pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),
    context_aggregator.assistant(),
])

Pick ashu, neha, shruti or sameer as the voice (you can hear them here). Each sentence of the LLM’s reply goes to Mirai as soon as Pipecat has it, and Mirai synthesises the next sentence while the current one plays, so there is no pause between them. The service also:

asks Mirai for audio at your pipeline’s sample rate, so a phone pipeline at 8 kHz downloads a sixth of the data that 48 kHz would need, and nothing is resampled on your side;
cancels the reply on Mirai the moment the caller interrupts, drops any of its audio still arriving, and you are not charged for the cut sentence;
waits and resends once if Mirai is briefly at capacity and says when to try again (up to 5 seconds);
reopens a dropped socket at once and resends a reply the drop cut short, from the first sentence the caller had not started hearing;
keeps a quiet socket open through long pauses in the call;
holds 150 ms of audio before each sentence starts, so a short first chunk cannot start playback and then stall.

To send every LLM token as it arrives and let Mirai find the sentence ends, pass text_aggregation_mode=TextAggregationMode.TOKEN (from pipecat.services.tts_service). Mirai knows the danda (।), abbreviations like “Rs.” and “Dr.”, and numbers like 3.5. Billing is per character, as on the HTTP endpoint, and each sentence Mirai delivers shows up in your request log. The WebSocket page has the protocol underneath, if you want to know what the service sends.

Streaming from the edge

The service streams from Mirai’s edge, a server next to the speech GPUs, instead of through the API region. You don’t change anything. The socket opens on the edge with your API key, the same way it would on the API, and the key is sent in the header, never in a URL. In our production test from India (10 calls at once on the default settings), the first audio of a reply came ~100 ms sooner at the median and up to ~500 ms sooner at p95. If the edge can’t be reached or doesn’t answer within 3 seconds, the socket opens on the API (url) as before and the edge is left alone for a minute. If that keeps happening, the wait doubles, up to 16 minutes. A rejected key, an empty wallet or a rate limit always comes back with the API’s own error message. edge=False (or MIRAI_TTS_EDGE=off in the environment) keeps every socket on the API. tts.connected_url shows where a call is streaming from.

Recommended phone setup

For a voice agent that answers phone calls through a websocket transport (Twilio, Plivo, Exotel or Telnyx), run the pipeline at 8 kHz and use MiraiTTSService with its defaults. That’s all: the service sets the output lead on your transport itself, shares warm connections across every call in the process, and races a second connect when one stalls.

import os
from pipecat.pipeline.task import PipelineParams
from pipecat_mirai import MiraiTTSService

transport = FastAPIWebsocketTransport(websocket, FastAPIWebsocketParams(
    audio_in_enabled=True,
    audio_out_enabled=True,
    add_wav_header=False,
    serializer=serializer,          # TwilioFrameSerializer, PlivoFrameSerializer, ...
))

tts = MiraiTTSService(
    api_key=os.environ["MIRA_API_KEY"],
    settings=MiraiTTSService.Settings(voice="shruti"),
)

# Run the pipeline at the phone line's rate, so Mirai sends 8 kHz audio.
params = PipelineParams(audio_in_sample_rate=8000, audio_out_sample_rate=8000)

Each part fixes a different cause of broken audio:

With the pipeline at 8 kHz, Mirai sends 128 kbit/s per call instead of 768 kbit/s at 48 kHz. At 48 kHz, six calls on an ordinary link already arrive slower than real time, and callers hear gaps.
Pipecat normally sends audio to the phone line at exactly real time, so the provider never holds more than about 40 ms of it. If your server stalls for longer (a blocking call, a heavy parse, many calls on one CPU), the line runs dry and the voice breaks up. The Mirai services let Pipecat send up to 0.4 seconds ahead (output_lead_secs, applied automatically when the pipeline starts), so short stalls never reach the caller. When the caller interrupts, Pipecat still tells the provider to clear its queue, so at most 0.4 seconds of sent audio is thrown away. With another TTS vendor, call apply_output_lead(transport) yourself.
Calls in the same process share connections that are already open, so after the first call a greeting doesn’t wait for a TCP and TLS handshake. When many calls start at once, a connect that stalls (a lost packet costs a full second) is raced by a second attempt after 300 ms.
No sentence waits for a new connection, and a sentence waits briefly for capacity instead of failing.

With only these defaults, we started 10 phone calls at the same instant in one Pipecat process against our production API, stalled the bot’s event loop for 150–300 ms every 2 seconds, and interrupted a fifth of the replies: about 100 turns per run, no audible gaps, no errors, nothing played after an interruption and every sentence in order. Earlier, with the same setup, we ran 12 and then 16 simultaneous phone calls through our API to the speech engine: 235 and 262 conversational turns, about a fifth of them interrupted by the caller. Every call played with no audible gaps and no errors, and every sentence arrived in order. The test harness is in the repository. We also measured the output lead on its own, with stalls of 50 to 250 ms every 3 seconds or so on each call. At 10 calls on one server, breaks stretched the bot’s speech by 29% with stock Pipecat and by 0.01% with apply_output_lead. The lead has one side effect: Pipecat’s “bot stopped speaking” event fires up to 0.4 seconds before the caller actually stops hearing the bot. If callers still hear gaps, see Fixing choppy or broken audio.

Limits

Two limits apply per workspace, shared by every process that uses your key (the limits page has the full table):

Sentences in flight: 10. A call holds a slot while one of its sentences is streaming, which is most of the time its bot is speaking, and none while the caller talks. Ten slots carry ten bots speaking at the same moment. A sentence that starts when all ten are taken waits for one (up to 3 seconds), which shows up as a longer time to first byte; past that the service reports at_capacity.
Open sockets. One per live call, plus the few each worker process keeps waiting (two, or its recent peak of calls plus two).

Before a load test with more than 10 bots speaking at once, tell us the number and we raise your workspace’s limits. Nothing changes in your code.

HTTP only

You shouldn’t need it: if WebSockets can’t get through to Mirai at all (a proxy that only allows plain HTTPS, say), MiraiTTSService speaks the call over Mirai’s HTTP endpoint by itself and logs it once. If you want HTTP and nothing else, MiraiHttpTTSService makes one HTTPS request per sentence over shared, kept-warm connections, with the same voices and settings.

Settings

Argument	Default	What it does
`api_key`	`$MIRAI_API_KEY`, then `$MIRA_API_KEY`	Your Mirai API key.
`settings`	`voice="neha"`, `model="mira-tts"`	`MiraiTTSService.Settings(...)`. A new voice applies from the next sentence.
`sample_rate`	The pipeline’s `audio_out_sample_rate`	The rate the service outputs.
`server_sample_rate`	`"auto"`	The rate to ask Mirai for. `"auto"` asks for the output rate. `None` asks for nothing, so Mirai sends 48 kHz and the service resamples.
`prebuffer_secs`	`0.15`	Audio held before a sentence starts playing. `0` plays from the first byte.
`keepalive_secs`	`30`	How often a quiet socket is kept open. `None` turns it off; Mirai closes a socket after 120 seconds of quiet.
`text_aggregation_mode`	sentence	`TextAggregationMode.TOKEN` sends every token as it arrives.
`edge`	`"auto"`	Stream from Mirai’s edge when it can. `False` (or `MIRAI_TTS_EDGE=off`) always uses `url`.
`url`	`wss://sandbox.voice.miraiminds.co/v1/audio/speech/stream`	Mirai’s streaming endpoint.

Code written for the HTTP service of 0.4 and earlier still runs: base_url, http_client, warm_connection and keep_warm_secs are accepted.

Upgrading

Run uv add "pipecat-mirai>=0.5.0" or pip install -U pipecat-mirai. Your code keeps working as it is:

MiraiTTSService now streams each call over a WebSocket from the edge (before 0.5.0 it made one HTTPS request per sentence). Nothing to change.
MiraiWebsocketTTSService is another name for MiraiTTSService, the same class. Nothing to change.
Any apply_output_lead(transport) call can go: the service applies the lead itself (calling it as well is harmless).

Coming from 0.2, two more things you may notice: Mirai now sends audio at your pipeline’s rate instead of 48 kHz (server_sample_rate=None brings back the old behaviour), and the first audio of each sentence waits until 150 ms is in hand (prebuffer_secs=0 plays from the first byte).

Check that it works

export MIRA_API_KEY='your_api_key'
uv run https://raw.githubusercontent.com/MiraiMinds/pipecat-mirai/main/examples/foundational/01-say-hello.py --voice shruti

This speaks one line through a real Pipecat pipeline and saves it as hello.wav. It uses a few paise of credit. 03-websocket-say-hello.py does the same over the WebSocket, feeding the reply in a few words at a time the way an LLM would.

Run the whole turn on Mirai

The services above replace only your TTS. You can also hand us the whole turn: speech recognition, turn detection, the model and the voice. Use MiraiRealtimeLLMService from the same package. It connects to our Realtime API and replaces your STT, LLM and TTS services:

from pipecat_mirai import MiraiRealtimeLLMService

llm = MiraiRealtimeLLMService(voice="shruti", language="hi")

pipeline = Pipeline([transport.input(), ctx.user(), llm, transport.output(), ctx.assistant()])

Every turn makes one round trip to your server instead of three. A session is billed per minute like a browser call, not per character or token. To start from one of your agents, pass agent_id (and variables for its placeholders). Each reply’s timing on our side arrives through an event handler:

llm = MiraiRealtimeLLMService(agent_id="agt_...", variables={"customer_name": "Rahul"})

@llm.event_handler("on_turn_metrics")
async def on_turn_metrics(service, m):
    print(f"{m.v2v_ms} ms voice to voice")

It connects to the sandbox by default. Once your company has gone live, pass base_url=PRODUCTION_REALTIME_URL (also from pipecat_mirai). On a phone line the pipeline can stay at 8 kHz: the service converts the caller’s audio for you. Pipecat’s own OpenAIRealtimeLLMService works too, with base_url="wss://sandbox.voice.miraiminds.co/v2/realtime". The package version adds our settings (language, temperature, tool timeout), the per-turn timing and the agent parameters that the stock service can’t send. The Realtime API page has the protocol details.

Moving from OpenAITTSService

If you set Mirai up through Pipecat’s OpenAITTSService with the VALID_VOICES workaround, replace it with MiraiTTSService. You can delete the voice-list patch, and the 24 kHz warning goes away. Your key, voices and billing stay the same.

pipecat-mirai on GitHub

Source, a Twilio phone-bot example and the phone-call benchmarks.

Streaming text to speech

The WebSocket protocol the service speaks.

Start here

Build

Prompting

Text to speech

Speech to text

Money

Reference

Add it to your pipeline

Streaming from the edge

Recommended phone setup

Limits

HTTP only

Settings

Upgrading

Check that it works

Run the whole turn on Mirai

Moving from OpenAITTSService

pipecat-mirai on GitHub

Streaming text to speech

​Add it to your pipeline

​Streaming from the edge

​Recommended phone setup

​Limits

​HTTP only

​Settings

​Upgrading

​Check that it works

​Run the whole turn on Mirai

​Moving from OpenAITTSService

pipecat-mirai on GitHub

Streaming text to speech

Add it to your pipeline

Streaming from the edge

Recommended phone setup

Limits

HTTP only

Settings

Upgrading

Check that it works

Run the whole turn on Mirai

Moving from OpenAITTSService