Skip to main content
The Realtime API speaks the OpenAI Realtime protocol. If your bot already uses Pipecat’s OpenAIRealtimeLLMService, or any other client for that protocol, you change the API key and the URL and keep everything else:
Each session runs Mirai’s speech recognition, turn detection, language model and speech synthesis on one voice worker, next to each other. Your process sends the caller’s audio up and plays the agent’s audio back. Compared with calling our speech-to-text, LLM and text-to-speech APIs one by one, a turn makes one round trip to your server instead of three. A session is a call, like a browser call. It has a call ID, it shows up in your call list and it sends webhooks. It costs ₹1 a minute on every plan, with speech recognition, the model and the voice included. The call ID is the session.id you receive in session.created, without its sess_ prefix: session sess_call_01K8… is call call_01K8…. Read the call with GET /v2/calls/{id}, and once the session has ended, its transcript with GET /v2/calls/{id}/transcript. Realtime sessions are not recorded yet, so a session has no recording.

Connect

Use the API address shown under Developers in the console: prod.voice once your company has gone live, sandbox.voice.miraiminds.co with a sandbox key. A malformed or out-of-range parameter fails the upgrade with 400 invalid_request, and error.message names the parameter. error.param names it as well when we cannot read the value at all, such as metadata that is not a JSON object. It is null for a readable value outside its limits, such as max_duration_secs outside 30 to 1800. Pipecat builds its URL as base_url + "?model=…". A base_url that already carries a query, such as …/v2/realtime?agent_id=agt_…, therefore reaches us as ?agent_id=agt_…?model=…. That is fine: we read your parameters correctly.

Authentication

Send your workspace API key in the Authorization header, as every OpenAI Realtime client does:
Clients that cannot set headers can send the key as a WebSocket subprotocol instead, openai-insecure-api-key.sk_live_…, alongside realtime. We answer with the realtime subprotocol. To run a session in one of your sub-workspaces, send the Mirai-Workspace header with its ID or external_id, beside Authorization, as on any other request.
The Realtime API is for servers. Never put your API key in a web page or a mobile app: anyone who opens the page can read it. Short-lived browser credentials are not available yet.

With Pipecat

The easiest way is MiraiRealtimeLLMService from our Pipecat package (uv add pipecat-mirai). It takes agent_id, variables and the other session parameters as arguments, sends the mirai block, handles Mirai events and reports each turn’s timing:
The current release is pipecat-mirai 0.6.2. It needs pipecat-ai 1.8.1 or newer, below 1.13. It connects to the sandbox by default. Once your company has gone live, pass base_url=PRODUCTION_REALTIME_URL, which the package also exports. Stock Pipecat 1.8.1 works unchanged too. Here is a complete microphone-and-speaker agent. Save it as pipecat_realtime.py, put on headphones and run it with uv, which installs Pipecat for you. The microphone needs PortAudio (brew install portaudio on macOS).
Two things to set in your own pipeline:
  • Send and play 16-bit PCM at 24 kHz. Set audio_in_sample_rate=24000 and audio_out_sample_rate=24000 in PipelineParams.
  • Remove your own STT, LLM and TTS services. The Realtime service replaces all three, and your transport and context aggregators stay as they are.
The agent never greets on its own in a Realtime session. Queue an LLMRunFrame (or send response.create) when you want it to speak first.

Sessions

  1. You open the socket. We check your key and balance and start a call on the realtime channel.
  2. We send session.created.
  3. You send session.update with your settings. We apply them and answer session.updated. We wait up to 1.5 seconds after session.created for this first update. If it does not arrive in time, the session starts with the agent’s settings (or the defaults), and your update is applied when it comes, apart from the language and audio formats.
  4. You stream audio with input_audio_buffer.append. We detect when the caller has finished, transcribe, run the model and stream the reply as audio. Audio you send before the session is ready is kept, up to 2 seconds of it.
  5. You close the socket, or the session reaches its maximum duration. The call ends and is billed.
You can send session.update again at any point. Most changes apply straight away; session settings lists the exceptions. Every event travels as a JSON text frame, audio included (base64). Binary frames are ignored.

Events

The events are the OpenAI Realtime GA events. These are the ones we handle. You send output_audio_buffer.clear and transcription_session.update are accepted and ignored. Any other event type returns an error with code invalid_event. We send Not supported yet
  • rate_limits.updated.
  • Interim transcripts. Transcription deltas carry final segments only, so a client that joins the deltas never prints a word twice.
  • Text-only replies. Every reply is spoken, so there is no response.output_text.delta.
  • Per-response settings in response.create.
  • Image and audio content in conversation.item.create. Only text parts are read.

Session settings

“First update only” means the session.update that arrives within 1.5 seconds of session.created. A later change to the language or an audio format is ignored, because both are fixed once the session has started. Fields we do not know are ignored. Beta-shaped fields are accepted too: input_audio_format and output_audio_format (pcm16, g711_ulaw, g711_alaw), input_audio_transcription, turn_detection, voice, speed, modalities, max_response_output_tokens and a top-level temperature. With Mirai events on, mirai.session.applied lists each ignored field and the reason. A value that is wrong, such as a string where a number belongs or a number out of range, rejects the whole update, even in a field we do not apply: audio.output.speed of 5 is an invalid_value error. You get an error event whose param names the field, and nothing in that update is applied.

The mirai block

session.mirai carries settings that the OpenAI protocol has no field for. We validate it strictly: an unknown key returns an error with code unknown_parameter, so a typo is never silently ignored.
Pipecat 1.8.1’s SessionProperties drops fields it does not know, so stock Pipecat cannot send the mirai block. MiraiRealtimeLLMService from pipecat-mirai sends it: pass language= and mirai={...}.

Languages

A Realtime session speaks Hindi, English, Hinglish, Gujarati, Tamil and Telugu; Voices says which voice speaks which. Kannada, Malayalam, Bengali and Marathi, in beta on phone and browser calls, come to Realtime sessions later (see the language roadmap). The language a session listens for is set once, when it starts:
  1. audio.input.transcription.language (or mirai.language) in your first session.update, if you send one.
  2. Otherwise the agent’s language, when you pass agent_id.
  3. Otherwise auto: we detect the caller’s language on each turn.
With agent_id, speech recognition also listens for the agent’s secondary_languages. When the session detects languages, with auto or with secondary languages, each turn reports the caller’s language, such as hi-IN or gu-IN. It is mirai_user_language in response.metadata and user_language in mirai.turn.metrics. A session with one fixed language leaves both out. The setting decides what speech recognition listens for. The language of the reply comes from your instructions, so if the agent should answer in the caller’s language, say so in the prompt.

Words to catch

A session started with agent_id listens for the agent’s keyterms: your product names, places and other words that speech recognition should get right. You can’t send keyterms from the client yet, and the mirai block has no key for them.

Tools

You can declare function tools in session.update. When the model calls one, you receive response.output_item.added with a function_call item, then response.function_call_arguments.delta and .done with the call ID, the tool name and its arguments. Run the tool, then send:
  1. conversation.item.create with an item of type function_call_output, the same call_id and your result in output.
  2. response.create, so the agent continues. The agent also continues on its own once the result arrives, so a response.create sent right after the result does not start a second reply.
Pipecat does both for functions registered with llm.register_function. We wait 10 seconds for a result by default (mirai.tool_timeout_ms changes it, from 1 to 30 seconds). After that, the model gets a tool_timeout result that asks it to tell the caller the request could not be completed, so the caller is never left in silence. With agent_id, the agent’s own tools keep running on our side, as on any call. A tool you declare with the same name as one of the agent’s replaces it for the session.

Interruptions

When our turn detection hears the caller start speaking over the agent, you receive input_audio_buffer.speech_started. Every reply still in progress ends with response.done and status cancelled (status_details.reason is turn_detected). Stop playback, then send conversation.item.truncate with audio_end_ms, how much of the reply the caller actually heard. We cut the agent’s line in the conversation to those words, so the model does not believe it said things the caller never heard. Pipecat does this for you. response.cancel stops a reply the same way, with reason client_cancelled.

Fillers and check-ins

When we start each reply for you (create_response on, the default), the agent also covers silences the way a person on a phone call does:
  • If a reply has no audio about 1.5 seconds after the caller stopped, the agent may say one short filler, such as “हाँ, एक सेकंड।”. If there is still no audio after about 4 seconds, we cancel that reply and run the model once more.
  • If the caller only checks the line, with “hello?” or “are you there?”, after the agent has been quiet for 2 seconds or more, the agent says a short line such as “हाँ, मैं यहीं हूँ।” and then repeats its last question.
A filler or check-in line arrives as a response you did not request, with its own response.created, audio and response.done. It can play while the real reply is still being generated. It is not added to the model’s context or to the call’s transcript. None of this happens before the agent’s first words, while the caller or the agent is speaking, or once the call is ending. With create_response: false or turn_detection: null, you decide when the agent speaks, and we add nothing.

Turn metrics

Every response.done carries the turn’s numbers in response.metadata. The values are strings, as the OpenAI protocol requires, so any Realtime client accepts the event:
A time that was not measured for a turn is left out. In the example the model’s time is missing, and speech synthesis ran on its fallback. Stock Pipecat 1.8.1 does not keep response.metadata when it parses response.done. MiraiRealtimeLLMService from pipecat-mirai reads these numbers for you and passes them to its on_turn_metrics handler.

Mirai events

Two events are ours rather than OpenAI’s. We send them only when you ask, by adding mirai_events=1 to the session URL, or by sending a mirai block in session.update:
Do not set mirai_events with stock Pipecat. Pipecat 1.8.1’s Realtime service raises an error on any event type it does not know, and that stops it reading the socket, so the session goes silent. MiraiRealtimeLLMService from pipecat-mirai handles them, and turns them on for you.
mirai.session.applied is sent when the session starts and after every session.update. It shows what is actually running: the voice, the formats, the transcription language, turn_detection ("mirai" or "client"), create_response, interrupt_response, max_output_tokens, the model settings in llm, tool_timeout_ms, your tools and the agent’s (server_tools), the providers serving each step, and the length of the prompt in instructions_chars. Beside it, changed lists the settings that update changed (["*"] when the session starts), and ignored lists each field we did not apply, with the reason. Abridged:
mirai.turn.metrics is sent just before each response.done, with the same numbers as its metadata as JSON numbers, plus the switches between providers during the turn:
A time that was not measured is null. user_language is there only when the session detects languages. Each entry in failovers is {"kind", "from", "to", "reason"}, for example {"kind": "tts", "from": "mira", "to": "sarvam", "reason": "no_audio"}.

Fallback

Every session runs Mirai’s own speech recognition, model and speech synthesis. If one of them is unavailable during a session, that step moves to a third-party fallback and the conversation continues:
  • Speech recognition and speech synthesis fall back to Sarvam.
  • The language model falls back to a model hosted in India.
You pay ₹1 a minute either way. The providers that served each turn are in response.done, so you can tell turns that used a fallback apart and compare their latency.

Billing

A session costs ₹1 a minute, on every plan and in the sandbox. Speech recognition, the model and the voice are all in that price. A session has no phone line, so there is no line charge, and there is no insights add-on. Prices exclude GST. See Realtime API pricing.
  • Billing starts when the session’s audio goes live and stops when the socket closes.
  • Minutes are billed in 10-second blocks, rounded up, with a one-block minimum.
  • There is no charge per token, per request or per tool call.
  • A session is a call on the realtime channel: it appears in your call list and is charged to your wallet like any other call.
A session needs enough balance to start, and ends if your balance runs out while it is open.

Errors

If a session cannot start, the WebSocket upgrade fails with an HTTP status and an OpenAI-style error body:
429 Too Many Requests
Clients that authenticate with the subprotocol cannot read an HTTP status, so they get the upgrade, one error event with the same code, and a close code of 4000 plus the status (for example 4429). During a session, errors arrive as error events:
Stock Pipecat 1.8.1 treats every error event as fatal and stops reading the socket, except conversation_already_has_active_response, response_cancel_not_active and item_retrieve_invalid_item_id. Check the values you send in session.update before you go live: a rejected update ends a Pipecat session.

Limits

  • Maximum duration. A session without agent_id runs for up to 300 seconds. With agent_id, the agent’s own limit applies. Set max_duration_secs in the URL to change it, from 30 to 1800 seconds. At the limit you get an error with code session_expired, and the socket closes.
  • Concurrency. Sessions and calls draw on the same capacity, and sessions count towards your workspace’s concurrent calls (see limits). When the platform has spare capacity, a session can start even above that number. A session that finds no free slot is refused with 429 concurrency_limit and Retry-After: 5. Sessions never queue.
  • Setup time. If setting up a session takes longer than about 8 seconds, for example because an agent’s pre-call tools are slow, the upgrade fails with 504 start_timeout.
  • First update. We wait 1.5 seconds after session.created for your first session.update.
  • Message size. A single WebSocket message can be up to 16 MB.
  • Idle connections. We ping your client every 30 seconds. A client that stops answering pings is disconnected, and the session ends.
  • Servers only. Browser and mobile clients need short-lived credentials, which are not available yet.

Pipecat

Use Mirai voices in a Pipecat pipeline that keeps its own STT and LLM.

Browser calls

Run one of your agents in a user’s browser, without your own server in the audio path.