RTVI Protocol
A complete specification of the RTVI protocol for client-server communication.
Architecture
RTVI operates with two primary components:- RTVIProcessor - A frame processor residing in the pipeline that serves as the entry point for sending and receiving messages to/from the client.
-
RTVIObserver - An observer that monitors pipeline events and translates them into client-compatible messages, handling:
- Speaking state changes
- Transcription updates
- LLM responses
- TTS events
- Performance metrics
RTVI is enabled by default. When you create a
PipelineWorker, it
automatically adds RTVIProcessor to the start of your pipeline and registers
an RTVIObserver. The default on_client_ready handler calls
set_bot_ready() automatically.Basic Example
With automatic RTVI setup, your pipeline code can focus on core functionality:Customizing RTVI
You can customize RTVI behavior throughPipelineWorker parameters:
Protocol Flow
- Client connects and sends a
client-readymessage - Server responds with
bot-ready, which includes the bot’scapabilities(audio_in,audio_out,video_in,screen_in,video_out,metrics) so clients can show only the UI that applies, such as hiding video panels for a voice-only bot. A field the bot can’t determine is omitted, and the client keeps its default. - Client and server exchange real-time events:
- Speaking state changes (
user/bot-started/stopped-speaking) - Transcriptions (
user-transcription/bot-output) - LLM processing (
bot-llm-started/stopped,bot-llm-text,llm-function-call) - TTS events (
bot-tts-started/stopped,bot-tts-text,bot-tts-audio)
- Speaking state changes (
Key Components
RTVIProcessor
Handle client messages and the client-ready handshake
RTVIObserver
Translate internal pipeline events to standardized client messages
Client Integration
RTVI is implemented in Pipecat client SDKs, providing a high-level API to interact with the protocol. Visit the Pipecat Client SDKs documentation:Client SDKs
Learn how to implement RTVI on the client-side with our JavaScript, React, and
mobile SDKs