Skip to main content
Pipecat’s RTVI (Real-Time Voice Interaction) protocol provides a standardized communication layer between clients and servers for building real-time voice and multimodal applications. It handles the synchronization of user and bot interactions, transcriptions, LLM processing, and text-to-speech delivery. This page provides an overview of RTVI from the server’s perspective and how to use it in your bot applications.

RTVI Protocol

A complete specification of the RTVI protocol for client-server communication.

Architecture

RTVI operates with two primary components:
  1. RTVIProcessor - A frame processor residing in the pipeline that serves as the entry point for sending and receiving messages to/from the client.
  2. RTVIObserver - An observer that monitors pipeline events and translates them into client-compatible messages, handling:
    • Speaking state changes
    • Transcription updates
    • LLM responses
    • TTS events
    • Performance metrics
RTVI is enabled by default. When you create a PipelineWorker, it automatically adds RTVIProcessor to the start of your pipeline and registers an RTVIObserver. The default on_client_ready handler calls set_bot_ready() automatically.

Basic Example

With automatic RTVI setup, your pipeline code can focus on core functionality:

Customizing RTVI

You can customize RTVI behavior through PipelineWorker parameters:
To disable RTVI entirely:

Protocol Flow

  1. Client connects and sends a client-ready message
  2. Server responds with bot-ready, which includes the bot’s capabilities (audio_in, audio_out, video_in, screen_in, video_out, metrics) so clients can show only the UI that applies, such as hiding video panels for a voice-only bot. A field the bot can’t determine is omitted, and the client keeps its default.
  3. Client and server exchange real-time events:
    • Speaking state changes (user/bot-started/stopped-speaking)
    • Transcriptions (user-transcription/bot-output)
    • LLM processing (bot-llm-started/stopped, bot-llm-text, llm-function-call)
    • TTS events (bot-tts-started/stopped, bot-tts-text, bot-tts-audio)

Key Components

RTVIProcessor

Handle client messages and the client-ready handshake

RTVIObserver

Translate internal pipeline events to standardized client messages

Client Integration

RTVI is implemented in Pipecat client SDKs, providing a high-level API to interact with the protocol. Visit the Pipecat Client SDKs documentation:

Client SDKs

Learn how to implement RTVI on the client-side with our JavaScript, React, and mobile SDKs