Skip to main content
Pipecat’s RTVI (Real-Time Voice Interaction) protocol provides a standardized communication layer between clients and servers for building real-time voice and multimodal applications. It handles the synchronization of user and bot interactions, transcriptions, LLM processing, and text-to-speech delivery. This page provides an overview of RTVI from the server’s perspective and how to use it in your bot applications.

RTVI Protocol

A complete specification of the RTVI protocol for client-server communication.

​
Architecture

RTVI operates with two primary components, both of which are required for full functionality:
  1. RTVIProcessor - A frame processor residing in the pipeline that serves as the entry point for sending and receiving messages to/from the client.
  2. RTVIObserver - An observer that monitors pipeline events and translates them into client-compatible messages, handling:
    • Speaking state changes
    • Transcription updates
    • LLM responses
    • TTS events
    • Performance metrics

​
Basic Example

Here’s how to set up RTVI in your Pipecat application: 
from pipecat.processors.frameworks.rtvi import RTVIObserver, RTVIProcessor

# Create the RTVI processor
rtvi = RTVIProcessor()

# Include the RTVIProcessor in your pipeline
pipeline = Pipeline(
    [
        transport.input(),
        rtvi,
        stt,
        context_aggregator.user(),
        llm,
        tts,
        transport.output(),
        context_aggregator.assistant(),
    ]
)

# Add the RTVIObserver to your pipeline task
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        allow_interruptions=True,
        enable_metrics=True,
    ),
    observers=[RTVIObserver(rtvi)],
)

# Handle client connection
@rtvi.event_handler("on_client_ready")
async def on_client_ready(rtvi):
    # Signal bot is ready to receive messages
    await rtvi.set_bot_ready()
    # Initialize the conversation
    await task.queue_frames([LLMRunFrame()])

# Handle participant disconnection
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
    await task.cancel()

# Run the pipeline
runner = PipelineRunner()
await runner.run(task)

​
Protocol Flow

  1. Client connects and sends a client-ready message
  2. Server responds with bot-ready and initial configuration
  3. Client and server exchange real-time events:
    • Speaking state changes (user/bot-started/stopped-speaking)
    • Transcriptions (user-transcription/bot-output)
    • LLM processing (bot-llm-started/stopped, bot-llm-text, llm-function-call)
    • TTS events (bot-tts-started/stopped, bot-tts-text, bot-tts-audio)

​
Key Components

RTVIProcessor

Configure and manage RTVI services, actions, and client communication

RTVIObserver

Translate internal pipeline events to standardized client messages

​
Client Integration

RTVI is implemented in Pipecat client SDKs, providing a high-level API to interact with the protocol. Visit the Pipecat Client SDKs documentation:

Client SDKs

Learn how to implement RTVI on the client-side with our JavaScript, React, and mobile SDKs