RTVIProcessor

The RTVIProcessor manages bidirectional communication between clients and your Pipecat application. It processes client messages and handles the client-ready handshake.

Initialization

RTVIProcessor is automatically added to your pipeline when you create a PipelineWorker. Access it via worker.rtvi:

from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.worker import PipelineWorker

pipeline = Pipeline([
    transport.input(),
    stt,
    # ... other processors ...
    transport.output()
])

worker = PipelineWorker(pipeline)

# Access the processor
rtvi = worker.rtvi

To provide a custom processor (e.g., for Google RTVI):

from pipecat.services.google.rtvi import GoogleRTVIProcessor

worker = PipelineWorker(pipeline, rtvi_processor=GoogleRTVIProcessor())

To disable RTVI entirely, set enable_rtvi=False.

Properties

client_version

Returns the negotiated client protocol version as [major, minor, patch]. Defaults to [0, 0, 0] until the client sends a client-ready message.

version = rtvi.client_version
if version[0] == 2:
    # Client is using protocol v2
    pass

Readiness Protocol

Client Ready State

Clients indicate readiness by sending a client-ready message, triggering the on_client_ready event in the processor:

@rtvi.event_handler("on_client_ready")
async def on_client_ready(rtvi):
    # PipelineWorker calls set_bot_ready() for you
    # Initialize conversation
    await worker.queue_frames([...])

The processor validates the client’s RTVI protocol version during the handshake. The server protocol version is 2.2.0. Clients with major version 2 are fully compatible. Clients with major version 1 (deprecated) are still supported with the legacy bot-output format. If the client’s major version doesn’t match the server’s protocol version, an error response is sent but the connection continues. Check server logs for version compatibility warnings.

Bot Ready State

PipelineWorker calls set_bot_ready() when the client sends client-ready. It sends the client a bot-ready message containing:

RTVI protocol version
about information (the Pipecat library and version by default)
The bot’s capabilities

Function Calls

RTVIObserver reports the LLM’s function calls to the client as llm-function-call-started, llm-function-call-in-progress, and llm-function-call-stopped messages; see RTVIObserver. When the client sends an llm-function-call-result message, the processor pushes it downstream as a FunctionCallResultFrame.

Frames

Sent to RTVI:

Frame	Fields	Effect
`ErrorFrame`	`error`	Sent to the client as an `error` message. A processor reports one with `await self.push_error("...")`, which pushes it upstream to the `RTVIProcessor` at the start of the pipeline.
`RTVIServerResponseFrame`	`client_msg`, `data`, `error`	Answers a client request. When the client sends a `client-message`, the processor pushes an `RTVIClientMessageFrame` downstream; push a response carrying that frame as `client_msg`. With `error` set, the client receives an `error-response`.

Interrupt the bot with an InterruptionWorkerFrame.

Methods

An on_client_message handler receives the request as an RTVI.ClientMessage, not the RTVIClientMessageFrame that RTVIServerResponseFrame takes, so it answers with these methods (see Requesting Data from the Server).

send_server_response()

async def send_server_response(client_msg: RTVI.ClientMessage, data: Any)

Sends the client a server-response to the request client_msg, carrying data.

send_error_response()

async def send_error_response(client_msg: RTVI.ClientMessage, error: str)

Sends the client an error-response to the request client_msg, carrying error.

Custom Messaging

Server messages let you push unsolicited data from the server to the client at any time — notifications, status updates, real-time results, etc. They are distinct from server responses, which reply to a specific client request (see Requesting Information from the Server).

Sending server messages

Any FrameProcessor in the pipeline can push an RTVIServerMessageFrame. The RTVIObserver picks it up and delivers it to the client:

from pipecat.processors.frameworks.rtvi import RTVIServerMessageFrame

class MyProcessor(FrameProcessor):
    async def process_frame(self, frame, direction):
        await super().process_frame(frame, direction)
        if isinstance(frame, SomeEventFrame):
            await self.push_frame(
                RTVIServerMessageFrame(
                    data={"type": "event", "value": frame.value}
                )
            )
        await self.push_frame(frame, direction)

RTVIServerMessageFrame is a SystemFrame, so it uses the high-priority SystemFrame lane, remains ordered with other SystemFrames, and is not discarded by interruptions.

Client-side handling

The message arrives at the client with the wire format { label: "rtvi-ai", type: "server-message", data: ... }. Handle it with the onServerMessage callback:

pcClient.onServerMessage((message) => {
  console.log("Server message:", message);
  // message.data contains whatever you passed on the server
});

See Handling Custom Messages from the Server for more details and examples.

DTMF Input

The processor handles dtmf client messages for keypad input (e.g., phone menu navigation). Each dtmf message carries one or more keypad entries (0-9, *, #) as a list. The processor pushes one InputDTMFFrame per key, in order, downstream to the bot’s DTMF handling (e.g., a DTMFAggregator) exactly as a telephony transport’s rapid keypresses would arrive.

# Client message format (JSON) — protocol 2.2.0
{
  "label": "rtvi-ai",
  "type": "dtmf",
  "id": "msg-123",
  "data": {
    "buttons": ["1", "2", "#"]
  }
}

When a bot uses a DTMFAggregator, the aggregator accumulates keys and flushes them (on # or its idle timeout) into a transcription like "DTMF: 123#" that the LLM can react to.

Send File

The processor handles send-file client messages (protocol 2.2.0) that send a file to the LLM along with accompanying text. Clients gate sendFile() on the bot advertising protocol 2.2.0 or later.

# Client message format (JSON)
{
  "label": "rtvi-ai",
  "type": "send-file",
  "id": "msg-123",
  "data": {
    "content": "Summarize this document",
    "file": {
      "format": "application/pdf",
      "name": "report.pdf",
      "source": {"type": "url", "url": "https://example.com/report.pdf"}
    },
    "options": {
      "run_immediately": true,
      "audio_response": true,
      "custom_options": {}
    }
  }
}

Field	Description
`content`	Text to accompany the file.
`file.format`	MIME type of the file, e.g. `application/pdf`.
`file.name`	Optional file name.
`file.source`	Either `{"type": "bytes", "bytes": "<base64>"}` (images may also include `width` and `height`) or `{"type": "url", "url": "..."}`.
`options.run_immediately`	Interrupt the bot and run the LLM right away. Defaults to `true`.
`options.audio_response`	Speak the response. When `false`, TTS is skipped for this response. Defaults to `true`.
`options.custom_options`	Provider-specific options, such as `detail` for OpenAI or `citations` for Bedrock. Not passed to the LLM service, so they have no effect.

Image files sent as bytes are pushed downstream as a UserImageRawFrame; all other files are pushed as a UserFileRawFrame, both appended to the context.

URL sources

The URL is passed through on the frame and resolved right before each LLM completion, where the handling depends on the provider:

URLs the LLM provider can consume directly are passed straight through for the provider to fetch itself: publicly routable http:// and https:// URLs (per supported MIME type), s3:// URIs on AWS Bedrock, and gs:// URIs on Google Vertex, each resolved with the provider’s own cloud IAM.
Everything else — including the pipecat:<id> URLs minted by the development runner’s upload endpoints and non-public http(s) URLs — is downloaded by the LLM service’s FileResolver and inlined as base64; its CIDR allowlist governs which private-network URLs may be fetched.

UI Messaging

The processor handles the User Interface messages that let a server-side agent observe and drive the client GUI. Inbound UI messages from the client are turned into pipeline frames and surfaced via the on_ui_message event handler:

Wire message	Frame pushed downstream	`on_ui_message` payload
`ui-event`	`RTVIUIEventFrame`	`UIEventMessage`
`ui-snapshot`	`RTVIUISnapshotFrame`	`UISnapshotMessage`
`ui-cancel-job-group`	`RTVIUICancelJobGroupFrame`	`UICancelJobGroupMessage`

@rtvi.event_handler("on_ui_message")
async def on_ui_message(processor, message):
    # message is a UIEventMessage / UISnapshotMessage / UICancelJobGroupMessage
    ...

When you use UIWorker, PipelineWorker wires this up automatically (RTVI is enabled by default): inbound UI messages are republished onto the bus for the worker, and outbound ui-command / ui-job-group frames are translated by the RTVI Observer. You only handle on_ui_message directly when targeting the wire format from a single-LLM app without a UIWorker.

Get Started

Pipecat Server

Client SDKs

Pipecat Flows

Pipecat Cloud

CLI

Pipecat Context Hub

Initialization

Properties

client_version

Readiness Protocol

Client Ready State

Bot Ready State

Function Calls

Frames

Methods

send_server_response()

send_error_response()

Custom Messaging

Sending server messages

Client-side handling

DTMF Input

Send File

URL sources

UI Messaging

​Initialization

​Properties

​client_version

​Readiness Protocol

​Client Ready State

​Bot Ready State

​Function Calls

​Frames

​Methods

​send_server_response()

​send_error_response()

​Custom Messaging

​Sending server messages

​Client-side handling

​DTMF Input

​Send File

​URL sources

​UI Messaging

Initialization

Properties

client_version

Readiness Protocol

Client Ready State

Bot Ready State

Function Calls

Frames

Methods

send_server_response()

send_error_response()

Custom Messaging

Sending server messages

Client-side handling

DTMF Input

Send File

URL sources

UI Messaging