RTVIObserver

The RTVIObserver translates Pipecat’s internal pipeline events into standardized RTVI protocol messages. It monitors frame flow through the pipeline and generates corresponding client messages based on event types.

Purpose

The RTVIObserver primarily serves to convert internal pipeline frames into to client-compatible RTVI messages. It is required for any application using RTVI as the client protocol to ensure proper communication of events such as speech start/stop, user transcript, bot output, metrics, and server messages.

Automatic Setup

RTVIObserver is automatically created and attached when you create a PipelineWorker, unless the worker has enable_rtvi=False or is bridged (where enable_rtvi defaults to off). No manual setup is required for standard usage. If you add your own RTVIProcessor to the pipeline, also pass an RTVIObserver in observers: the worker uses the pair instead of its defaults, and sets up no RTVI at all when only one of them is present. To customize the observer’s behavior, pass RTVIObserverParams to the worker:

from pipecat.pipeline.worker import PipelineWorker
from pipecat.processors.frameworks.rtvi import RTVIObserverParams

worker = PipelineWorker(
    pipeline,
    rtvi_observer_params=RTVIObserverParams(
        bot_llm_enabled=False,
        metrics_enabled=False,
    ),
)

Configuration

RTVIObserverParams accepts the following fields:

default:"True"

Indicates if bot output messages should be sent.

default:"True"

Indicates if the bot’s LLM messages should be sent. The bot-llm-text message is deprecated in favor of llm-raw-text (llm_raw_text_enabled) and will be removed in 2.0.0. This flag stays, for bot-llm-started and bot-llm-stopped.

default:"False"

Indicates if the bot’s LLM marker reports (LLMMarkerResponseFrame, e.g. from FilterIncompleteUserTurnStrategies) should be sent. A report carries the marker the LLM emitted and the raw text of the whole response, which is meant for evaluation, not for clients, so this is off by default.

default:"True"

Indicates if the bot’s TTS messages should be sent. The bot-tts-text message is deprecated in favor of tts-raw-text (tts_raw_text_enabled) and will be removed in 2.0.0. This flag stays, for bot-tts-started and bot-tts-stopped.

default:"True"

Indicates if the bot’s started/stopped speaking and interrupted messages should be sent.

default:"False"

Indicates if bot’s audio level messages should be sent.

default:"True"

Indicates if the user’s LLM input messages should be sent.

default:"True"

Indicates if the user’s started/stopped speaking messages should be sent.

default:"False"

Indicates if raw VAD user started/stopped speaking messages should be sent. These reflect the VAD signal directly, independent of turn finalization (unlike user_speaking_enabled, which a turn strategy may gate or defer).

default:"True"

Indicates if user mute started/stopped messages (user-mute-started, user-mute-stopped) should be sent.

default:"True"

deprecated

Indicates if user’s transcription messages should be sent. No replacement. It only turns off user-transcription, which is deprecated in favor of user-input and stt-raw-text. Will be removed in 2.0.0.

default:"True"

Indicates if the user’s input messages (user-input) should be sent. They report transcriptions (including interim ones), user backchannels, and the text the client sent with send-text.

default:"False"

Indicates if user’s audio level messages should be sent.

default:"True"

Indicates if metrics messages should be sent.

default:"False"

Indicates if system logs should be sent.

default:"False"

Indicates if the STT’s raw text messages (stt-raw-text) should be sent. They report each transcription, interim ones included, as the STT produced it. Only sent to clients of protocol 2.2.0 or later.

default:"False"

Indicates if the LLM’s raw text messages (llm-raw-text) should be sent. They report each chunk of text the LLM streams. Only sent to clients of protocol 2.2.0 or later.

default:"False"

Indicates if the TTS’s raw text messages (tts-raw-text) should be sent. They report the text the TTS speaks, as it is spoken, and leave out the same text types as bot output messages. Only sent to clients of protocol 2.2.0 or later.

list[FrameProcessor]

default:"[]"

Processors whose frames the observer ignores, so no RTVI messages are sent for their activity (for example, a silent evaluation LLM in a parallel branch). Add and remove sources at runtime with observer.add_ignored_source(processor) and observer.remove_ignored_source(processor).

default:"None"

List of text types to skip sending as tts/output messages. This also hides the words of skipped text that the TTS service reports with timestamps. Text of type backchannel is only sent to clients of protocol 2.2.0 or later; add "backchannel" to keep it from every client. Replaces the deprecated skip_aggregator_types.

If using this to avoid sending secure information, be sure to also disable bot_llm_enabled, and leave llm_raw_text_enabled off, to avoid leaking through LLM messages.

default:"None"

A list of tuples to transform text before sending it to the client. Each tuple should be of the form (text_type, transform_function).The preferred transform signature is:

async def my_transform(
    text: str,
    text_type: TextType | str,
    accumulated_text: str | None = None,
    remaining_text: str | None = None,
) -> BotOutputTransformResult:
    ...

When accumulated_text and remaining_text are None, the transform is being called for the full segment text (bot-output message). When they are provided, the transform is being called for a progress event and must return a BotOutputTransformResult with accumulated_text and remaining_text set, enabling word-level transforms on the client side.The legacy 2-parameter signature (text, text_type) -> str is deprecated but still works. Transforms using it will emit a DeprecationWarning at registration time.Example:

from pipecat.processors.frameworks.rtvi import (
    BotOutputTransformResult,
    RTVIObserverParams,
)

async def redact_sensitive(
    text: str,
    text_type: str,
    accumulated_text: str | None = None,
    remaining_text: str | None = None,
) -> BotOutputTransformResult:
    transformed = "XXXX-XXXX-XXXX-" + text[-4:]
    if accumulated_text is not None and remaining_text is not None:
        ratio = len(accumulated_text) / max(len(text), 1)
        split = int(ratio * len(transformed))
        return BotOutputTransformResult(
            text=transformed,
            accumulated_text=transformed[:split],
            remaining_text=transformed[split:],
        )
    return BotOutputTransformResult(text=transformed)

async def uppercase(
    text: str,
    text_type: str,
    accumulated_text: str | None = None,
    remaining_text: str | None = None,
) -> BotOutputTransformResult:
    if accumulated_text is not None and remaining_text is not None:
        return BotOutputTransformResult(
            text=text.upper(),
            accumulated_text=accumulated_text.upper(),
            remaining_text=remaining_text.upper(),
        )
    return BotOutputTransformResult(text=text.upper())

bot_output_transforms = [
    ("credit_card", redact_sensitive),  # Only for 'credit_card' type
    ("*", uppercase),  # For all types, make uppercase
]

worker = PipelineWorker(
    pipeline,
    rtvi_observer_params=RTVIObserverParams(bot_output_transforms=bot_output_transforms),
)

default:"0.15"

How often audio levels should be sent if enabled.

Dict[str, RTVIFunctionCallReportLevel]

default:"{\"*\": RTVIFunctionCallReportLevel.NONE}"

Controls what information is exposed in function call lifecycle events (llm-function-call-started, llm-function-call-in-progress, llm-function-call-stopped). Maps function names to security levels, where "*" sets the default for unlisted functions.Levels:

DISABLED: No events emitted for this function
NONE: Events with tool_call_id only (most secure when events are needed). llm-function-call-started carries no tool_call_id, so its data is empty at this level
NAME: Adds function name to events
ARGUMENTS: Adds function name and arguments, no results
FULL: Adds function name, arguments, and results. A falsy result such as {}, [] or 0 is reported as returned

from pipecat.processors.frameworks.rtvi import (
    RTVIFunctionCallReportLevel,
    RTVIObserverParams,
)

worker = PipelineWorker(
    pipeline,
    rtvi_observer_params=RTVIObserverParams(
        function_call_report_level={
            "*": RTVIFunctionCallReportLevel.NONE,
            "get_weather": RTVIFunctionCallReportLevel.FULL,
        },
    ),
)

Frame Translation

The observer maps Pipecat’s internal frames to RTVI protocol messages:

Pipeline Frame	RTVI Message
Speech Events
`UserStartedSpeakingFrame`	`UserStartedSpeakingMessage`
`UserStoppedSpeakingFrame`	`UserStoppedSpeakingMessage`
`VADUserStartedSpeakingFrame`	`VADUserStartedSpeakingMessage`
`VADUserStoppedSpeakingFrame`	`VADUserStoppedSpeakingMessage`
`BotStartedSpeakingFrame`	`BotStartedSpeakingMessage`
`BotStoppedSpeakingFrame`	`BotStoppedSpeakingMessage`
`InterruptionFrame`	`BotInterruptedMessage` (`bot-interrupted`, when `bot_speaking_enabled`)
User Mute
`UserMuteStartedFrame`	`UserMuteStartedMessage`
`UserMuteStoppedFrame`	`UserMuteStoppedMessage`
Transcription
`TranscriptionFrame`	`UserTranscriptionMessage(final=true)` (deprecated)
`InterimTranscriptionFrame`	`UserTranscriptionMessage(final=false)` (deprecated)
`TranscriptionFrame`	`stt-raw-text` (`final=true`, when enabled)
`InterimTranscriptionFrame`	`stt-raw-text` (`final=false`, when enabled)
`TranscriptionFrame`	`user-input` (`input_type="transcription"`, `final=true`)
`InterimTranscriptionFrame`	`user-input` (`input_type="transcription"`, `final=false`)
`UserBackchannelFrame`	`user-input` (`input_type="backchannel"`)
`RTVISendTextFrame`	`user-input` (`input_type="chat"`)
Bot Output
`AggregatedTextFrame`	`BotOutputMessage`
LLM Processing
`LLMFullResponseStartFrame`	`BotLLMStartedMessage`
`LLMFullResponseEndFrame`	`BotLLMStoppedMessage`
`LLMTextFrame`	`BotLLMTextMessage` (deprecated)
`LLMTextFrame`	`llm-raw-text` (when enabled)
`LLMMarkerResponseFrame`	`bot-llm-marker` (when enabled)
TTS Events
`TTSStartedFrame`	`BotTTSStartedMessage`
`TTSStoppedFrame`	`BotTTSStoppedMessage`
`TTSTextFrame`	`BotTTSTextMessage` (deprecated)
`TTSTextFrame`	`tts-raw-text` (when enabled)
Function Calls
`FunctionCallsStartedFrame`	`llm-function-call-started`
`FunctionCallInProgressFrame`	`llm-function-call-in-progress`
`FunctionCallResultFrame`	`llm-function-call-stopped` (final results only)
`FunctionCallCancelFrame`	`llm-function-call-stopped` (`cancelled=true`)
`ExternalFunctionCall*Frame`	Same messages as the matching `FunctionCall*Frame`
Context/Metrics
`LLMContextFrame`	`UserLLMTextMessage`
`MetricsFrame`	`MetricsMessage`
`RTVIServerMessageFrame`	`ServerMessage`
`RTVIServerResponseFrame`	`server-response`, or `error-response` when `error` is set
User Interface
`RTVIUICommandFrame`	`UICommandMessage` (`ui-command`)
`RTVIUIJobGroupFrame`	`UIJobGroupMessage` (`ui-job-group`)

The User Interface frames are emitted by PipelineWorker when a UIWorker sends a command or runs a job group; the observer translates them into the outbound ui-command / ui-job-group messages the client renders.

Frames

Sent to the observer:

Frame	Fields	Effect
`RTVIConfigureObserverFrame`	`function_call_report_level`, `vad_user_speaking_enabled`, `bot_llm_marker_enabled`	Changes the matching `RTVIObserverParams` fields at runtime. Fields left as `None` keep their value.

Get Started

Standards

Pipecat Server

Client SDKs

Pipecat Flows

Pipecat Cloud

CLI

Pipecat Context Hub

Purpose

Automatic Setup

Configuration

Frame Translation

Frames

​Purpose

​Automatic Setup

​Configuration

​Frame Translation

​Frames

Purpose

Automatic Setup

Configuration

Frame Translation

Frames