> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# RTVI

> The RTVI standard for real-time voice interaction: the open protocol connecting Pipecat clients and servers.

The RTVI (Real-Time Voice and Video Inference) standard defines a set of message types and structures sent between clients and servers. It is designed to facilitate real-time interactions between clients and AI applications that require voice, video, and text communication. It provides a consistent framework for building applications that can communicate with AI models and the backends running those models in real-time.

> [!NOTE]
>
> This page documents version 2.3.0 of the RTVI standard (Pipecat 1.15.0,
> client-js 1.16.0). See [Protocol Versions](#protocol-versions) for what each
> version changed.

## Key Features

> **Connection Management**
>
> RTVI provides a flexible connection model that allows clients to connect to
> AI services and coordinate state.

> **Transcriptions**
>
> The standard includes built-in support for real-time transcription of audio
> streams.

> **Client-Server Messaging**
>
> The standard defines a messaging protocol for sending and receiving messages
> between clients and servers, allowing for efficient communication of
> requests and responses.

> **Advanced LLM Interactions**
>
> The standard supports advanced interactions with large language models
> (LLMs), including context management, function call handling, and search
> results.

> **Service-Specific Insights**
>
> RTVI supports events to provide insight into the input/output and state for
> typical services that exist in speech-to-speech workflows.

> **Metrics and Monitoring**
>
> RTVI provides mechanisms for collecting metrics and monitoring the
> performance of server-side services.

> **User Interface**
>
> RTVI lets a server-side agent observe and drive the client's GUI through
> accessibility snapshots, UI events, and commands.

## Terms

* **Client**: The front-end application or user interface that interacts with the RTVI server.
* **Server**: The backend service that runs the AI framework and processes requests from the client.
* **User**: The end user interacting with the client application.
* **Bot**: The AI interacting with the user, technically an amalgamation of a large language model (LLM) and a text-to-speech (TTS) service.

## RTVI Message Format

The messages defined as part of the RTVI protocol adhere to the following format:

```json theme={null}
{
    "id": string,
    "label": "rtvi-ai",
    "type": string,
    "data": unknown
}
```

**id** `string`

A unique identifier for the message, used to correlate requests and responses.

**label** `string` *required* (default: rtvi-ai)

A label that identifies this message as an RTVI message. This field is
required and should always be set to `'rtvi-ai'`.

**type** `string` *required*

The type of message being sent. This field is required and should be set to
one of the predefined RTVI message types listed below.

**data** `unknown`

The payload of the message, which can be any data structure relevant to the
message type.

If the server receives a message with an unknown `type`, or a message whose `data` fails validation, it replies with an [`error-response`](#error-response-🤖) carrying the message's `id`.

## Protocol Versions

The client sends its protocol version in `client-ready`, and the server sends its own in `bot-ready`. A server serves each client in the format of the client's version:

* A client on the server's major version is fully compatible.
* A client on any 1.x version is still served, with the 1.x `bot-output` format.
* Any other client version, or a missing or malformed one, gets an `error-response` warning that compatibility issues may occur. The connection still proceeds.

Messages added in a minor version are only sent to clients of that version or later, and clients should only send a new client message when the bot's `bot-ready` advertises a version that supports it.

| Version | Pipecat | client-js | Changes |
| - | - | - | - |
| 2.3.0 | 1.15.0 | 1.16.0 | `delivery` in `bot-output`; `expression_sound` and `expression_pause` output types; `perception` in `user-input`. Carries [RCS](https://docs.pipecat.ai/api-reference/standards/real-time-context-standard.md) 0.1. |
| 2.2.0 | 1.13.0 | 1.14.0 | `send-file` client message; `capabilities` in `bot-ready`; `stt-raw-text`, `llm-raw-text`, and `tts-raw-text`; `backchannel` text type. |
| 2.1.0 | 1.6.0 | 1.13.0 | `dtmf` carries a `buttons` list instead of a single `button`. |
| 2.0.0 | 1.4.0 | 1.12.0 | `bot-output` adds `will_be_spoken`, `spoken_status`, `spoken_progress`, and `segment_id`, and reports word-level progress through them instead of separate `word` messages. |

## RTVI Message Types

Following the above format, this section describes the various message types defined by the RTVI standard. Each message type has a specific purpose and structure, allowing for clear communication between clients and servers.

> [!TIP]
>
> Each message type below includes either a 🤖 or 🏄 emoji to denote whether the
> message is sent from the bot (🤖) or client (🏄).

### Connection Management

#### client-ready 🏄

Indicates that the client is ready to receive messages and interact with the server. Typically sent after the transport media channels have connected.

* **type**: `'client-ready'`
* **data**:
  * **version**: `string`

    The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations.

  * **about**: `AboutClient Object`

    An object containing information about the client, such as its rtvi-version, client library, and any other relevant metadata. The `AboutClient` object follows this structure:

    **AboutClient**

    **library** `string` *required*

    **library\_version** `string`

    **platform** `string`

    **platform\_version** `string`

    **platform\_details** `any`

    Any platform-specific details that may be relevant to the server. This
    could include information about the browser, operating system, or any
    other environment-specific data needed by the server. This field is
    optional and open-ended, so please be mindful of the data you include
    here and any security concerns that may arise from exposing sensitive or
    personal-identifiable information.

#### bot-ready 🤖

Indicates that the bot is ready to receive messages and interact with the client. Typically sent after the transport media channels have connected.

* **type**: `'bot-ready'`
* **data**:
  * **version**: `string`

    The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations.

  * **about**: `any` (Optional)

    An object containing information about the server or bot. Its structure and value are both undefined by default. This provides flexibility to include any relevant metadata your client may need to know about the server at connection time, without any built-in security concerns. Please be mindful of the data you include here and any security concerns that may arise from exposing sensitive information.

  * **capabilities**: `BotCapabilities Object` (Optional)

    > [!NOTE]
    >
    > Introduced in RTVI version 2.2.0.

    What the bot does in this session, so the client can show only the UI that applies, such as hiding video panels for a voice-only bot. Each field is optional: `true` means the bot does it, `false` means it doesn't, and a missing field means unknown, in which case the client should keep its default behavior.

    **BotCapabilities**

    **audio\_in** `boolean`

    Whether the bot receives the user's audio.

    **audio\_out** `boolean`

    Whether the bot sends audio to the user.

    **video\_in** `boolean`

    Whether the bot receives the user's video.

    **screen\_in** `boolean`

    Whether the bot receives the user's screen share.

    **video\_out** `boolean`

    Whether the bot sends video to the user.

    **metrics** `boolean`

    Whether the bot reports metrics.

#### disconnect-bot 🏄

Indicates that the client wishes to disconnect from the bot. Typically used when the client is shutting down or no longer needs to interact with the bot. Note: Disconnects should happen automatically when either the client or bot disconnects from the transport, so this message is intended for the case where a client may want to remain connected to the transport but no longer wishes to interact with the bot.

* **type**: `'disconnect-bot'`
* **data**: `undefined`

#### error 🤖

Indicates an error occurred during bot initialization or runtime.

* **type**: `'error'`
* **data**:
  * **error**: `string`

    Description of the error.

  * **fatal**: `boolean`

    Indicates if the error is fatal to the session.

#### system-log 🤖

> [!NOTE]
>
> Disabled by default; enable with
> `RTVIObserverParams(system_logs_enabled=True)`.

A server log line, forwarded to the client for debugging.

* **type**: `'system-log'`
* **data**:
  * **text**: `string`

    The log message.

### Speaking and Transcription

#### user-started-speaking 🤖

Emitted when the user begins speaking

* **type**: `'user-started-speaking'`
* **data**: None

#### user-stopped-speaking 🤖

Emitted when the user stops speaking

* **type**: `'user-stopped-speaking'`
* **data**: None

#### vad-user-started-speaking 🤖

> [!NOTE]
>
> Disabled by default; enable with
> `RTVIObserverParams(vad_user_speaking_enabled=True)`.

Emitted when the VAD (Voice Activity Detection) detects the user started speaking. This is the raw VAD signal, emitted independently of turn finalization (unlike `user-started-speaking`, which a turn strategy may gate or defer).

* **type**: `'vad-user-started-speaking'`
* **data**: None

#### vad-user-stopped-speaking 🤖

> [!NOTE]
>
> Disabled by default; enable with
> `RTVIObserverParams(vad_user_speaking_enabled=True)`.

Emitted when the VAD (Voice Activity Detection) detects the user stopped speaking. This is the raw VAD signal, emitted independently of turn finalization (unlike `user-stopped-speaking`, which a turn strategy may gate or defer).

* **type**: `'vad-user-stopped-speaking'`
* **data**: None

#### bot-started-speaking 🤖

Emitted when the bot begins speaking

* **type**: `'bot-started-speaking'`
* **data**: None

#### bot-stopped-speaking 🤖

Emitted when the bot stops speaking

* **type**: `'bot-stopped-speaking'`
* **data**: None

#### bot-interrupted 🤖

> [!NOTE]
>
> Introduced in Pipecat 1.4.0.

Emitted when the bot's in-flight output is cut off, whether by the user barging in or by a programmatic interruption (e.g. a `send-text` with `run_immediately`). The client can drop whatever the bot was in the middle of saying. Sent when the server's `bot_speaking_enabled` option is on, as it is by default.

* **type**: `'bot-interrupted'`
* **data**: None

#### user-audio-level 🤖

> [!NOTE]
>
> Disabled by default; enable with
> `RTVIObserverParams(user_audio_level_enabled=True)`.

The level of the user's audio as the server receives it, sent periodically (every 0.15 seconds by default).

* **type**: `'user-audio-level'`
* **data**:
  * **value**: `number`

    The audio level.

#### bot-audio-level 🤖

> [!NOTE]
>
> Disabled by default; enable with
> `RTVIObserverParams(bot_audio_level_enabled=True)`.

The level of the bot's audio output, sent periodically (every 0.15 seconds by default).

* **type**: `'bot-audio-level'`
* **data**:
  * **value**: `number`

    The audio level.

#### user-mute-started 🤖

> [!NOTE]
>
> Introduced in RTVI version 1.2.0 (Pipecat 0.0.102, client-js 1.6.0).

Emitted when the server begins ignoring audio from the client (server-side muting). The client should continue sending audio normally but may want to show an indication to the user that their input is not being processed.

* **type**: `'user-mute-started'`
* **data**: None

#### user-mute-stopped 🤖

> [!NOTE]
>
> Introduced in RTVI version 1.2.0 (Pipecat 0.0.102, client-js 1.6.0).

Emitted when the server stops ignoring audio from the client (server-side muting ends). The client can update its UI to indicate that the user's input is being processed again.

* **type**: `'user-mute-stopped'`
* **data**: None

#### user-transcription 🤖

> [!WARNING]
>
> DEPRECATED in Pipecat 1.13.0, and will be removed in 2.0.0. Use `user-input`,
> or [`stt-raw-text`](#stt-raw-text-🤖) for the STT's raw output.

Real-time transcription of user speech, including both partial and final results.

* **type**: `'user-transcription'`
* **data**:
  * **text**: `string`

    The transcribed text of the user.

  * **final**: `boolean`

    Indicates if this is a final transcription or a partial result.

  * **timestamp**: `string`

    The timestamp when the transcription was generated.

  * **user\_id**: `string`

    Identifier for the user who spoke.

#### user-input 🤖

What the user said or typed. The observer sends one for each transcription (before the matching `user-transcription`), for each user backchannel (a short acknowledgment such as "mhm" said while the bot talks, which doesn't interrupt the bot or reach the LLM), and as an acknowledgment of each `send-text` message.

* **type**: `'user-input'`
* **data**:
  * **text**: `string`

    What the user said or typed.

  * **input\_type**: `'transcription' | 'chat' | 'backchannel'`

    The kind of input: speech transcribed by the STT, text the client sent, or a backchannel.

  * **timestamp**: `string`

    The timestamp when the user said or sent it.

  * **final**: `boolean`

    Whether the text is final. Only an interim transcription is not.

  * **user\_id**: `string` (optional)

    Identifier for the user who spoke or typed. For `chat`, only set when the transport knows the sender, such as Daily and LiveKit.

  * **msg\_id**: `string` (optional)

    For `chat`, the id of the client's `send-text` message.

  * **perception**: `array` (2.3.0+, optional)

    For a transcription, how the user said it, when the server reads it. Each item is a signal of a [perception](https://docs.pipecat.ai/api-reference/standards/real-time-context-standard.md#perception) reading:

    * **name**: `string`

      What is read, such as `"tone"`.

    * **labels**: `array`

      For a signal read as labels, its likeliest labels, likeliest first, each with a `label` (`string`) and a `confidence` (`number`) from 0 to 1. Empty for a signal read as a value.

    * **value**: `number` (optional)

      For a signal read as a value, the measure, such as a speech rate.

#### bot-output 🤖

The best-effort representation of the bot's output text, including both spoken and unspoken text. In addition to transcriptions of spoken text, this message type may also include text that the bot outputs but does not speak (e.g., text sent to the client for display purposes only). Along with the text, this event includes an `output_type` field to indicate what the text represents (e.g. "sentence", "word", "code", "url") and fields describing whether and how far the bot has spoken it.

The bot's output is split into segments, such as sentences, each with its own `segment_id`. For clients of protocol 2.0.0 or later, a spoken segment is reported as it is spoken:

* If the TTS service reports word timestamps, the server sends the segment once with `spoken_status: "new"` before it is spoken, then again as each word is spoken, with `spoken_status: "in-progress"` and the last with `"completed"`. Every message carries the segment's full `text` and the same `segment_id`.
* Otherwise, the server sends the segment once, with `spoken_status: "completed"`, after it is spoken.

Text the bot won't speak is sent once, with `will_be_spoken: false` and `spoken_status: "new"`. Clients of protocol 2.0.0 or later don't receive separate `bot-output` messages for individual words (`output_type` `"word"`); word-level progress is in `spoken_progress` instead. Clients on 1.x receive each spoken word as its own message and should read `spoken`.

* **type**: `'bot-output'`
* **data**:
  * **text**: `string`

    The output text from the bot.

  * **output\_type**: `string`

    Indicates what kind of text this is (e.g., "sentence", "word", "code", "url"). "sentence", "word", and "backchannel" are reserved output types defined by the RTVI standard. Other output types may be defined by custom text aggregators used by the server.

    A short acknowledgment the bot says, such as "Mm-hmm.", has type `backchannel`. It is only sent to clients of protocol 2.2.0 or later.

    A sound or pause the bot performs in [expressive conversation](https://docs.pipecat.ai/pipecat/features/expressive-conversation.md) has type `expression_sound` or `expression_pause`. Its `text` is the sound's name, such as `"laugh"`, or the pause's length in milliseconds, such as `"600"`. Each is a segment of its own, sent once as it's performed, with `spoken_status: "completed"`. They are only sent to clients of protocol 2.3.0 or later.

  * **text\_type**: `string` (deprecated)

    Deprecated. Use `output_type` instead, which carries the same value. Will be removed in 2.0.0.

  * **aggregated\_by**: `string` (deprecated)

    Deprecated. Use `output_type` instead, which carries the same value. Will be removed in 2.0.0.

  * **segment\_id**: `number` (optional)

    Identifies the segment this text belongs to, so a client can match a segment's `"new"`, `"in-progress"`, and `"completed"` messages.

  * **will\_be\_spoken**: `boolean` (2.0.0+)

    Indicates if the TTS will speak this text.

  * **spoken\_status**: `'new' | 'in-progress' | 'completed'` (2.0.0+)

    How far the bot has spoken the segment. Always `'new'` when `will_be_spoken` is `false`.

  * **spoken\_progress**: `object` (2.0.0+, optional)

    How much of the segment has been spoken. Only present when `will_be_spoken` is `true`.

    * **accumulated\_text**: `string`

      The text already spoken, including the current word.

    * **remaining\_text**: `string`

      The text not yet spoken.

  * **delivery**: `string[]` (2.3.0+, optional)

    In [expressive conversation](https://docs.pipecat.ai/pipecat/features/expressive-conversation.md), the delivery of the text just spoken, by its [universal names](https://docs.pipecat.ai/api-reference/standards/real-time-context-standard.md#reporting-delivery), such as `["warm", "emphasis"]` for a stressed word in a warm delivery. Empty for the voice's baseline. It changes within a segment where the delivery does, so a segment's `"new"` message has none.

  * **spoken**: `boolean` (1.x)

    Indicates if this text was spoken by the bot. Clients of protocol 2.0.0 or later should read `will_be_spoken` and `spoken_status` instead.

#### bot-transcription 🤖

> [!WARNING]
>
> DEPRECATED in favor of `bot-output` in Pipecat version 0.0.95 and client-js
> version 1.5.0

Transcription of the bot's speech. Note: This protocol currently does not match the user transcription format to support real-time timestamping for bot transcriptions. Rather, the event is typically sent for each sentence of the bot's response. This difference is currently due to limitations in TTS services which mostly do not support (or support well), accurate timing information. If/when this changes, this protocol may be updated to include the necessary timing information. For now, if you want to attempt real-time transcription to match your bot's speaking, you can try using the `bot-tts-text` message type.

* **type**: `'bot-transcription'`
* **data**:
  * **text**: `string`

    The transcribed text from the bot, typically aggregated at a per-sentence level.

### Client-Server Messaging

#### server-message 🤖

An arbitrary message sent from the server to the client. This can be used for custom interactions or commands. This message may be coupled with the `client-message` message type to handle responses from the client.

* **type**: `'server-message'`
* **data**: `any`

  The `data` can be any JSON-serializable object, formatted according to your own specifications.

#### client-message 🏄

An arbitrary message sent from the client to the server. This can be used for custom interactions or commands. This message may be coupled with the `server-response` message type to handle responses from the server.

* **type**: `'client-message'`
* **data**:

  * **t**: `string`
  * **d**: `unknown` (optional)

  The data payload should contain a `t` field indicating the type of message and an optional `d` field containing any custom, corresponding data needed for the message.

#### server-response 🤖

A message sent from the server to the client in response to a `client-message`. **IMPORTANT**: The `id` should match the `id` of the original `client-message` to correlate the response with the request.

* **type**: `'server-response'`
* **data**:

  * **t**: `string`
  * **d**: `unknown` (optional)

  The data payload should contain a `t` field indicating the type of message and an optional `d` field containing any custom, corresponding data needed for the message.

#### error-response 🤖

Error response to a specific client message, such as a `client-message` the application answers with an error, a message of an unknown `type`, or a message that failed validation. **IMPORTANT**: The `id` should match the `id` of the original client message to correlate the response with the request.

* **type**: `'error-response'`
* **data**:
  * **error**: `string`

    Description of the error.

### Advanced LLM Interactions

#### send-text 🏄

A message sent from the client to the server to send text input to the LLM, appended to the user's context.

* **type**: `'send-text'`
* **data**:
  * **content**: `string`

    The text content to be appended to the user context.

  * **options**: `object` (optional)
    * **run\_immediately**: `boolean` (optional)

      If `true`, the pipeline should be interrupted and the LLM should process the input immediately after appending it to the context. Defaults to `true`.

    * **audio\_response**: `boolean` (optional)

      If `false`, the bot should bypass the TTS so that the bot does not respond to the text in audio. Defaults to `true`.

#### send-file 🏄

> [!NOTE]
>
> Introduced in RTVI version 2.2.0 (Pipecat 1.13.0, client-js 1.14.0).

A message sent from the client to the server to send a file to the LLM along with accompanying text, both appended to the user's context. Clients should only send this message when the bot advertises protocol version 2.2.0 or later in its `bot-ready` message.

* **type**: `'send-file'`
* **data**:
  * **content**: `string`

    The text to accompany the file.

  * **file**: `object`
    * **format**: `string`

      The file's MIME type, e.g. `application/pdf`.

    * **name**: `string` (optional)

      The file's name.

    * **source**: `object`

      Either `{ "type": "bytes", "bytes": "<base64>" }` (images may also include `width` and `height`), or `{ "type": "url", "url": "..." }`. The URL may be one returned by a file upload endpoint (e.g. `pipecat:<id>` from the Pipecat development runner), a cloud-storage URI (`s3://`, `gs://`), or a plain `http(s)` URL.

  * **options**: `object` (optional)
    * **run\_immediately**: `boolean` (optional)

      If `true`, the pipeline should be interrupted and the LLM should process the input immediately after appending it to the context. Defaults to `true`.

    * **audio\_response**: `boolean` (optional)

      If `false`, the bot should bypass the TTS so that the bot does not respond to the file in audio. Defaults to `true`.

    * **custom\_options**: `object` (optional)

      Provider-specific options passed through with the file, e.g. `detail` for OpenAI or `citations` for Bedrock.

See the [RTVIProcessor](https://docs.pipecat.ai/api-reference/server/rtvi/rtvi-processor.md#send-file) documentation for how Pipecat handles this message server-side.

#### dtmf 🏄

> [!NOTE]
>
> Introduced in Pipecat 1.5.0. RTVI version 2.1.0 (Pipecat 1.6.0, client-js
> 1.13.0) replaced the single `button` field with `buttons`.

Keypad input from the client, such as for navigating a phone menu. The server handles the keys in order, exactly as it would rapid keypresses from a telephony transport.

* **type**: `'dtmf'`
* **data**:
  * **buttons**: `Array<string>`

    One or more keypad entries: `'0'`-`'9'`, `'*'`, or `'#'`. Must contain at least one entry.

See the [RTVIProcessor](https://docs.pipecat.ai/api-reference/server/rtvi/rtvi-processor.md#dtmf-input) documentation for how Pipecat handles this message server-side.

#### llm-function-call-started 🤖

> [!NOTE]
>
> Introduced in RTVI version 1.2.0 (Pipecat 0.0.102, client-js 1.6.0).

Indicates that a function call has been initiated by the LLM. The amount of metadata included in this event, and in `llm-function-call-in-progress` and `llm-function-call-stopped`, is controlled by the server's `function_call_report_level` configuration, set per function:

| Level | Metadata included |
| - | - |
| `DISABLED` | No function call events are sent for the function. |
| `NONE` | `tool_call_id` on `in-progress` and `stopped`, plus `cancelled` on `stopped`; `started` carries none. The default. |
| `NAME` | Adds `function_name`. |
| `ARGUMENTS` | Adds `function_name` and `arguments`. |
| `FULL` | Adds `function_name`, `arguments`, and `result`. |

* **type**: `'llm-function-call-started'`
* **data**:
  * **function\_name**: `string` (optional)

    Name of the function being called. Only included if the server's report level is `NAME`, `ARGUMENTS`, or `FULL`.

#### llm-function-call-in-progress 🤖

> [!NOTE]
>
> Introduced in RTVI version 1.2.0 (Pipecat 0.0.102, client-js 1.6.0).

Indicates that a function call is in progress. This is the successor to the deprecated `llm-function-call` message and is the event that triggers registered `FunctionCallHandler`s when a `function_name` is present.

* **type**: `'llm-function-call-in-progress'`
* **data**:
  * **function\_name**: `string` (optional)

    Name of the function being called. Only included if the server's report level is `NAME`, `ARGUMENTS`, or `FULL`.

  * **tool\_call\_id**: `string`

    Unique identifier for this function call.

  * **arguments**: `Record<string, unknown>` (optional)

    Arguments passed to the function. Only included if the server's report level is `ARGUMENTS` or `FULL`.

#### llm-function-call-stopped 🤖

> [!NOTE]
>
> Introduced in RTVI version 1.2.0 (Pipecat 0.0.102, client-js 1.6.0).

Indicates that a function call has completed or been cancelled.

* **type**: `'llm-function-call-stopped'`
* **data**:
  * **function\_name**: `string` (optional)

    Name of the function that was called. Only included if the server's report level is `NAME`, `ARGUMENTS`, or `FULL`.

  * **tool\_call\_id**: `string`

    Identifier matching the original function call.

  * **cancelled**: `boolean`

    Indicates whether the function call was cancelled before completing.

  * **result**: `unknown` (optional)

    The result of the function call, if available. Only included if the server's report level is `FULL`.

#### llm-function-call 🤖

> [!WARNING]
>
> DEPRECATED in favor of `llm-function-call-in-progress` in Pipecat version
> 0.0.102 and client-js version 1.6.0

A function call request from the LLM, sent from the bot to the client. Note that for most cases, an LLM function call will be handled completely server-side. However, in the event that the call requires input from the client or the client needs to be aware of the function call, this message/response schema is required.

* **type**: `'llm-function-call'`
* **data**:
  * **function\_name**: `string`

    Name of the function to be called.

  * **tool\_call\_id**: `string`

    Unique identifier for this function call.

  * **args**: `Record<string, unknown>`

    Arguments to be passed to the function.

#### llm-function-call-result 🏄

The result of the function call requested by the LLM, returned from the client.

* **type**: `'llm-function-call-result'`
* **data**:
  * **function\_name**: `string`

    Name of the called function.

  * **tool\_call\_id**: `string`

    Identifier matching the original function call.

  * **arguments**: `Record<string, unknown>`

    Arguments that were passed to the function.

  * **result**: `Record<string, unknown> | string`

    The result returned by the function.

#### bot-llm-search-response 🤖

Search results from the LLM's knowledge base.

> [!NOTE]
>
> Currently, Google Gemini is the only LLM that supports built-in search.
> However, we expect other LLMs to follow suit, which is why this message type
> is defined as part of the RTVI standard. As more LLMs add support for this
> feature, the format of this message type may evolve to accommodate
> discrepancies.

* **type**: `'bot-llm-search-response'`
* **data**:
  * **search\_result**: `string` (optional)

    Raw search result text.

  * **rendered\_content**: `string` (optional)

    Formatted version of the search results.

  * **origins**: `Array<Origin Object>`

    Source information and confidence scores for search results. The `Origin Object` follows this structure:

    ```json theme={null}
    {
        "site_uri": string (optional),
        "site_title": string (optional),
        "results": Array<{
            "text": string,
            "confidence": number[]
        }>
    }
    ```

**Example:**

```json theme={null}
"id": undefined
"label": "rtvi-ai"
"type": "bot-llm-search-response"
"data": {
    "origins": [
    {
        "results": [
            {
                "confidence": [0.9881149530410768],
                "text": "* Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm."
            },
            {
                "confidence": [0.9692034721374512],
                "text": "* A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm."
            }
        ],
        "site_title": "vanderbilt.edu",
        "site_uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHwif83VK9KAzrbMSGSBsKwL8vWfSfC9pgEWYKmStHyqiRoV1oe8j1S0nbwRg_iWgqAr9wUkiegu3ATC8Ll-cuE-vpzwElRHiJ2KgRYcqnOQMoOeokVpWqi"
    },
    {
        "results": [
            {
                "confidence": [0.6554043292999268],
                "text": "In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields."
            }
        ],
        "site_title": "wikipedia.org",
        "site_uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQESbF-ijx78QbaglrhflHCUWdPTD4M6tYOQigW5hgsHNctRlAHu9ktfPmJx7DfoP5QicE0y-OQY1cRl9w4Id0btiFgLYSKIm2-SPtOHXeNrAlgA7mBnclaGrD7rgnLIbrjl8DgUEJrrvT0CKzuo"
    }],
    "rendered_content": "<style>\n.container ... </div>\n</div>\n",
    "search_result": "Several events are happening at Vanderbilt University:\n\n* Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm.\n* A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm.\n\n  In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields.  For the most recent news, you should check Vanderbilt's official news website.\n"
}
```

### Service-Specific Insights

#### stt-raw-text 🤖

Each transcription, including interim ones, as the STT produced it. Sent to clients of protocol 2.2.0 or later, when enabled on the server.

* **type**: `'stt-raw-text'`
* **data**:
  * **text**: `string`

    The transcribed text.

  * **final**: `boolean`

    Indicates if this is a final transcription or a partial result.

  * **timestamp**: `string`

    The timestamp when the transcription was generated.

  * **user\_id**: `string`

    Identifier for the user who spoke.

#### bot-llm-started 🤖

Indicates LLM processing has begun

* **type**: `bot-llm-started`
* **data**: None

#### bot-llm-stopped 🤖

Indicates LLM processing has completed

* **type**: `bot-llm-stopped`
* **data**: None

#### user-llm-text 🤖

Aggregated user input text that is sent to the LLM.

* **type**: `'user-llm-text'`
* **data**:
  * **text**: `string`

    The user's input text to be processed by the LLM.

#### bot-llm-text 🤖

> [!WARNING]
>
> DEPRECATED in Pipecat 1.13.0, and will be removed in 2.0.0. Use
> `llm-raw-text`.

Individual tokens streamed from the LLM as they are generated.

* **type**: `'bot-llm-text'`
* **data**:
  * **text**: `string`

    The token text from the LLM.

#### llm-raw-text 🤖

Each chunk of text the LLM streams. Sent to clients of protocol 2.2.0 or later, when enabled on the server.

* **type**: `'llm-raw-text'`
* **data**:
  * **text**: `string`

    The text chunk from the LLM.

#### bot-llm-marker 🤖

> [!NOTE]
>
> Introduced in Pipecat 1.11.0. Disabled by default; enable with
> `RTVIObserverParams(bot_llm_marker_enabled=True)`.

The sideband marker the bot's LLM emitted in a response: structured output kept out of the spoken text, such as the turn-completion markers of [`FilterIncompleteUserTurnStrategies`](https://docs.pipecat.ai/api-reference/server/utilities/turn-management/filter-incomplete-turns). Sent once per response, when it ends. Because it carries the raw text of the whole response, it's meant for evaluation and debugging rather than for end-user clients.

* **type**: `'bot-llm-marker'`
* **data**:
  * **text**: `string`

    The marker as the LLM produced it; empty when the response carried none.

  * **kind**: `string` (optional)

    What the marker means, in the emitter's vocabulary (e.g. `complete`, `short`, or `long` for turn completion).

  * **raw**: `string`

    The response's text as the LLM produced it, markers included.

  * **markers**: `Array<string>`

    Every marker the emitter recognizes.

#### bot-tts-started 🤖

Indicates text-to-speech (TTS) processing has begun.

* **type**: `'bot-tts-started'`
* **data**: None

#### bot-tts-stopped 🤖

Indicates text-to-speech (TTS) processing has completed.

* **type**: `'bot-tts-stopped'`
* **data**: None

#### bot-tts-text 🤖

> [!WARNING]
>
> DEPRECATED in Pipecat 1.13.0, and will be removed in 2.0.0. Use
> `tts-raw-text`.

The per-token text output of the text-to-speech (TTS) service (what the TTS actually says).

* **type**: `'bot-tts-text'`
* **data**:
  * **text**: `string`

    The text representation of the generated bot speech.

#### tts-raw-text 🤖

The text the TTS service speaks, as it is spoken. Sent to clients of protocol 2.2.0 or later, when enabled on the server.

* **type**: `'tts-raw-text'`
* **data**:
  * **text**: `string`

    The text being spoken.

### Metrics and Monitoring

#### metrics 🤖

Performance metrics for various processing stages and services. Each message will contain entries for one or more of the metrics types below.

* **type**: `'metrics'`
* **data**:
  * **processing**: \[See Below] (optional)

    Processing time metrics.

  * **ttfb**: \[See Below] (optional)

    Time to first byte metrics.

  * **ttfa**: `Array<object>` (optional)

    TTS time to first audio: time to first byte plus any leading silence. Instead of `value`, each entry has `ttfa`, `ttfb`, and `leading_silence`, in seconds.

  * **ttfat**: `Array<object>` (optional)

    LLM time to first answer token: time to first byte plus any reasoning the model streamed first. Instead of `value`, each entry has `ttfat`, `ttfb`, and `thinking_time`, in seconds.

  * **characters**: \[See Below] (optional)

    TTS character usage metrics.

  * **tokens**: `Array<object>` (optional)

    LLM token usage. Each entry holds the token counts reported by the LLM service, such as `prompt_tokens`, `completion_tokens`, and `total_tokens`, without `processor` or `model`.

  * **stt\_usage**: \[See Below] (optional)

    STT usage metrics, where `value` is an object with `audio_seconds`: the seconds of audio sent to the service since the previous report.

For `processing`, `ttfb`, `characters`, and `stt_usage`, the data structure is an array of objects with the following structure:

* **processor**: `string`

  The name of the processor or service that generated the metric.

* **value**: `number`

  The value of the metric: seconds for `processing` and `ttfb`, a character count for `characters`.

* **model**: `string` (optional)

  The model of the service that generated the metric, if applicable.

**Example:**

```json theme={null}
{
  "type": "metrics",
  "data": {
    "processing": [
      {
        "model": "eleven_flash_v2_5",
        "processor": "ElevenLabsTTSService#0",
        "value": 0.0005140304565429688
      }
    ],
    "ttfb": [
      {
        "model": "eleven_flash_v2_5",
        "processor": "ElevenLabsTTSService#0",
        "value": 0.1573178768157959
      }
    ],
    "characters": [
      {
        "model": "eleven_flash_v2_5",
        "processor": "ElevenLabsTTSService#0",
        "value": 43
      }
    ]
  }
}
```

### User Interface

These messages let a server-side agent observe and drive a GUI on the client. The client streams the screen as accessibility snapshots and dispatches user events; the agent acts on the page with commands and surfaces long-running work as job-group cards. On the server, [`UIWorker`](https://docs.pipecat.ai/api-reference/server/workers/ui-worker) builds higher-level abstractions on top of these messages, but single-LLM apps can target the wire format directly. On the client, see the [JavaScript client methods](https://docs.pipecat.ai/api-reference/client/js/client-methods.md#user-interface) and [React hooks](https://docs.pipecat.ai/api-reference/client/react/hooks.md#user-interface).

#### ui-snapshot 🏄

A serialized accessibility tree describing what is currently on screen. Sent whenever the page changes so the agent always reasons over the latest state.

* **type**: `'ui-snapshot'`
* **data**:
  * **tree**: `A11ySnapshot Object`

    The accessibility snapshot. The agent renders it into a `<ui_state>` block for its LLM. The `A11ySnapshot` object follows this structure:

    **A11ySnapshot**

    **root** `A11yNode` *required*

    The root accessibility node. Each `A11yNode` has `ref` (a stable
    client-assigned element reference like `"e42"`), `role`, and optional
    `name`, `value`, `state` (short tags such as `"focused"`, `"disabled"`,
    `"offscreen"`), `level`, `colcount`, `rowcount`, and `children`.

    **captured\_at** `number` *required*

    Client-side epoch milliseconds when the snapshot was captured.

    **selection** `A11ySelection`

    The user's current text selection, if any: `ref`, `text`, and optional
    `start_offset` / `end_offset`. Used so the agent can resolve deictic
    references like "this paragraph".

    Extra fields are allowed on the snapshot models so clients can add platform-specific or future metadata without breaking older servers.

#### ui-event 🏄

An app-defined event dispatched by the client (e.g. a button click, a custom interaction). On the server these route to `@ui_event(name)` handlers.

* **type**: `'ui-event'`
* **data**:
  * **event**: `string`

    The app-defined event name.

  * **payload**: `any` (optional)

    App-defined payload, schemaless by design.

#### ui-cancel-job-group 🏄

Requests cancellation of an in-flight job group the agent started (see `ui-job-group`). Honored only when the group was started as cancellable.

* **type**: `'ui-cancel-job-group'`
* **data**:
  * **job\_id**: `string`

    The job-group id the client wants cancelled.

  * **reason**: `string` (optional)

    A human-readable reason.

#### ui-command 🤖

A command from the agent for the client to act on the page. Client-side handlers dispatch on the command name; `@pipecat-ai/client-react` ships default handlers for the standard commands below.

* **type**: `'ui-command'`
* **data**:
  * **command**: `string`

    The command name (e.g. `"scroll_to"`, `"toast"`, or any app-specific name).

  * **payload**: `any` (optional)

    The command payload. Standard command payloads:

    **Standard command payloads**

    **scroll\_to** `ScrollTo`

    Scroll a target into view. `ref` (snapshot ref) or `target_id`
    (`document.getElementById`), plus optional `behavior` (`"smooth"` /
    `"instant"`).

    **highlight** `Highlight`

    Briefly emphasize a target. `ref` or `target_id`, plus optional
    `duration_ms`.

    **select\_text** `SelectText`

    Select text on the page (deixis). `ref` or `target_id`, plus optional
    `start_offset` / `end_offset` (whole element when omitted).

    **focus** `Focus`

    Move keyboard focus to a target. `ref` or `target_id`.

    **click** `Click`

    Click an element (checkboxes, radios, submit buttons). `ref` or
    `target_id`. The standard handler no-ops on `disabled` targets.

    **set\_input\_value** `SetInputValue`

    Write into a text input/textarea. `value`, `ref` or `target_id`, and
    `replace` (overwrite when `true`, append when `false`). No-ops on
    `disabled` / `readonly` / hidden inputs.

    **toast** `Toast`

    Show a transient notification. `title`, plus optional `subtitle`,
    `description`, `image_url`, `duration_ms`.

    **navigate** `Navigate`

    Navigate to a named view. `view`, plus optional `params`.

#### ui-job-group 🤖

Lifecycle envelopes for a user-facing job group the agent dispatched to peer workers, so the client can render a progress card. Each envelope carries a `kind` field the client dispatches on. The kinds form the group's lifecycle:

```
group_started → job_update* → job_completed × N → group_completed
```

where N is the number of workers in the group.

* **type**: `'ui-job-group'`
* **data**: one of the four `kind`-discriminated shapes:

  **ui-job-group data kinds**

  **group\_started** `object`

  `kind: "group_started"`, `job_id`, `workers` (worker names), optional
  `label`, `cancellable`, `at` (epoch ms).

  **job\_update** `object`

  `kind: "job_update"`, `job_id`, `worker_name`, `data` (the worker's update
  payload, forwarded verbatim), `at`.

  **job\_completed** `object`

  `kind: "job_completed"`, `job_id`, `worker_name`, `status`, `response`,
  `at`. Emitted once per worker.

  **group\_completed** `object`

  `kind: "group_completed"`, `job_id`, `at`. Emitted once when the whole
  group finishes or is cancelled.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.