Callbacks and Events

The Pipecat JavaScript client listens for messages and events from the bot via the transport layer. This allows you to respond to changes in state, errors, and other events. The client implements the RTVI standard for these communications.

Event Handling Options

You can handle events in two ways:

1. Callbacks

Define handlers in the client constructor:

import { PipecatClient } from "@pipecat-ai/client-js";
import { DailyTransport } from "@pipecat-ai/daily-transport";

const pcClient = new PipecatClient({
  transport: new DailyTransport(),
  callbacks: {
    onBotReady: () => console.log("Bot ready via callback"),
    // ... other callbacks
  },
});

2. Event Listeners

Add handlers using the event emitter pattern:

pcClient.on(RTVIEvent.BotReady, () => console.log("Bot ready via event"));

Events and callbacks provide the same functionality. Choose the pattern that best fits your application’s architecture.

Callbacks

State and Connectivity

Local user successfully established a connection to the transport.

Local user disconnected from the transport, either intentionally by calling pcClient.disconnect() or due to an error.

state:TransportState

Provides a TransportState string representing the connectivity state of the local client. See transports for state explanation.

botResponse: unknown

A call to startBot() (i.e. a pre-connection REST endpoint) was successful and the bot should now be started or in the process of starting. The callback receives any data returned from your endpoint.

botReadyData:BotReadyData

The bot has been instantiated, its pipeline is configured, and it is receiving user media and interactions. This method is passed a BotReadyData object, which contains the RTVI version number. Since the bot is remote and may be using a different version of RTVI than the client, you can use the passed version string to check for compatibility. Bots on RTVI 2.2.0+ also send capabilities, described under botCapabilities.

participant: Participant

Bot connected to the transport and is configuring. Note: bot connectivity does not infer that its pipeline is yet ready to run. Please use onBotReady instead.

participant: Participant

Bot disconnected from the transport. This may occur due to session expiry, a pipeline error or for any reason the server deems the session over. By default, the client will also disconnect when this fires. Set disconnectOnBotDisconnect: false in the constructor to keep the client connected.

participant: Participant

A participant joined the session. Fires for all participants, including the bot and the local user.

participant: Participant

A participant left the session. Fires for all participants, including the bot and the local user.

Messages and Errors

data:any

Receives custom messages sent from the server to the client. This provides a generic channel for server-to-client communication. The data structure is flexible and defined by the server implementation.

message:RTVIMessage

Response error when an action fails or an unknown message type is sent from the client.

error:UnsupportedFeatureError

The transport doesn’t support a feature requested through enableCam(), enableScreenShare(), or selectedCam, such as enabling a camera on an audio-only transport. The error carries a feature naming what was unavailable. Without this callback such a request appears to succeed and does nothing. sendDTMF() and sendFile() throw UnsupportedFeatureError instead of calling this callback.

Screen sharing failed to start, most often because the user dismissed the browser’s picker.

message:RTVIMessage

An error signalled by the bot, or raised locally: by the client when a message is too large or fails to send, and by some transports when their connection fails (a Daily fatal error, a WebSocket error or close, a MoQ relay that can’t be reached). The message parameter is of type error and matches the RTVI standard. Its data field includes an error string that describes the error and a fatal boolean indicating if the error is unrecoverable and resulted in a bot disconnection. If fatal is true, the client will automatically disconnect. The message field is deprecated; use error instead. When startBot() fails, the message is of type error-response and its data is { message, fatal: true }, with no error field.

Media and Devices

mics:MediaDeviceInfo[]

Lists available local media microphone devices. Triggered when a new device becomes available, a device is removed, or in response to pcClient.initDevices().

cams:MediaDeviceInfo[]

Lists available local media camera devices. Triggered when a new device becomes available, a device is removed, or in response to pcClient.initDevices().

speakers:MediaDeviceInfo[]

Lists available local speaker devices. Triggered when a new device becomes available, a device is removed, or in response to pcClient.initDevices().

mic:MediaDeviceInfo

User selected a new microphone as their selected/active device.

cam:MediaDeviceInfo

User selected a new camera as their selected/active device.

mediaState:MediaState

Per-device mic/cam state changed: a permission granted or denied, or a device that couldn’t be acquired. See mediaState for the shape and what each state means.

mediaSupport:MediaSupport

Which kinds of media can flow in the session changed, for example when the bot reports its capabilities. See mediaSupport for the shape.

speaker:MediaDeviceInfo

User selected a new speaker as their selected/active device.

error:DeviceError

Error related to media devices, such as camera or microphone issues. This could be due to permissions, device unavailability, or other related problems. See the DeviceError section for more details about the return type.

track: MediaStreamTrack, participant?: Participant

Media track from a local or remote participant/bot was started and playable. Can be either an audio or video track.

track: MediaStreamTrack, participant?: Participant

Media track from a local or remote participant/bot was stopped and no longer playable.

track: MediaStreamTrack, participant?: Participant

Media track from a local or remote participant’s screenshare was started and playable. Can be either an audio or video track.

track: MediaStreamTrack, participant?: Participant

Media track from a local or remote participant’s screenshare was stopped and no longer playable.

Audio and Voice Activity

level:number

Local audio gain level (0 to 1).

level: number, participant: Participant

Remote audio gain level (0 to 1). Note: if more than one participant is connected to the transport, the participant property details the associated peer/bot.

The bot started speaking/sending speech audio.

The bot stopped speaking/sending speech audio.

The bot’s in-flight output was cut off, either by the user speaking or by text sent with run_immediately. Whatever the bot was mid-saying will not be spoken. Sent by bots on RTVI 2.0.0 and later.

The local user started speaking. This method is more reliable than using audio gain and is the result of the bot’s VAD (voice activity detection) model. This provides a more accurate result in noisy environments.

The local user stopped speaking, indicated by the VAD model.

The server has started ignoring audio from the client (server-side muting). The client should continue sending audio normally but may want to show an indication to the user that their input is not being processed. See User Input Muting for more details.

The server has stopped ignoring audio from the client (server-side muting ended). The client can update its UI to indicate that the user’s input is being processed again.

Transcription

TranscriptData

deprecated

Deprecated. Use onUserInput to show what the user said, or onSttRawText to inspect what the STT produced. Transcribed local user input (both partial and final).Callback receives a TranscriptData object:

Show TranscriptData

string

The transcribed text.

boolean

Indicates if the text is final (true) or partial (false).

string

The timestamp of the transcription.

string

The ID of the user the transcription is for.

UserInputData

What the user said or typed: transcriptions, text sent with sendText(), and backchannels (RTVI 2.2.0+). The callback receives a UserInputData object:

Show UserInputData

string

The text of the input.

'transcription' | 'chat' | 'backchannel'

transcription is speech the bot transcribed, chat is text sent with sendText(), and backchannel is a short acknowledgment the user said while the bot talked, such as “mhm”. A backchannel doesn’t interrupt the bot or reach the LLM.

string

The timestamp of the input.

boolean

Whether the text is final. Only an interim transcription isn’t.

string

The ID of the user who spoke or typed. For chat, set only when the transport knows the sender, such as Daily and LiveKit.

string

For chat, the ID of the send-text message, as returned by sendText().

BotOutputData

A best-effort stream of the bot’s output text, including both spoken and unspoken content. This callback is triggered as the bot aggregates the LLM’s response into sentences or other logical text blocks as well as word-by-word during TTS synthesis. The callback receives a BotOutputData object:

Show BotOutputData

string

The aggregated output text from the bot.

boolean

Whether this text is destined for TTS. false for content the bot produced but will not speak.

'new' | 'in-progress' | 'completed'

How far synthesis has got with this segment.

SpokenProgressData

{ accumulated_text, remaining_text }: what has been spoken so far and what is still to come, for rendering text in time with the audio.

number

Identifies the segment this output belongs to, so word-level updates can be matched to the sentence they refine.

boolean

deprecated

RTVI protocol 1.4.x only. Use will_be_spoken instead.

'backchannel' | 'sentence' | 'token' | 'word' | string

The type of text in this output (e.g., “sentence”, “code”). “backchannel”, “sentence”, “token” and “word” are reserved output types defined by the RTVI standard. A “backchannel” is a short acknowledgment the bot says while the user talks, such as “Mm-hmm.” (RTVI 2.2.0+). Other output types may be defined by custom text aggregators used by the server. “word” outputs are sent at the time of TTS synthesis for real-time word-level streaming and can be used in lieu of onBotTtsText if desired. Filled in from text_type or aggregated_by when the bot sends only those.

'backchannel' | 'sentence' | 'token' | 'word' | string

deprecated

Deprecated. Use output_type instead, which carries the same value.

'backchannel' | 'sentence' | 'token' | 'word' | string

deprecated

Deprecated. Use output_type instead, which carries the same value.

text:BotLLMTextData

deprecated

Deprecated since client-js 1.5.0. Use onBotOutput instead. Finalized bot output text generated by the LLM. Sentence aggregated.

Service-specific Events

BotLLMSearchResponseData

Bot LLM search response text generated by the LLM service. This is typically used for search or retrieval tasks.

Search capabilities are currently only supported by Google Gemini. To take advantage of this event, your pipeline must include a GoogleLLMService and your pipeline worker should include the GoogleRTVIObserver in lieu of the typical RTVIObserver.

Show BotLLMSearchResponseData

string

The search result text.

string

The rendered content of the search result.

LLMSearchOrigin[]

The origins of the search result.

Show LLMSearchOrigin

string

The URI of the site where the search result was found.

string

The title of the site where the search result was found.

LLMSearchResult[]

The individual search results.

Show LLMSearchResult

string

The text of the search result.

number[]

The confidence scores for the search result.

UserLLMTextData

The text the bot received as user input, as {text}. Useful when the user typed rather than spoke, or to see the transcript the LLM actually acted on.

BotLLMTextData

deprecated

Deprecated. Use onBotOutput to show the bot’s text, or onLlmRawText to inspect what the LLM produced. Streamed LLM token response text generated by the LLM service.

Show BotLLMTextData

string

The text of the LLM response.

LLM service inference started.

LLM service inference concluded.

BotTTSTextData

deprecated

Deprecated. Use onBotOutput to show the bot’s text, or onTtsRawText to inspect what the TTS produced. If your TTS service supports streamed responses over sockets, the text parameter contains the words from TTS service as they are spoken. If you are using a HTTP based TTS service, the text parameter will contain the full text of the TTS response.

Hide BotTTSTextData

string

The text of the LLM response.

TTS service started inference.

TTS service inference concluded.

Raw Text Events

Raw text events are for debugging what the STT, LLM, and TTS services produced (RTVI 2.2.0+). The bot sends them only when the corresponding stt_raw_text_enabled, llm_raw_text_enabled, or tts_raw_text_enabled RTVIObserverParams option is enabled.

STTRawTextData

Each transcription as the STT produced it, both partial and final.

Show STTRawTextData

string

The transcribed text.

boolean

Indicates if the text is final (true) or interim (false).

string

The timestamp of the transcription.

string

The ID of the user the transcription is for.

LLMRawTextData

Each chunk of text as the LLM streamed it.

Show LLMRawTextData

string

The chunk of text.

TTSRawTextData

Text the TTS speaks, as it is spoken.

Show TTSRawTextData

string

The spoken text.

Function Calling

LLMFunctionCallStartedData

A function call has been initiated by the LLM. The metadata included depends on the server’s function_call_report_level configuration.

Show LLMFunctionCallStartedData

string

Name of the function being called. Only included if report level is NAME or FULL.

LLMFunctionCallInProgressData

A function call is in progress. This replaces the deprecated onLLMFunctionCall callback and is the event that triggers registered FunctionCallHandlers when a function_name is present.

Show LLMFunctionCallInProgressData

string

Name of the function being called. Only included if report level is NAME or FULL.

string

Unique identifier for this function call.

Record<string, unknown>

Arguments passed to the function. Only included if report level is FULL.

LLMFunctionCallStoppedData

A function call has completed or been cancelled.

Show LLMFunctionCallStoppedData

string

Name of the function that was called. Only included if report level is NAME or FULL.

string

Identifier matching the original function call.

boolean

Whether the function call was cancelled before completing.

unknown

The result of the function call, if available. Only included if report level is FULL.

LLMFunctionCallData

deprecated

Deprecated since client-js 1.6.0. Use onLLMFunctionCallInProgress instead. A function call request from the LLM.

User Interface

data:UICommandData

The bot sent a UI command for your app to act on. data is { command, payload }; standard commands are scroll_to, highlight, select_text, click, focus, set_input_value, toast, and navigate (the React SDK ships default handlers). See the RTVI standard for payload shapes.

data:UIJobGroupData

A lifecycle envelope for background work the bot dispatched, for rendering a progress card. data.kind is one of group_started, job_update, job_completed, group_completed. Cancel a group with cancelUIJobGroup().

Other

data:PipecatMetricsData

Pipeline metrics data provided by Pipecat. Learn more.

Events

Each callback described above has a corresponding event that can be listened for using the .on() method. This allows you to handle the same functionality using either callbacks or event listeners, depending on your preferred architecture. Here’s the complete reference mapping events to their corresponding callbacks:

State and Connectivity Events

Event Name	Callback Name	Data Type
`Connected`	`onConnected`	-
`Disconnected`	`onDisconnected`	-
`TransportStateChanged`	`onTransportStateChanged`	`TransportState`
`BotStarted`	`onBotStarted`	`unknown`
`BotReady`	`onBotReady`	`BotReadyData`
`BotConnected`	`onBotConnected`	`Participant`
`BotDisconnected`	`onBotDisconnected`	`Participant`
`ParticipantConnected`	`onParticipantJoined`	`Participant`
`ParticipantLeft`	`onParticipantLeft`	`Participant`

Message and Error Events

Event Name	Callback Name	Data Type
`ServerMessage`	`onServerMessage`	`any`
`MessageError`	`onMessageError`	`RTVIMessage`
`Error`	`onError`	`RTVIMessage`
`UnsupportedFeature`	`onUnsupportedFeature`	`UnsupportedFeatureError`

Media Events

Event Name	Callback Name	Data Type
`TrackStarted`	`onTrackStarted`	`MediaStreamTrack, Participant`
`TrackStopped`	`onTrackStopped`	`MediaStreamTrack, Participant`
`AvailableMicsUpdated`	`onAvailableMicsUpdated`	`MediaDeviceInfo[]`
`AvailableCamsUpdated`	`onAvailableCamsUpdated`	`MediaDeviceInfo[]`
`MicUpdated`	`onMicUpdated`	`MediaDeviceInfo`
`CamUpdated`	`onCamUpdated`	`MediaDeviceInfo`
`SpeakerUpdated`	`onSpeakerUpdated`	`MediaDeviceInfo`
`AvailableSpeakersUpdated`	`onAvailableSpeakersUpdated`	`MediaDeviceInfo[]`
`ScreenTrackStarted`	`onScreenTrackStarted`	`MediaStreamTrack, Participant`
`ScreenTrackStopped`	`onScreenTrackStopped`	`MediaStreamTrack, Participant`
`ScreenShareError`	`onScreenShareError`	`string`
`MediaStateUpdated`	`onMediaStateChanged`	`MediaState`
`MediaSupportUpdated`	`onMediaSupportChanged`	`MediaSupport`
`DeviceError`	`onDeviceError`	`DeviceError`

Audio Activity Events

Event Name	Callback Name	Data Type
`LocalAudioLevel`	`onLocalAudioLevel`	`number`
`RemoteAudioLevel`	`onRemoteAudioLevel`	`number, Participant`
`BotStartedSpeaking`	`onBotStartedSpeaking`	-
`BotStoppedSpeaking`	`onBotStoppedSpeaking`	-
`BotInterrupted`	`onBotInterrupted`	-
`UserStartedSpeaking`	`onUserStartedSpeaking`	-
`UserStoppedSpeaking`	`onUserStoppedSpeaking`	-
`UserMuteStarted`	`onUserMuteStarted`	-
`UserMuteStopped`	`onUserMuteStopped`	-

Text and Transcription Events

Event Name	Callback Name	Data Type
~~`UserTranscript`~~	~~`onUserTranscript`~~	`TranscriptData`
`UserInput`	`onUserInput`	`UserInputData`
`UserLlmText`	`onUserLlmText`	`UserLLMTextData`
`BotOutput`	`onBotOutput`	`BotOutputData`
~~`BotTranscript`~~	~~`onBotTranscript`~~	`BotLLMTextData`
~~`BotLlmText`~~	~~`onBotLlmText`~~	`BotLLMTextData`
~~`BotTtsText`~~	~~`onBotTtsText`~~	`BotTTSTextData`
`SttRawText`	`onSttRawText`	`STTRawTextData`
`LlmRawText`	`onLlmRawText`	`LLMRawTextData`
`TtsRawText`	`onTtsRawText`	`TTSRawTextData`

Service State Events

Event Name	Callback Name	Data Type
`BotLlmSearchResponse`	`onBotLlmSearchResponse`	`BotLLMSearchResponseData`
`BotLlmStarted`	`onBotLlmStarted`	-
`BotLlmStopped`	`onBotLlmStopped`	-
`BotTtsStarted`	`onBotTtsStarted`	-
`BotTtsStopped`	`onBotTtsStopped`	-

Function Call Events

Event Name	Callback Name	Data Type
`LLMFunctionCallStarted`	`onLLMFunctionCallStarted`	`LLMFunctionCallStartedData`
`LLMFunctionCallInProgress`	`onLLMFunctionCallInProgress`	`LLMFunctionCallInProgressData`
`LLMFunctionCallStopped`	`onLLMFunctionCallStopped`	`LLMFunctionCallStoppedData`
~~`LLMFunctionCall`~~	~~`onLLMFunctionCall`~~	`LLMFunctionCallData`

User Interface Events

Event Name	Callback Name	Data Type
`UICommand`	`onUICommand`	`UICommandData`
`UIJobGroup`	`onUIJobGroup`	`UIJobGroupData`

Streaming snapshots and sending UI events are client methods; see the RTVI standard for payload types.

Other Events

Event Name	Callback Name	Data Type
`Metrics`	`onMetrics`	`PipecatMetricsData`

Usage Example

import { PipecatClient, RTVIEvent } from "@pipecat-ai/client-js";
import { DailyTransport } from "@pipecat-ai/daily-transport";

// Using callbacks
const pcClient = new PipecatClient({
  transport: new DailyTransport(),
  callbacks: {
    onBotReady: () => console.log("Bot ready via callback"),
    onUserInput: (data) => console.log("User said:", data.text),
  },
});

// Alternate approach: Using event listeners
pcClient.on(RTVIEvent.BotReady, () => {
  console.log("Bot ready via event");
});

Get Started

Standards

Pipecat Server

Client SDKs

Pipecat Flows

Pipecat Cloud

CLI

Pipecat Context Hub

Event Handling Options

1. Callbacks

2. Event Listeners

Callbacks

State and Connectivity

Messages and Errors

Media and Devices

Audio and Voice Activity

Transcription

Service-specific Events

Raw Text Events

Function Calling

User Interface

Other

Events

State and Connectivity Events

Message and Error Events

Media Events

Audio Activity Events

Text and Transcription Events

Service State Events

Function Call Events

User Interface Events

Other Events

Usage Example

Transport Compatibility

​Event Handling Options

​1. Callbacks

​2. Event Listeners

​Callbacks

​State and Connectivity

​Messages and Errors

​Media and Devices

​Audio and Voice Activity

​Transcription

​Service-specific Events

​Raw Text Events

​Function Calling

​User Interface

​Other

​Events

​State and Connectivity Events

​Message and Error Events

​Media Events

​Audio Activity Events

​Text and Transcription Events

​Service State Events

​Function Call Events

​User Interface Events

​Other Events

​Usage Example

​Transport Compatibility

Event Handling Options

1. Callbacks

2. Event Listeners

Callbacks

State and Connectivity

Messages and Errors

Media and Devices

Audio and Voice Activity

Transcription

Service-specific Events

Raw Text Events

Function Calling

User Interface

Other

Events

State and Connectivity Events

Message and Error Events

Media Events

Audio Activity Events

Text and Transcription Events

Service State Events

Function Call Events

User Interface Events

Other Events

Usage Example

Transport Compatibility