Expressive Conversation API Reference

Overview

With expressive conversation on, the LLM writes expression tags into its reply and the TTS service performs them, and a perception producer, such as PipecatToneReader, tells the LLM how the user sounded. Tags and readings follow the Real-time Context Standard, which defines their syntax. For a walkthrough, see Expressive Conversation.

Expressive conversation is alpha. Its tags, types and defaults may change between releases.

Expressive TTS Services

The TTS services and models that perform tags, and what each performs

Pipecat Tone

The reader, which hears how the user sounded, and the tagger, which writes tags for the LLM

from pipecat.expressive.expression import (
    CORE_DELIVERY,
    CORE_SOUNDS,
    ExpressionDelivery,
    ExpressionSound,
)
from pipecat.expressive.perception import (
    PerceptionCapabilities,
    PerceptionSignal,
    PerceptionTerm,
    format_perception,
)
from pipecat.expressive.policy import ExpressionGuidance, ExpressionPolicy

Configuration

PipelineParams

bool

default:"False"

Whether the agent takes part in expressive conversation. The LLM is taught the tags the TTS service performs and how to read the user’s perception, and the TTS service performs the tags instead of speaking them.

ExpressionPolicy | None

default:"None"

Which tags the LLM is offered, and when to use them. None offers the core names the TTS service performs, with no guidance. Has no effect, with a warning, unless enable_expressive_conversation is True.

ExpressionPolicy

list[ExpressionGuidance | str]

default:"[]"

Lines on when and how much to express: presets or your own, joined in order, so a later line can refine an earlier one. Your own lines reach the LLM whenever it has tags to write or a reading to act on. The presets go to whoever writes the tags: the LLM, or PipecatToneTagger in its place.

list[ExpressionDelivery | str] | None

default:"None"

The deliveries to offer, by their universal names. None offers the core deliveries. A name that isn’t universal, such as an accent, is offered as written. Names the TTS service can’t perform are left out, with a warning.

list[ExpressionSound | str] | None

default:"None"

The sounds to offer, as delivery offers deliveries. None offers the core sounds, and breath only with ExpressionGuidance.BREATHE.

ExpressionGuidance

Presets for ExpressionPolicy.guidance. Each is a short paragraph on one aspect of delivery, written to combine with the others and with your own lines. A preset is about writing tags, so an LLM taught no tags isn’t given it.

Preset	Guidance
`MATCH_STYLE`	Match the user’s mood and energy: excited for good news, sympathetic with bad news, a laugh when they’re funny. Stay calm with hostility.
`SLOW_FOR_DETAILS`	Say just what the user may need to write down, such as a code or a phone number, slowly, in a span. For a TTS service that performs `slow`.
`BREATHE`	Take an audible breath now and then before a long or difficult sentence, at most once a reply. For a TTS service that performs a breath. Offers `breath`.

Universal Names

ExpressionDelivery and ExpressionSound name every delivery and sound the expressive TTS services perform, the same on each. A TTS service translates a universal name into its own, such as warm into ElevenLabs’ warmly, while transcripts, frames and RTVI messages keep the universal name. The members are strings: ExpressionDelivery.WARM is "warm", and ExpressionSound.CLEAR_THROAT is "clear throat". The Real-time Context Standard lists every name.

Core Names

The core names are what the LLM is offered when the policy names none, and what PipecatToneTagger writes:

Constant	Names
`CORE_DELIVERY`	`calm`, `sympathetic`, `apologetic`, `warm`, `excited`, `surprised`, `sad`, `grateful`, `curious`, `confident`, `slow`, `fast`
`CORE_SOUNDS`	`laugh`, `chuckle`, `sigh`, `breath`

A breath is offered only with ExpressionGuidance.BREATHE.

Perception

A perception producer reads how the user spoke each turn and sets it as the transcription’s perception. The LLM reads it at the head of the user’s message, formatted by format_perception().

PerceptionSignal

One signal of a reading, read either as labels or as a value.

Field	Type	Description
`name`	`str`	What is read, such as `"emotion"`.
`labels`	`list[tuple[str, float]]`	The likeliest labels, likeliest first, each with a confidence from 0 to 1. Empty for a signal read as a value.
`value`	`float \| None`	The measure, for a signal read as a value, such as a probability or words per second. `None` for one read as labels.

PerceptionTerm

One signal a perception producer reports, as it declares it at start.

Field	Type	Description
`name`	`str`	The signal’s name, as in `PerceptionSignal.name`.
`description`	`str`	A short line on how to read it, such as its unit or range. Defaults to `""`.
`labels`	`list[str]`	The labels it can report, for a signal read as labels. Empty for a value.

PerceptionCapabilities

Field	Type	Description
`signals`	`list[PerceptionTerm]`	Each signal the producer can report.

format_perception

format_perception(signals: list[PerceptionSignal]) -> str

Returns a reading as the LLM reads it: ([emotion angry=0.81, neutral=0.12],[sarcasm value=0.04]).

Frames

Frame	Field	Contents
`TTSMetadataFrame`	`expression`	What the TTS service performs, an `ExpressionCapabilities`, or `None`. Broadcast at start and when a settings change alters it.
`PerceptionMetadataFrame`	`perception`	The `PerceptionCapabilities` of a perception producer, broadcast at start.
`ExpressionMetadataFrame`		Broadcast at start by a processor that writes the tags after the LLM, such as `PipecatToneTagger`, so the LLM is taught none.
`TranscriptionFrame`, `InterimTranscriptionFrame`, `EagerTranscriptionFrame`	`perception`	`list[PerceptionSignal] \| None`: how the user said the words, when a perception producer read them.
`TTSTextFrame`	`delivery`	`list[str] \| None`: the delivery of the word just spoken, by its universal names, such as `["warm", "emphasis"]`. Empty for the voice’s baseline.
`LLMContextAssistantTurnFrame`	`text`	The reply’s words alone, without tags.

Events

The user aggregator’s on_user_turn_stopped message carries perception, the turn’s final reading, or None without a perception producer.

Notes

Context keeps the tags: the reply is stored in the context as the LLM wrote it, so the LLM sees the tags it used. With PipecatToneTagger, the context keeps the reply plain.
Tags never reach text: transcripts, LLMContextAssistantTurnFrame and RTVI bot-output text carry the words alone.
Tags the TTS service can’t perform: they are left out of the speech and kept in the context.
Unfinished tags: a tag cut off at the end of a reply or by an interruption is removed.
A TTS service after the LLM: the LLM is taught the tags of the TTS service that speaks its replies. When a settings change, such as a new model or voice, changes what it performs, the LLM is taught the new tags.

Get Started

Standards

Pipecat Server

Client SDKs

Pipecat Flows

Pipecat Cloud

CLI

Pipecat Context Hub

Expressive Conversation

Overview

Expressive TTS Services

Pipecat Tone

Configuration

PipelineParams

ExpressionPolicy

ExpressionGuidance

Universal Names

Core Names

Perception

PerceptionSignal

PerceptionTerm

PerceptionCapabilities

format_perception

Frames

Events

Notes

​Overview

Expressive TTS Services

Pipecat Tone

​Configuration

​PipelineParams

​ExpressionPolicy

​ExpressionGuidance

​Universal Names

​Core Names

​Perception

​PerceptionSignal

​PerceptionTerm

​PerceptionCapabilities

​format_perception

​Frames

​Events

​Notes

Overview

Configuration

PipelineParams

ExpressionPolicy

ExpressionGuidance

Universal Names

Core Names

Perception

PerceptionSignal

PerceptionTerm

PerceptionCapabilities

format_perception

Frames

Events

Notes