Skip to main content

Overview

With expressive conversation on, the LLM writes expression tags into its reply and the TTS service performs them, and a perception producer, such as PipecatToneReader, tells the LLM how the user sounded. Tags and readings follow the Real-time Context Standard, which defines their syntax. For a walkthrough, see Expressive Conversation.
Expressive conversation is alpha. Its tags, types and defaults may change between releases.

Expressive TTS Services

The TTS services and models that perform tags, and what each performs

Pipecat Tone

The reader, which hears how the user sounded, and the tagger, which writes tags for the LLM

Pipecat Voice Reader

The reader that hears how each user turn sounded

Behavioral Signals

A community perception producer that adds emotion and behavior sensing with the Behavioral Signals API

Configuration

PipelineParams

bool
default:"False"
Whether the agent takes part in expressive conversation. The LLM is taught the tags the TTS service performs and how to read the user’s perception, and the TTS service performs the tags instead of speaking them.
ExpressionPolicy | None
default:"None"
Which tags the LLM is offered, and when to use them. None offers the core names the TTS service performs, with no guidance. Has no effect, with a warning, unless enable_expressive_conversation is True.

ExpressionPolicy

list[ExpressionGuidance | str]
default:"[]"
Lines on when and how much to express: presets or your own, joined in order, so a later line can refine an earlier one. Your own lines reach the LLM whenever it has tags to write or a reading to act on. The presets go to whoever writes the tags: the LLM, or PipecatToneTagger in its place.
list[ExpressionDelivery | str] | None
default:"None"
The deliveries to offer, by their universal names. None offers the core deliveries. A name that isn’t universal, such as an accent, is offered as written. Names the TTS service can’t perform are left out, with a warning.
list[ExpressionSound | str] | None
default:"None"
The sounds to offer, as delivery offers deliveries. None offers the core sounds, and breath only with ExpressionGuidance.BREATHE.

ExpressionGuidance

Presets for ExpressionPolicy.guidance. Each is a short paragraph on one aspect of delivery, written to combine with the others and with your own lines. A preset is about writing tags, so an LLM taught no tags isn’t given it.

Universal Names

ExpressionDelivery and ExpressionSound name every delivery and sound the expressive TTS services perform, the same on each. A TTS service translates a universal name into its own, such as warm into ElevenLabs’ warmly, while transcripts, frames and RTVI messages keep the universal name. The members are strings: ExpressionDelivery.WARM is "warm", and ExpressionSound.CLEAR_THROAT is "clear throat". The Real-time Context Standard lists every name.

Core Names

The core names are what the LLM is offered when the policy names none, and what PipecatToneTagger writes: A breath is offered only with ExpressionGuidance.BREATHE.

Perception

A perception producer reads how the user spoke each turn and sets it as the transcription’s perception. A producer after another adds its reading to the one the transcription carries. The LLM reads it at the head of the user’s message, formatted by format_perception().

PerceptionReader

A perception producer that asks classifiers how the user spoke, and adds the answers to the transcription’s perception, one signal per question, after any signals the STT service read. Goes between the STT service and the user aggregator. Every classifier’s reading is also broadcast in a PerceptionFrame as soon as it’s made. A reader asks each of its sources about the same turn at once, and their signals go in the turn’s one reading, in the sources’ order. A signal both the transcription and the reader report keeps the higher of each label, as merge_perception() does. The reader declares the signals declared before it together with its own in a PerceptionMetadataFrame, so the latest declaration names every signal a transcription can carry. A transcription goes on once every source has read it or run out of time. A source that’s late or whose classifier fails leaves only its own signals out, and a transcription none of them read passes unread. A classifier that can’t be set up is reported as an error that leaves the reader unusable. A text classifier reads each final transcription, with the bot’s previous reply. A classifier that can hear, as its supports_audio says, such as HumeAudioClassifier or PipecatVoiceClassifier, hears the user’s speech each time they stop: the latest, topped up with the end of what they said before it since the bot last spoke when that’s shorter than the classifier’s preferred_audio_secs. Speech from before a mute or before the bot stopped speaking is never joined to later speech, and audio heard while the user is muted isn’t kept. A classifier hears one clip at a time, and of the clips that come while it’s busy, only the newest. The transcriptions of that speech are held until the speech is read and pushed on together, with the reading, and only the last one is finalized. The STT’s ttfs_p99_latency is raised by the longest source timeout so the user’s turn waits for them. PipecatToneReader is a PerceptionReader.
Sequence[PerceptionSource]
required
The classifiers to ask, each with its questions and timeout. Raises ValueError when there are no sources, when two sources report a signal of the same name, or when two sources ask the same classifier.
bool
default:"False"
Whether the reader’s reading replaces the one a transcription carries, such as the STT service’s, instead of being added to it. A transcription the reader doesn’t read keeps the reading it carries.
A ChoiceQuestion is reported as labels, a YesNoQuestion as its probability, and a ScoreQuestion as its position on the scale.

PerceptionSource

One classifier a PerceptionReader asks, and the questions it’s asked. A signal’s name belongs to one source, and a classifier goes in one source, with all its questions. A classifier whose questions aren’t known until it’s set up, such as Pipecat Tone’s, gets a subclass: PipecatToneSource.
BaseClassifier
required
What answers the questions about each turn.
Mapping[str, PerceptionQuestion]
required
The questions, by the name of the signal each reports. Raises ValueError when a name has a space or one of =, ,, [, ], ( or ).
float | None
default:"None"
Seconds a transcription waits for the classifier’s answers. By default 0.5 for a text classifier, counted from when the transcription arrives, and 2 for a classifier that hears audio, counted from when the user stops speaking.

PerceptionQuestion

One signal a PerceptionSource reports.

PerceptionSignal

One signal of a reading, read either as labels or as a value.

PerceptionTerm

One signal a perception producer reports, as it declares it at start.

PerceptionCapabilities

merge(other) returns the signals of both, these first. A signal both report keeps every label either can report and the first non-empty description.

format_perception

Returns a reading as the LLM reads it: ([emotion angry=0.81, neutral=0.12],[sarcasm value=0.04]).

merge_perception

Merges a reading into the readings of the same turn so far, keeping the higher of each signal. A signal read as a value keeps its highest value. A signal read as labels keeps every label either reading has, each with its highest confidence, likeliest first. A signal only one reading has is kept as it is, and signals stay in the order they were first read. Pass None as merged before the first reading. The user aggregator uses it to merge all the final readings of a user turn, so a reading from earlier in the turn still reaches the LLM.

Frames

Events

The user aggregator’s on_user_turn_stopped message carries perception, the turn’s final readings merged with merge_perception(), or None without a perception producer.

Notes

  • Context keeps the tags: the reply is stored in the context as the LLM wrote it, so the LLM sees the tags it used. With PipecatToneTagger, the context keeps the reply plain.
  • Tags never reach text: transcripts, LLMContextAssistantTurnFrame and RTVI bot-output text carry the words alone.
  • Tags the TTS service can’t perform: they are left out of the speech and kept in the context.
  • Unfinished tags: a tag cut off at the end of a reply or by an interruption is removed.
  • A TTS service after the LLM: the LLM is taught the tags of the TTS service that speaks its replies. When a settings change, such as a new model or voice, changes what it performs, the LLM is taught the new tags.