Overview
With expressive conversation on, the LLM writes expression tags into its reply and the TTS service performs them, and a perception producer, such asPipecatToneReader, tells the LLM how the user sounded. Tags and readings follow the Real-time Context Standard, which defines their syntax.
For a walkthrough, see Expressive Conversation.
Expressive TTS Services
The TTS services and models that perform tags, and what each performs
Pipecat Tone
The reader, which hears how the user sounded, and the tagger, which writes
tags for the LLM
Pipecat Voice Reader
The reader that hears how each user turn sounded
Behavioral Signals
A community perception producer that adds emotion and behavior sensing with
the Behavioral Signals API
Configuration
PipelineParams
Whether the agent takes part in expressive conversation. The LLM is taught the
tags the TTS service performs and how to read the user’s perception, and the
TTS service performs the tags instead of speaking them.
Which tags the LLM is offered, and when to use them.
None offers the core
names the TTS service performs, with no guidance. Has no effect,
with a warning, unless enable_expressive_conversation is True.ExpressionPolicy
Lines on when and how much to express: presets or your
own, joined in order, so a later line can refine an earlier one. Your own
lines reach the LLM whenever it has tags to write or a reading to act on. The
presets go to whoever writes the tags: the LLM, or
PipecatToneTagger
in its place.The deliveries to offer, by their universal names.
None offers the core
deliveries. A name that isn’t universal, such as an accent, is offered as
written. Names the TTS service can’t perform are left out, with a warning.The sounds to offer, as
delivery offers deliveries. None offers the core
sounds, and breath only with ExpressionGuidance.BREATHE.ExpressionGuidance
Presets forExpressionPolicy.guidance. Each is a short paragraph on one aspect of delivery, written to combine with the others and with your own lines. A preset is about writing tags, so an LLM taught no tags isn’t given it.
Universal Names
ExpressionDelivery and ExpressionSound name every delivery and sound the expressive TTS services perform, the same on each. A TTS service translates a universal name into its own, such as warm into ElevenLabs’ warmly, while transcripts, frames and RTVI messages keep the universal name. The members are strings: ExpressionDelivery.WARM is "warm", and ExpressionSound.CLEAR_THROAT is "clear throat". The Real-time Context Standard lists every name.
Core Names
The core names are what the LLM is offered when the policy names none, and whatPipecatToneTagger writes:
A breath is offered only with
ExpressionGuidance.BREATHE.
Perception
A perception producer reads how the user spoke each turn and sets it as the transcription’sperception. A producer after another adds its reading to the one the transcription carries. The LLM reads it at the head of the user’s message, formatted by format_perception().
PerceptionReader
A perception producer that asks classifiers how the user spoke, and adds the answers to the transcription’sperception, one signal per question, after any signals the STT service read. Goes between the STT service and the user aggregator. Every classifier’s reading is also broadcast in a PerceptionFrame as soon as it’s made. A reader asks each of its sources about the same turn at once, and their signals go in the turn’s one reading, in the sources’ order. A signal both the transcription and the reader report keeps the higher of each label, as merge_perception() does. The reader declares the signals declared before it together with its own in a PerceptionMetadataFrame, so the latest declaration names every signal a transcription can carry. A transcription goes on once every source has read it or run out of time. A source that’s late or whose classifier fails leaves only its own signals out, and a transcription none of them read passes unread. A classifier that can’t be set up is reported as an error that leaves the reader unusable.
A text classifier reads each final transcription, with the bot’s previous reply. A classifier that can hear, as its supports_audio says, such as HumeAudioClassifier or PipecatVoiceClassifier, hears the user’s speech each time they stop: the latest, topped up with the end of what they said before it since the bot last spoke when that’s shorter than the classifier’s preferred_audio_secs. Speech from before a mute or before the bot stopped speaking is never joined to later speech, and audio heard while the user is muted isn’t kept. A classifier hears one clip at a time, and of the clips that come while it’s busy, only the newest. The transcriptions of that speech are held until the speech is read and pushed on together, with the reading, and only the last one is finalized. The STT’s ttfs_p99_latency is raised by the longest source timeout so the user’s turn waits for them. PipecatToneReader is a PerceptionReader.
The classifiers to ask, each with its questions and timeout. Raises
ValueError when there are no sources, when two sources report a signal of
the same name, or when two sources ask the same classifier.Whether the reader’s reading replaces the one a transcription carries, such as
the STT service’s, instead of being added to it. A transcription the reader
doesn’t read keeps the reading it carries.
ChoiceQuestion is reported as labels, a YesNoQuestion as its probability, and a ScoreQuestion as its position on the scale.
PerceptionSource
One classifier aPerceptionReader asks, and the questions it’s asked. A signal’s name belongs to one source, and a classifier goes in one source, with all its questions. A classifier whose questions aren’t known until it’s set up, such as Pipecat Tone’s, gets a subclass: PipecatToneSource.
The questions, by the name of the signal each reports. Raises
ValueError
when a name has a space or one of =, ,, [, ], ( or ).Seconds a transcription waits for the classifier’s answers. By default
0.5
for a text classifier, counted from when the transcription arrives, and 2
for a classifier that hears audio, counted from when the user stops speaking.PerceptionQuestion
One signal aPerceptionSource reports.
PerceptionSignal
One signal of a reading, read either as labels or as a value.PerceptionTerm
One signal a perception producer reports, as it declares it at start.PerceptionCapabilities
merge(other) returns the signals of both, these first. A signal both report keeps every label either can report and the first non-empty description.
format_perception
([emotion angry=0.81, neutral=0.12],[sarcasm value=0.04]).
merge_perception
None as merged before the first reading.
The user aggregator uses it to merge all the final readings of a user turn, so a reading from earlier in the turn still reaches the LLM.
Frames
Events
The user aggregator’son_user_turn_stopped message carries perception, the turn’s final readings merged with merge_perception(), or None without a perception producer.
Notes
- Context keeps the tags: the reply is stored in the context as the LLM wrote it, so the LLM sees the tags it used. With
PipecatToneTagger, the context keeps the reply plain. - Tags never reach text: transcripts,
LLMContextAssistantTurnFrameand RTVIbot-outputtext carry the words alone. - Tags the TTS service can’t perform: they are left out of the speech and kept in the context.
- Unfinished tags: a tag cut off at the end of a reply or by an interruption is removed.
- A TTS service after the LLM: the LLM is taught the tags of the TTS service that speaks its replies. When a settings change, such as a new model or voice, changes what it performs, the LLM is taught the new tags.