Skip to main content

Overview

With expressive conversation on, the LLM writes expression tags into its reply, such as <<sympathetic>> or [[laugh]], and the TTS service performs them instead of speaking them. Add Pipecat’s readers, and the LLM also reads how the user sounded: Pipecat Tone reads their words and Pipecat Voice Reader hears their voice. This is Pipecat’s implementation of the Real-time Context Standard.
Expressive conversation is alpha. Its tags, types and defaults may change between releases.

Expressive Conversation Reference

Parameters, policy, universal names, frames and events

Expressive TTS Services

The TTS services and models that perform tags

Pipecat Tone

The reader and tagger models

Pipecat Voice Reader

The model that hears the user’s voice

Examples

A bot for each supported TTS service

Prerequisites

  • Pipecat 1.15.0 or later
  • A TTS service and model that performs tags: see Expressive TTS Services. Any other TTS service speaks only the words.

Turn It On

Set enable_expressive_conversation=True in PipelineParams. The pipeline is an ordinary STT, LLM and TTS pipeline:
At start, the TTS service declares what it performs, and the LLM is taught those tags in its system instruction. A reply such as <<sympathetic>> I'm so sorry about the wait. is spoken sympathetically. Tags never reach the speech, the transcripts or the client’s text, and the conversation’s context keeps them, so the LLM sees what it wrote.

Choose What the Bot Expresses

By default, the LLM is offered Pipecat’s core names that the TTS service performs, with no guidance on when to use them. An ExpressionPolicy chooses the names and adds guidance, from presets or your own lines:
Names the TTS service can’t perform are left out, with a warning in the log.
Offer about 15 deliveries or fewer. Offered more, the LLM starts inventing names of its own as the conversation goes on, and the TTS service leaves them out.

Hear How the User Sounds

Pipecat Tone’s reader reads each user turn and sets how it was meant on the transcription: the user’s emotion and how likely the turn is sarcastic. Install it:
Before Pipecat 1.16.0 the extra is named tone. Add PipecatToneReader between the STT service and the user aggregator:
The LLM reads the reading at the head of each user message, such as ([emotion angry=0.81, neutral=0.12],[sarcasm value=0.04]) Where's my order?, and is taught how to read it. Say what to do with a reading in the policy’s guidance:
The reader’s model downloads from Hugging Face the first time the pipeline starts, which holds the start. Download it ahead, such as when you build your image.

Hear the Voice Too

Pipecat Tone reads the words. Pipecat Voice Reader hears the audio: the user’s tone of voice, how activated it is, laughter, their speaking rate and how sarcastic the delivery sounds. It needs Pipecat 1.16.0 or later and a VAD in the pipeline, and the expressivity extra installs it too. On its own, PipecatVoiceReader() goes where PipecatToneReader() does. To read the words and the voice together, give both sources to one PerceptionReader, which writes their signals in one reading. Pipecat Tone reports the sarcasm of the words, and Pipecat Voice Reader reports the voice’s as sarcasm_voice:
It takes PipecatToneReader’s place in the pipeline. The LLM now reads the voice beside the words:
The voice’s sarcasm is a weak cue on its own, so have the LLM weigh it with the words:
The read takes about a tenth of a second on the CPU each time the user stops speaking. See Pipecat Voice Reader for its signals and settings.

Let Pipecat Tone Write the Tags

Pipecat Tone’s tagger writes the tags for the LLM. The LLM is taught no tags, writes plain words, and the context keeps its replies plain. The tagger labels each sentence from the call so far and how the user sounded, in about 15 milliseconds on the CPU. Add PipecatToneTagger between the LLM and the TTS service:
The policy’s presets steer the tagger, and its own lines go to the LLM.

Show Expression in Your App

Client apps receive expression through RTVI 2.3.0 and later: bot-output carries the delivery of the words just spoken, each sound and pause is a bot-output segment of its own, and user-input carries the reading of a transcription. On the server, the user aggregator’s on_user_turn_stopped event carries the turn’s reading:

Use an Avatar

With TavusTransport, the replica’s face follows the bot’s delivery, such as happiness for warm. There is nothing to set: see Avatars for the emotion each delivery shows.