Expressive Conversation

Overview

With expressive conversation on, the LLM writes expression tags into its reply, such as <<sympathetic>> or [[laugh]], and the TTS service performs them instead of speaking them. Add Pipecat Tone’s reader, and the LLM also reads how the user sounded. This is Pipecat’s implementation of the Real-time Context Standard.

Expressive conversation is alpha. Its tags, types and defaults may change between releases.

Expressive Conversation Reference

Parameters, policy, universal names, frames and events

Expressive TTS Services

The TTS services and models that perform tags

Pipecat Tone

The reader and tagger models

Examples

A bot for each supported TTS service

Prerequisites

Pipecat 1.15.0 or later
A TTS service and model that performs tags: see Expressive TTS Services. Any other TTS service speaks only the words.

Turn It On

Set enable_expressive_conversation=True in PipelineParams. The pipeline is an ordinary STT, LLM and TTS pipeline:

import os

from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.worker import PipelineParams, PipelineWorker
from pipecat.services.elevenlabs.dialogue.tts import ElevenLabsDialogueTTSService

tts = ElevenLabsDialogueTTSService(
    api_key=os.getenv("ELEVENLABS_API_KEY"),
    settings=ElevenLabsDialogueTTSService.Settings(
        voice=os.getenv("ELEVENLABS_VOICE_ID"),
        model="eleven_v4_turbo",
    ),
)

pipeline = Pipeline(
    [transport.input(), stt, user_aggregator, llm, tts, transport.output(), assistant_aggregator]
)

worker = PipelineWorker(
    pipeline,
    params=PipelineParams(enable_expressive_conversation=True),
)

At start, the TTS service declares what it performs, and the LLM is taught those tags in its system instruction. A reply such as <<sympathetic>> I'm so sorry about the wait. is spoken sympathetically. Tags never reach the speech, the transcripts or the client’s text, and the conversation’s context keeps them, so the LLM sees what it wrote.

Choose What the Bot Expresses

By default, the LLM is offered Pipecat’s core names that the TTS service performs, with no guidance on when to use them. An ExpressionPolicy chooses the names and adds guidance, from presets or your own lines:

from pipecat.expressive.expression import ExpressionDelivery, ExpressionSound
from pipecat.expressive.policy import ExpressionGuidance, ExpressionPolicy

policy = ExpressionPolicy(
    guidance=[
        ExpressionGuidance.MATCH_STYLE,
        ExpressionGuidance.SLOW_FOR_DETAILS,
        "Some feelings have no delivery of their own: to reassure, use calm or sympathetic.",
    ],
    delivery=[
        ExpressionDelivery.SLOW,
        ExpressionDelivery.CALM,
        ExpressionDelivery.WARM,
        ExpressionDelivery.SYMPATHETIC,
        ExpressionDelivery.EXCITED,
    ],
    sounds=[ExpressionSound.LAUGH, ExpressionSound.SIGH],
)

worker = PipelineWorker(
    pipeline,
    params=PipelineParams(enable_expressive_conversation=True, expression_policy=policy),
)

Names the TTS service can’t perform are left out, with a warning in the log.

Offer about 15 deliveries or fewer. Offered more, the LLM starts inventing names of its own as the conversation goes on, and the TTS service leaves them out.

Hear How the User Sounds

Pipecat Tone’s reader reads each user turn and sets how it was meant on the transcription: the user’s emotion and how likely the turn is sarcastic. Install it:

uv add "pipecat-ai[tone]"

Add PipecatToneReader between the STT service and the user aggregator:

from pipecat.processors.tone.reader import PipecatToneReader

pipeline = Pipeline(
    [
        transport.input(),
        stt,
        PipecatToneReader(),
        user_aggregator,
        llm,
        tts,
        transport.output(),
        assistant_aggregator,
    ]
)

The LLM reads the reading at the head of each user message, such as ([emotion angry=0.81, neutral=0.12],[sarcasm value=0.04]) Where's my order?, and is taught how to read it. Say what to do with a reading in the policy’s guidance:

policy = ExpressionPolicy(
    guidance=[
        ExpressionGuidance.MATCH_STYLE,
        "If the user sounds angry at 0.7 or above, apologize once, then focus on helping.",
        "If sarcasm is 0.7 or above, the user means the opposite of their words: answer what they mean.",
    ],
)

The reader’s model downloads from Hugging Face the first time the pipeline starts, which holds the start. Download it ahead, such as when you build your image.

Let Pipecat Tone Write the Tags

Pipecat Tone’s tagger writes the tags for the LLM. The LLM is taught no tags, writes plain words, and the context keeps its replies plain. The tagger labels each sentence from the call so far and how the user sounded, in about 15 milliseconds on the CPU. Add PipecatToneTagger between the LLM and the TTS service:

from pipecat.processors.tone.tagger import PipecatToneTagger

pipeline = Pipeline(
    [
        transport.input(),
        stt,
        PipecatToneReader(),
        user_aggregator,
        llm,
        PipecatToneTagger(),
        tts,
        transport.output(),
        assistant_aggregator,
    ]
)

The policy’s presets steer the tagger, and its own lines go to the LLM.

Show Expression in Your App

Client apps receive expression through RTVI 2.3.0 and later: bot-output carries the delivery of the words just spoken, each sound and pause is a bot-output segment of its own, and user-input carries the reading of a transcription. On the server, the user aggregator’s on_user_turn_stopped event carries the turn’s reading:

from pipecat.expressive.perception import format_perception


@user_aggregator.event_handler("on_user_turn_stopped")
async def on_user_turn_stopped(aggregator, strategy, message):
    if message.perception:
        logger.info(f"Tone: {format_perception(message.perception)}")

Use an Avatar

With TavusTransport, the replica’s face follows the bot’s delivery, such as happiness for warm. There is nothing to set: see Avatars for the emotion each delivery shows.

Get Started

Migration

Learning Pipecat

Fundamentals

Flows

Evals

Features

Telephony

Deployment

Examples & Recipes

Expressive Conversation

Overview

Expressive Conversation Reference

Expressive TTS Services

Pipecat Tone

Examples

Prerequisites

Turn It On

Choose What the Bot Expresses

Hear How the User Sounds

Let Pipecat Tone Write the Tags

Show Expression in Your App

Use an Avatar

​Overview

Expressive Conversation Reference

Expressive TTS Services

Pipecat Tone

Examples

​Prerequisites

​Turn It On

​Choose What the Bot Expresses

​Hear How the User Sounds

​Let Pipecat Tone Write the Tags

​Show Expression in Your App

​Use an Avatar

Overview

Prerequisites

Turn It On

Choose What the Bot Expresses

Hear How the User Sounds

Let Pipecat Tone Write the Tags

Show Expression in Your App

Use an Avatar