Skip to main content

Overview

Pipecat Tone is two small models for expressive conversation that run on the CPU: For a walkthrough, see Expressive Conversation.

Models

Pipecat Tone’s models on Hugging Face

Examples

Bots with the reader, and with the tagger

Installation

The expressivity extra installs what Pipecat Tone and Pipecat Voice Reader need to download and run their models. Before Pipecat 1.16.0 it is named tone, which is deprecated and installs the same packages until 2.0.0.

PipecatToneReader

Reads each final user transcription, with the bot’s reply before it, and adds the reading to the transcription’s perception. Goes between the STT service and the user aggregator. A read takes a few milliseconds, and the transcription waits for it, up to timeout seconds. It is a PerceptionReader whose one source is a PipecatToneSource. With metrics enabled, it pushes the classifier’s processing metrics.

Configuration

str
default:"\"reader-rc2\""
The model’s release on Hugging Face. Pass "reader-rc1" for the earlier release.
str | Path | None
default:"None"
A folder to load the model from instead of downloading it, laid out as a release’s reader folder. revision is then unused.
Literal["fp16", "fp32"]
default:"\"fp16\""
The weights to load.
int
default:"4"
The CPU threads for one read.
float | None
default:"None"
Seconds a transcription waits for its reading, 0.5 by default. After that it passes unread.

Signals

The LLM reads them as ([emotion angry=0.81, neutral=0.12],[sarcasm value=0.04]) at the head of the user’s message, and is taught what each signal means. The emotion labels are the model’s: see its model card.

PipecatToneSource

Pipecat Tone as one of a PerceptionReader’s sources, reporting the same emotion and sarcasm signals. Give it to a reader beside the sources of other classifiers, such as one that hears the user’s audio. Another source’s signals need names other than these two.
Takes the same revision, model_dir, precision, threads and timeout as PipecatToneReader.

PipecatToneClassifier

A classifier backed by the reader model. It answers only the questions built into the model, listed in questions, and any other question raises ClassifierError. setup() loads the model, and questions raises ClassifierError until then.

Configuration

Takes the same revision, model_dir, precision and threads as PipecatToneReader.

Questions

The state is the turn’s text, or {"user": ..., "assistant": ...} to read it with the bot’s words before it. Any other state raises ClassifierError.

PipecatToneTagger

Labels each sentence of the LLM’s reply with the call so far, how the user sounded and the pipeline’s expression policy, and sends it on with tags the TTS service performs. Goes between the LLM and the TTS service. A sentence takes about 15 milliseconds with the 17m model, and waits for it. At start, the tagger broadcasts an ExpressionMetadataFrame, so the LLM is taught no tags, writes plain words, and the context keeps its replies plain.

Configuration

Literal["17m", "32m"]
default:"\"17m\""
The model’s size. 32m agrees with the model it was trained from a little more often, and takes nearly three times as long a sentence.
str
default:"\"tagger-rc1\""
The model’s release on Hugging Face.
str | Path | None
default:"None"
A folder to load the model from instead of downloading it, laid out as a release’s tagger/17m folder. size and revision are then unused.
int
default:"1"
The CPU threads for one sentence.
dict[ExpressionDelivery | ExpressionSound, list[str]] | None
default:"None"
Names to write in place of a label’s universal name, best first, such as {ExpressionDelivery.WARM: ["cheerful"]}. A name is written only where the TTS service performs it and the policy offers it. A name for ExpressionSound.BREATH is written for every breath. Raises ValueError when a key is a label the tagger doesn’t write.

What It Writes

It writes only tags the TTS service performs. The policy’s delivery and sounds narrow the names it writes, and its ExpressionGuidance presets steer it. The policy’s own lines go to the LLM.

Notes

  • Model download: each model loads while the pipeline is set up, once a process, shared by every reader or tagger. The first load downloads it from Hugging Face, which holds the pipeline’s start, so download it ahead, such as when you build your image, or pass model_dir.
  • A model that can’t load: it’s reported as an error that leaves the processor unusable. The reader passes transcriptions through unread, and with the tagger the LLM writes tags itself. The worker’s processor_unusable_policy decides whether the pipeline goes on.
  • Expressive conversation off: both processors need PipelineParams(enable_expressive_conversation=True) for the LLM to read readings or for tags to be performed.