Skip to main content

​
Overview

Pipecat Tone is two small models for expressive conversation that run on the CPU:
  • PipecatToneReader reads how each user turn was meant from its words, a perception producer that tells the LLM how the user sounded.
  • PipecatToneTagger writes the expression tags for the LLM’s plain replies, a sentence at a time.
For a walkthrough, see Expressive Conversation.

Models

Pipecat Tone’s models on Hugging Face

Examples

Bots with the reader, and with the tagger

​
Installation

​
PipecatToneReader

Reads each final user transcription, with the bot’s reply before it, and sets the reading as the transcription’s perception. Goes between the STT service and the user aggregator. A read takes a few milliseconds, and the transcription waits for it.

​
Configuration

​
str
default:"\"reader-rc2\""
The model’s release on Hugging Face. Pass "reader-rc1" for the earlier release.
​
str | Path | None
default:"None"
A folder to load the model from instead of downloading it, laid out as a release’s reader folder. revision is then unused.
​
Literal["fp16", "fp32"]
default:"\"fp16\""
The weights to load.
​
int
default:"4"
The CPU threads for one read.

​
Signals

The LLM reads them as ([emotion angry=0.81, neutral=0.12],[sarcasm value=0.04]) at the head of the user’s message, and is taught what each signal means. The emotion labels are the model’s: see its model card.

​
PipecatToneTagger

Labels each sentence of the LLM’s reply with the call so far, how the user sounded and the pipeline’s expression policy, and sends it on with tags the TTS service performs. Goes between the LLM and the TTS service. A sentence takes about 15 milliseconds with the 17m model, and waits for it. At start, the tagger broadcasts an ExpressionMetadataFrame, so the LLM is taught no tags, writes plain words, and the context keeps its replies plain.

​
Configuration

​
Literal["17m", "32m"]
default:"\"17m\""
The model’s size. 32m agrees with the model it was trained from a little more often, and takes nearly three times as long a sentence.
​
str
default:"\"tagger-rc1\""
The model’s release on Hugging Face.
​
str | Path | None
default:"None"
A folder to load the model from instead of downloading it, laid out as a release’s tagger/17m folder. size and revision are then unused.
​
int
default:"1"
The CPU threads for one sentence.
​
dict[ExpressionDelivery | ExpressionSound, list[str]] | None
default:"None"
Names to write in place of a label’s universal name, best first, such as {ExpressionDelivery.WARM: ["cheerful"]}. A name is written only where the TTS service performs it and the policy offers it. A name for ExpressionSound.BREATH is written for every breath. Raises ValueError when a key is a label the tagger doesn’t write.

​
What It Writes

It writes only tags the TTS service performs. The policy’s delivery and sounds narrow the names it writes, and its ExpressionGuidance presets steer it. The policy’s own lines go to the LLM.

​
Notes

  • Model download: each model loads while the pipeline is set up, once a process, shared by every reader or tagger. The first load downloads it from Hugging Face, which holds the pipeline’s start, so download it ahead, such as when you build your image, or pass model_dir.
  • A model that can’t load: it’s reported as an error that leaves the processor unusable. The reader passes transcriptions through unread, and with the tagger the LLM writes tags itself. The worker’s processor_unusable_policy decides whether the pipeline goes on.
  • Expressive conversation off: both processors need PipelineParams(enable_expressive_conversation=True) for the LLM to read readings or for tags to be performed.