Overview
Pipecat Voice Reader is a small model for expressive conversation that runs on the CPU and hears how a clip of speech sounds: the speaker’s tone of voice, sarcasm, sadness, laughter, and how activated and how fast the voice is.PipecatVoiceReaderhears each user turn, a perception producer that tells the LLM how the user sounded.PipecatVoiceSourceoffers the same reading as a source for a reader with others.PipecatVoiceClassifierputs the same model behind the classifier interface.
Model
Pipecat Voice Reader on Hugging Face
Example
A bot with the voice reader beside the tone reader
Installation
PipecatVoiceReader
Hears the user’s speech each time they stop, up to its last 8 seconds, and sets the reading as theperception of the turn’s transcriptions. Goes between the STT service and the user aggregator, in a pipeline with a VAD. The read runs off the event loop, and the transcriptions wait for it, up to timeout seconds. Speech shorter than 0.8 seconds isn’t read. It is a PerceptionReader whose one source is a PipecatVoiceSource.
Configuration
Literal["base", "tiny"]
default:"\"base\""
The model’s size.
"base" reads tone and sadness best, and "tiny" costs
about half the CPU.str
default:"\"rc6\""
The model’s release on Hugging Face. Each Pipecat version pins the release it
was tested with, so the readings change only when you upgrade Pipecat. Pass
another tag to pin a release yourself, from
"rc4" on, or "main" to follow
the latest.str | Path | None
default:"None"
A folder to load the model from instead of downloading it, laid out as one of
a release’s variants.
variant and revision are then unused.Literal["int8", "fp32"]
default:"\"int8\""
The weights to load.
int
default:"2"
The CPU threads for one read.
bool
default:"False"
Whether to also report the
sad signal.float | None
default:"None"
Seconds a transcription waits for its reading after the user stops speaking,
2 by default. After that it passes unread.Signals
PipecatVoiceSource
Pipecat Voice Reader as one of aPerceptionReader’s sources, reporting the same signals. Give it to a reader beside the sources of other classifiers, such as PipecatToneSource, which reads the words. Another source’s signals need names other than these.
variant, revision, model_dir, precision, threads, sad and timeout as PipecatVoiceReader.
PipecatVoiceClassifier
A classifier backed by the model. It answers only the questions built into the model, listed inquestions, and any other question raises ClassifierError. setup() loads the model.
Configuration
Takes the samevariant, revision, model_dir, precision and threads as PipecatVoiceReader.
Questions
Every call takes exactly one
ClassifierAudio clip in inputs, from 0.8 to 8 seconds long, at 16 kHz. Any other number of clips, or a clip outside that length, raises ClassifierError. The model hears the audio alone, so the state is ignored.
The model’s sarcasm rarely passes 0.5, so is_yes is seldom true of it. Compare its probability with sarcasm_threshold, the probability from which the model’s release counts a turn as sarcastic. sarcasm_threshold raises ClassifierError until setup() has loaded the model.
Notes
- Model download: the model loads while the pipeline is set up, once a process, shared by every reader of the same variant. The first load downloads it from Hugging Face, which holds the pipeline’s start, so download it ahead, such as when you build your image, or pass
model_dir. - A model that can’t load: it’s reported as an error that leaves the reader unusable, and transcriptions pass through unread. The worker’s
processor_unusable_policydecides whether the pipeline goes on. - Expressive conversation off: the reader needs
PipelineParams(enable_expressive_conversation=True)for the LLM to read readings.