> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Pipecat Voice Reader

> Reference for PipecatVoiceReader, PipecatVoiceSource, and PipecatVoiceClassifier, which hear how each user turn sounded on the CPU.

## Overview

Pipecat Voice Reader is a small model for [expressive conversation](https://docs.pipecat.ai/api-reference/server/expressive/overview.md) that runs on the CPU and hears how a clip of speech sounds: the speaker's tone of voice, sarcasm, sadness, laughter, and how activated and how fast the voice is.

* [`PipecatVoiceReader`](#pipecatvoicereader) hears each user turn, a perception producer that tells the LLM how the user sounded. [`PipecatVoiceSource`](#pipecatvoicesource) offers the same reading as a source for a reader with others. [`PipecatVoiceClassifier`](#pipecatvoiceclassifier) puts the same model behind the [classifier](https://docs.pipecat.ai/api-reference/server/classifiers/overview.md) interface.

For the reader of the user's words, see [Pipecat Tone](https://docs.pipecat.ai/api-reference/server/expressive/pipecat-tone.md). For a walkthrough with both, see [Expressive Conversation](https://docs.pipecat.ai/pipecat/features/expressive-conversation.md#hear-the-voice-too).

> [!WARNING]
>
> The model is a release candidate, in English only. Its readings are cues for
> the LLM to weigh with the words, not verdicts, and its outputs may change
> between releases. See the [model
> card](https://huggingface.co/pipecat-ai/voice-reader) for its evaluation and
> limitations.

> [card] [Model](https://huggingface.co/pipecat-ai/voice-reader)
>
> Pipecat Voice Reader on Hugging Face

> [card] [Example](https://github.com/pipecat-ai/pipecat/blob/main/examples/expressive-conversation/expressive-conversation-voice-reader.py)
>
> A bot with the voice reader beside the tone reader

## Installation

```bash theme={null}
uv add "pipecat-ai[expressivity]"
```

The same extra installs [Pipecat Tone](https://docs.pipecat.ai/api-reference/server/expressive/pipecat-tone.md). Pipecat Voice Reader needs Pipecat 1.16.0 or later.

## PipecatVoiceReader

Hears the user's speech each time they stop, up to its last 8 seconds, and sets the reading as the `perception` of the turn's transcriptions. Goes between the STT service and the user aggregator, in a pipeline with a VAD. The read runs off the event loop, and the transcriptions wait for it, up to `timeout` seconds. Speech shorter than 0.8 seconds isn't read. It is a [`PerceptionReader`](https://docs.pipecat.ai/api-reference/server/expressive/overview#perceptionreader) whose one source is a [`PipecatVoiceSource`](#pipecatvoicesource).

```python theme={null}
from pipecat.expressive.processors.voice_reader import PipecatVoiceReader
```

### Configuration

**variant** `Literal["base", "tiny"]` (default: "base")

The model's size. `"base"` reads tone and sadness best, and `"tiny"` costs
about half the CPU.

**revision** `str` (default: "rc6")

The model's release on Hugging Face. Each Pipecat version pins the release it
was tested with, so the readings change only when you upgrade Pipecat. Pass
another tag to pin a release yourself, from `"rc4"` on, or `"main"` to follow
the latest.

**model\_dir** `str | Path | None` (default: None)

A folder to load the model from instead of downloading it, laid out as one of
a release's variants. `variant` and `revision` are then unused.

**precision** `Literal["int8", "fp32"]` (default: "int8")

The weights to load.

**threads** `int` (default: 2)

The CPU threads for one read.

**sad** `bool` (default: False)

Whether to also report the `sad` signal.

**timeout** `float | None` (default: None)

Seconds a transcription waits for its reading after the user stops speaking,
`2` by default. After that it passes unread.

### Signals

| Signal | Read as | Reading |
| - | - | - |
| `tone` | Labels | The two likeliest tones of voice, with their confidences: `positive`, `neutral`, `sad` or `frustrated`. |
| `arousal` | Value | How activated the voice sounds, from 0 (flat, low energy) to 1 (highly animated). |
| `laughter` | Value | The chance the turn has laughter in it, from 0 to 1. |
| `speech_rate` | Value | The speaking rate in syllables per second. |
| `sarcasm_voice` | Value | The chance the delivery alone sounds sarcastic, from 0 to 1. Weigh it with the `sarcasm` of a source that reads the words. |
| `sad` | Value | The chance a listener would hear sadness in the turn, from 0 to 1. Reported only with `sad=True`. |

## PipecatVoiceSource

Pipecat Voice Reader as one of a `PerceptionReader`'s sources, reporting the same signals. Give it to a reader beside the sources of other classifiers, such as [`PipecatToneSource`](https://docs.pipecat.ai/api-reference/server/expressive/pipecat-tone#pipecattonesource), which reads the words. Another source's signals need names other than these.

```python theme={null}
from pipecat.expressive.processors.tone_reader import PipecatToneSource
from pipecat.expressive.processors.voice_reader import PipecatVoiceSource

reader = PerceptionReader(sources=[PipecatToneSource(), PipecatVoiceSource()])
```

Takes the same `variant`, `revision`, `model_dir`, `precision`, `threads`, `sad` and `timeout` as [`PipecatVoiceReader`](#configuration).

## PipecatVoiceClassifier

A [classifier](https://docs.pipecat.ai/api-reference/server/classifiers/overview.md) backed by the model. It answers only the questions built into the model, listed in `questions`, and any other question raises `ClassifierError`. `setup()` loads the model.

```python theme={null}
from pipecat.classifiers.voice.classifier import PipecatVoiceClassifier
```

### Configuration

Takes the same `variant`, `revision`, `model_dir`, `precision` and `threads` as [`PipecatVoiceReader`](#configuration).

### Questions

| Question | Kind | Asks |
| - | - | - |
| `tone` | `ChoiceQuestion` | How the voice sounds: `positive`, `neutral`, `sad` or `frustrated`. |
| `sarcasm` | `YesNoQuestion` | Whether a listener would hear the turn as sarcastic. |
| `laughter` | `YesNoQuestion` | Whether there is laughter in the turn. |
| `sad` | `YesNoQuestion` | Whether a listener would hear sadness in the turn. |
| `arousal` | `ScoreQuestion` | How activated the voice sounds. |
| `upset` | `ScoreQuestion` | How clearly the speaker is upset. |
| `speech_rate` | `ScoreQuestion` | The speaking rate, from 0 to 12 syllables per second. |

Every call takes exactly one `ClassifierAudio` clip in `inputs`, from 0.8 to 8 seconds long, at 16 kHz. Any other number of clips, or a clip outside that length, raises `ClassifierError`. The model hears the audio alone, so the state is ignored.

The model's `sarcasm` rarely passes 0.5, so `is_yes` is seldom true of it. Compare its probability with `sarcasm_threshold`, the probability from which the model's release counts a turn as sarcastic. `sarcasm_threshold` raises `ClassifierError` until `setup()` has loaded the model.

```python theme={null}
classifier = PipecatVoiceClassifier()
await classifier.setup(task_manager)

results = await classifier.ask(
    {},
    classifier.questions,
    inputs=[ClassifierAudio(audio=pcm, sample_rate=16000)],
)
tone = results["tone"].choice
```

## Notes

* **Model download**: the model loads while the pipeline is set up, once a process, shared by every reader of the same variant. The first load downloads it from Hugging Face, which holds the pipeline's start, so download it ahead, such as when you build your image, or pass `model_dir`.
* **A model that can't load**: it's reported as an error that leaves the reader unusable, and transcriptions pass through unread. The worker's [`processor_unusable_policy`](https://docs.pipecat.ai/api-reference/server/pipeline/pipeline-worker) decides whether the pipeline goes on.
* **Expressive conversation off**: the reader needs `PipelineParams(enable_expressive_conversation=True)` for the LLM to read readings.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.