Whisper Speech-to-Text

​

Overview

​

Installation

​

Standard Whisper (CPU/CUDA)

uv add "pipecat-ai[whisper]"

​

MLX Whisper (Apple Silicon)

uv add "pipecat-ai[mlx-whisper]"

​

Prerequisites

​

Local Model Setup

1. Model Selection: Choose appropriate Whisper model size (tiny, base, small, medium, large)
2. Hardware Configuration: Set up CPU, CUDA, or Apple Silicon acceleration
3. Storage Space: Ensure sufficient disk space for model downloads

​

Configuration Options

• Model Size: Balance between accuracy and performance based on your hardware
• Hardware Acceleration: Configure CUDA for NVIDIA GPUs or MLX for Apple Silicon
• Language Support: Whisper supports 99+ languages out of the box

​

Configuration

​

WhisperSTTService

​

WhisperSTTServiceMLX

​

WhisperSTTService Settings

​

WhisperSTTServiceMLX Settings

​

Usage

​

Basic Faster Whisper Setup

from pipecat.services.whisper.stt import WhisperSTTService

stt = WhisperSTTService(
    settings=WhisperSTTService.Settings(
        model="base",
    ),
)

​

With CUDA Acceleration

from pipecat.services.whisper.stt import WhisperSTTService, Model

stt = WhisperSTTService(
    device="cuda",
    compute_type="float16",
    settings=WhisperSTTService.Settings(
        model=Model.LARGE,
    ),
)

​

With Custom Language

from pipecat.services.whisper.stt import WhisperSTTService, Model
from pipecat.transcriptions.language import Language

stt = WhisperSTTService(
    settings=WhisperSTTService.Settings(
        model=Model.MEDIUM,
        language=Language.FR,
        no_speech_prob=0.5,
    ),
)

​

MLX Whisper on Apple Silicon

from pipecat.services.whisper.stt import WhisperSTTServiceMLX, MLXModel
from pipecat.transcriptions.language import Language

stt = WhisperSTTServiceMLX(
    settings=WhisperSTTServiceMLX.Settings(
        model=MLXModel.LARGE_V3_TURBO,
        language=Language.EN,
        temperature=0.0,
    ),
)

​

Notes

• First run downloads: If the selected model hasn’t been downloaded previously, the first run will download it from the Hugging Face model hub. This may take significant time depending on model size.
• Segmented transcription: Both WhisperSTTService and WhisperSTTServiceMLX extend SegmentedSTTService, meaning they process complete audio segments after VAD detects the user has stopped speaking.
• Audio sample rate: Whisper models expect 16 kHz audio and the services don’t resample, so keep the pipeline’s input sample rate (PipelineParams.audio_in_sample_rate) at its default of 16000.
• No-speech filtering: The no_speech_prob threshold helps filter out hallucinations. Increase it to be more permissive, decrease it to filter more aggressively.
• MLX platform requirement: WhisperSTTServiceMLX requires macOS on Apple Silicon (arm64). On other platforms (including Intel Macs), use WhisperSTTService instead.
• MLX quantization: The LARGE_V3_TURBO_Q4 model provides reduced memory usage with minimal quality loss on Apple Silicon.
• Model enums: Model and MLXModel are StrEnum types, meaning enum members can be compared directly to strings (e.g., Model.TINY == "tiny"). Both enum members and plain strings work when setting the model.
• Language support: Whisper supports 99+ languages. Use the Language enum for type-safe language selection. The default is Language.EN (English). Set language=None in settings to enable automatic language detection, which will transcribe whatever language the user speaks.
• English-only model validation: Models with .en in their name (including the default DISTIL_MEDIUM_EN) only support English. Constructing WhisperSTTService with a language its loaded model doesn’t support raises a ValueError naming the model and its supported languages, and a later switch to such a language pushes an ErrorFrame. WhisperSTTServiceMLX doesn’t run this check, and accepts runtime language changes. Use a multilingual model (e.g., LARGE_V3_TURBO) for non-English transcription.