OpenAI (Whisper)

​

Overview

OpenAISTTService provides speech-to-text capabilities using OpenAI’s hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements.

​

Installation

To use OpenAISTTService, install the required dependencies:

pip install pipecat-ai[openai]

You’ll need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY.

​

Configuration

​

Constructor Parameters

​

Input

The service processes audio data with the following requirements:

• PCM audio format
• 16-bit depth
• Single channel (mono)

​

Output Frames

The service produces two types of frames during transcription:

​

TranscriptionFrame

Generated for final transcriptions, containing:

​

ErrorFrame

Generated when transcription errors occur, containing error details.

​

Methods

​

Set Model

await service.set_model("whisper-1")

See the STT base class methods for additional functionality.

​

Usage Example

from pipecat.services.openai import OpenAISTTService

# Configure service
stt = OpenAISTTService(
    model="whisper-1",
    api_key="your-api-key"
)

# Use in pipeline
pipeline = Pipeline([
    transport.input(),
    stt,
    llm,
    ...
])

​

Frame Flow

​

Metrics Support

The service collects the following metrics:

• Time to First Byte (TTFB)
• Processing duration
• API response time

​

Notes

• Requires valid OpenAI API key
• Uses OpenAI’s hosted Whisper model
• Handles API rate limiting
• Automatic error handling
• Thread-safe processing

​

Error Handling

The service handles common API errors including:

• Authentication errors
• Rate limiting
• Invalid audio format
• Network connectivity issues
• API timeouts

Errors are propagated through ErrorFrames with descriptive messages.