OpenAI

​

Overview

OpenAISTTService provides speech-to-text capabilities using OpenAI’s latest models, including the GPT-4o transcription model and the hosted Whisper API. It offers high-accuracy transcription with minimal setup requirements, using Voice Activity Detection (VAD) to process only speech segments.

​

Installation

To use OpenAISTTService, install the required dependencies:

pip install "pipecat-ai[openai]"

You’ll need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY.

​

Configuration

​

Constructor Parameters

​

Input

The service processes audio data with the following requirements:

• PCM audio format
• 16-bit depth
• Single channel (mono)

​

Output Frames

The service produces two types of frames during transcription:

​

TranscriptionFrame

Generated for final transcriptions, containing:

​

ErrorFrame

Generated when transcription errors occur, containing error details.

​

Methods

​

Set Model

await service.set_model("gpt-4o-transcribe")  # For the latest GPT-4o transcription model
# or
await service.set_model("whisper-1")  # For the Whisper model

See the STT base class methods for additional functionality.

​

Models

​

Language Support

OpenAI’s speech-to-text models support a wide range of languages. The service automatically maps Language enum values to the appropriate language codes.

The service will automatically detect the language if none is specified, but specifying the language typically improves transcription accuracy.

​

Usage Example

from pipecat.services.openai import OpenAISTTService

# Configure service
stt = OpenAISTTService(
    model="gpt-4o-transcribe",
    api_key="your-api-key",
    language=Language.EN,
    prompt="Transcribe technical terms accurately. Format numbers as digits rather than words."
)

# Use in pipeline
pipeline = Pipeline([
    transport.input(),
    stt,
    llm,
    ...
])

​

Voice Activity Detection Integration

This service inherits from SegmentedSTTService, which uses Voice Activity Detection (VAD) to identify speech segments for processing. This approach:

• Processes only actual speech, not silence or background noise
• Maintains a small audio buffer (default 1 second) to capture speech that occurs slightly before VAD detection
• Receives UserStartedSpeakingFrame and UserStoppedSpeakingFrame from a VAD component in the pipeline
• Only sends complete utterances to the API when speech has ended

​

Metrics Support

The service collects the following metrics:

• Time to First Byte (TTFB)
• Processing duration
• API response time

​

Notes

• Requires valid OpenAI API key
• GPT-4o transcription model offers superior accuracy to Whisper
• Requires VAD component in transport
• Handles API rate limiting
• Automatic error handling
• Thread-safe processing

​

Error Handling

The service handles common API errors including:

• Authentication errors
• Rate limiting
• Invalid audio format
• Network connectivity issues
• API timeouts

Errors are propagated through ErrorFrames with descriptive messages.