OpenAI Realtime

Overview

OpenAIRealtimeLLMService provides real-time, multimodal conversation capabilities using OpenAI’s Realtime API. It supports speech-to-speech interactions with integrated LLM processing, function calling, and advanced conversation management with minimal latency response times.

OpenAI Realtime API Reference

Pipecat’s API methods for OpenAI Realtime integration

Example Implementation

Complete OpenAI Realtime conversation example

OpenAI Documentation

Official OpenAI Realtime API documentation

OpenAI Platform

Access Realtime models and manage API keys

Installation

To use OpenAI Realtime services, install the required dependencies:

uv add "pipecat-ai[openai]"

Prerequisites

OpenAI Account Setup

Before using OpenAI Realtime services, you need:

OpenAI Account: Sign up at OpenAI Platform
API Key: Generate an OpenAI API key from your account dashboard
Model Access: Ensure access to OpenAI Realtime models
Usage Limits: Configure appropriate usage limits and billing

Required Environment Variables

OPENAI_API_KEY: Your OpenAI API key for authentication

Key Features

Real-time Speech-to-Speech: Direct audio processing with minimal latency
Advanced Turn Detection: Multiple voice activity detection options including semantic detection
Function Calling: Seamless support for external functions and APIs
Voice Options: Multiple voice personalities and speaking styles
Conversation Management: Intelligent context handling and conversation flow control

Configuration

OpenAIRealtimeLLMService

str

required

OpenAI API key for authentication.

str

default:"gpt-realtime-2.1"

deprecated

Deprecated since 0.0.105, removed in 2.0.0. Use settings=OpenAIRealtimeLLMService.Settings(model=...) instead.

str

default:"wss://api.openai.com/v1/realtime"

WebSocket base URL for the Realtime API. Override for custom or proxied deployments.

SessionProperties

default:"None"

deprecated

Deprecated since 0.0.105, removed in 2.0.0. Use settings=OpenAIRealtimeLLMService.Settings(session_properties=...) instead.

OpenAIRealtimeLLMService.Settings

default:"None"

Runtime-configurable settings. See Settings below.

bool

default:"False"

Whether to start with audio input paused. Useful when you want to control when audio processing begins.

bool

default:"False"

Whether to start with video input paused.

Literal["auto", "low", "high"]

default:"auto"

Detail level for video processing. Can be "auto", "low", or "high". "auto" lets the model decide, "low" is faster and uses fewer tokens, "high" provides more detail.

float | None

default:"None"

In manual turn-detection mode (turn_detection=False, locally-driven turns), how much recent audio to replay after an interruption clears the input audio buffer, so the speech onset isn’t lost. Defaults to None: auto-sized to the upstream VAD’s start_secs plus 0.3 seconds, falling back to 0.5 seconds when no VAD is present. Auto-sizing assumes VAD drives turn starts (the default VADUserTurnStartStrategy); set this explicitly if you use a non-VAD turn-start strategy. No effect when server-side turn detection is enabled.

Any

Additional arguments passed to parent LLMService.

Settings

Runtime-configurable settings passed via the settings constructor argument using OpenAIRealtimeLLMService.Settings(...). system_instruction and session_properties can be updated mid-conversation with LLMUpdateSettingsFrame; model is fixed for the session. See Service Settings for details.

Parameter	Type	Default	Description
`model`	`str`	`"gpt-realtime-2.1"`	Model identifier. (Inherited from base settings.)
`system_instruction`	`str`	`None`	System instruction/prompt. (Inherited from base settings.)
`session_properties`	`SessionProperties`	`SessionProperties()`	Session-level configuration (modalities, audio, tools, etc.). In an update, it replaces the stored `SessionProperties` as a whole rather than merging into it.

None values are omitted, letting the service use its own defaults.

The inherited LLMSettings fields temperature, max_tokens, top_p, top_k, frequency_penalty, presence_penalty, and seed are not sent by this service, so they have no effect. Use session_properties.max_output_tokens to limit response length.

SessionProperties

Parameter	Type	Default	Description
`output_modalities`	`List[Literal["text", "audio"]]`	`None`	Modalities the model can respond with. The API supports single modality responses: either `["text"]` or `["audio"]`.
`instructions`	`str`	`None`	System instructions for the assistant.
`audio`	`AudioConfiguration`	`None`	Configuration for input and output audio (format, transcription, turn detection, voice, speed).
`tools`	`ToolsSchema \| list[FunctionSchema \| DirectFunction] \| list[dict] \| None`	`None`	Available function tools for the assistant. Tools set on the LLM context take precedence.
`tool_choice`	`Literal["auto", "none", "required"]`	`None`	Tool usage strategy.
`max_output_tokens`	`int \| Literal["inf"]`	`None`	Maximum tokens in response, or `"inf"` for unlimited.
`tracing`	`Literal["auto"] \| Dict`	`None`	Configuration options for tracing.
`prompt`	`dict`	`None`	Reference to a prompt template and its variables.
`include`	`list[str]`	`None`	Additional fields to include in server outputs.
`reasoning`	`Reasoning`	`None`	Reasoning configuration. Only supported by reasoning-capable models such as `gpt-realtime-2`.

AudioConfiguration

The audio field in SessionProperties accepts an AudioConfiguration with input and output sub-configurations: AudioInput (audio.input):

Parameter	Type	Default	Description
`format`	`AudioFormat`	`None`	Input audio format (`PCMAudioFormat`, `PCMUAudioFormat`, or `PCMAAudioFormat`).
`transcription`	`InputAudioTranscription`	`None`	Transcription settings: `model` (e.g. `"gpt-realtime-whisper"`, `"gpt-4o-transcribe"`), `language`, and `prompt` (not supported by `"gpt-realtime-whisper"`).
`noise_reduction`	`InputAudioNoiseReduction`	`None`	Noise reduction type: `"near_field"` or `"far_field"`.
`turn_detection`	`TurnDetection \| SemanticTurnDetection \| bool`	`None`	Turn detection config, or `False` to disable server-side turn detection.

AudioOutput (audio.output):

Parameter	Type	Default	Description
`format`	`AudioFormat`	`None`	Output audio format.
`voice`	`str`	`None`	Voice the model uses to respond (e.g. `"alloy"`, `"echo"`, `"shimmer"`).
`speed`	`float`	`None`	Speed of the model’s spoken response.

TurnDetection

Server-side VAD configuration via TurnDetection:

Parameter	Type	Default	Description
`type`	`Literal["server_vad"]`	`"server_vad"`	Detection type.
`threshold`	`float`	`0.5`	Voice activity detection threshold (0.0-1.0).
`prefix_padding_ms`	`int`	`300`	Padding before speech starts in milliseconds.
`silence_duration_ms`	`int`	`500`	Silence duration to detect speech end in milliseconds.

Alternatively, use SemanticTurnDetection for semantic-based detection:

Parameter	Type	Default	Description
`type`	`Literal["semantic_vad"]`	`"semantic_vad"`	Detection type.
`eagerness`	`Literal["low", "medium", "high", "auto"]`	`None`	Turn detection eagerness level.
`create_response`	`bool`	`None`	Whether to automatically create responses on turn detection.
`interrupt_response`	`bool`	`None`	Whether to interrupt ongoing responses on turn detection.

Reasoning

Reasoning configuration for reasoning-capable Realtime models (e.g. gpt-realtime-2):

Parameter	Type	Default	Description
`effort`	`Literal["minimal", "low", "medium", "high", "xhigh"]`	`None`	How much reasoning effort the model should apply. Omit to let the server pick the default for the model.

Reasoning configuration is automatically filtered out when used with non-reasoning models (e.g. gpt-realtime-1.5) to prevent session errors. A warning is logged when this filtering occurs.

Usage

Basic Setup

import os
from pipecat.services.openai.realtime.llm import OpenAIRealtimeLLMService

llm = OpenAIRealtimeLLMService(
    api_key=os.getenv("OPENAI_API_KEY"),
    settings=OpenAIRealtimeLLMService.Settings(model="gpt-realtime-2"),
)

With Session Configuration

from pipecat.services.openai.realtime.llm import OpenAIRealtimeLLMService
from pipecat.services.openai.realtime.events import (
    SessionProperties,
    AudioConfiguration,
    AudioInput,
    AudioOutput,
    InputAudioTranscription,
    SemanticTurnDetection,
    Reasoning,
)

session_properties = SessionProperties(
    audio=AudioConfiguration(
        input=AudioInput(
            transcription=InputAudioTranscription(model="gpt-realtime-whisper"),
            turn_detection=SemanticTurnDetection(eagerness="medium"),
        ),
        output=AudioOutput(
            voice="alloy",
            speed=1.0,
        ),
    ),
    max_output_tokens=4096,
)

llm = OpenAIRealtimeLLMService(
    api_key=os.getenv("OPENAI_API_KEY"),
    settings=OpenAIRealtimeLLMService.Settings(
        model="gpt-realtime-2",
        session_properties=session_properties,
        system_instruction="You are a helpful assistant.",
    ),
)

With Disabled Turn Detection (Manual Control)

session_properties = SessionProperties(
    audio=AudioConfiguration(
        input=AudioInput(
            turn_detection=False,
        ),
    ),
)

llm = OpenAIRealtimeLLMService(
    api_key=os.getenv("OPENAI_API_KEY"),
    settings=OpenAIRealtimeLLMService.Settings(
        model="gpt-realtime-2",
        session_properties=session_properties,
        system_instruction="You are a helpful assistant.",
    ),
)

With Reasoning Configuration (gpt-realtime-2)

from pipecat.services.openai.realtime.llm import OpenAIRealtimeLLMService
from pipecat.services.openai.realtime.events import SessionProperties, Reasoning

session_properties = SessionProperties(
    reasoning=Reasoning(effort="high"),
)

llm = OpenAIRealtimeLLMService(
    api_key=os.getenv("OPENAI_API_KEY"),
    settings=OpenAIRealtimeLLMService.Settings(
        model="gpt-realtime-2",
        session_properties=session_properties,
        system_instruction="You are a helpful assistant.",
    ),
)

Updating Settings at Runtime

from pipecat.frames.frames import LLMUpdateSettingsFrame
from pipecat.services.openai.realtime.llm import OpenAIRealtimeLLMService
from pipecat.services.openai.realtime.events import Reasoning

# Copy the full session config and change one field
session_properties = session_properties.model_copy(update={"max_output_tokens": 2048})

# Update system instruction and max tokens
await worker.queue_frame(
    LLMUpdateSettingsFrame(
        delta=OpenAIRealtimeLLMService.Settings(
            system_instruction="Now speak in Spanish.",
            session_properties=session_properties,
        )
    )
)

# Update reasoning effort (gpt-realtime-2 only)
session_properties = session_properties.model_copy(
    update={"reasoning": Reasoning(effort="xhigh")}
)
await worker.queue_frame(
    LLMUpdateSettingsFrame(
        delta=OpenAIRealtimeLLMService.Settings(
            session_properties=session_properties,
        )
    )
)

Methods

set_audio_input_paused()

def set_audio_input_paused(paused: bool)

Stops (True) or resumes (False) sending input audio to the model.

set_video_input_paused()

def set_video_input_paused(paused: bool)

Stops (True) or resumes (False) sending video frames to the model.

retrieve_conversation_item()

async def retrieve_conversation_item(item_id: str) -> ConversationItem

Returns the server’s copy of a conversation item, such as one reported to on_conversation_item_created. Raises an Exception with the API’s message when no item has that ID.

Notes

Model is connection-level: The model parameter is set via the WebSocket URL at connection time and cannot be changed during a session.
Output modalities are single-mode: The API supports either ["text"] or ["audio"] output, not both simultaneously.
Turn detection options: Use TurnDetection for traditional VAD, SemanticTurnDetection for AI-based turn detection, or False to disable server-side detection and manage turns manually.
Manual turn detection pre-roll: When server-side turn detection is disabled (turn_detection=False), the service maintains a rolling audio buffer that is replayed after interruptions to preserve speech onsets. Configure the buffer duration with user_audio_preroll_secs or let it auto-size from the upstream VAD’s start_secs.
Audio sample rates: Output audio frames are always labeled 24 kHz mono PCM, so leave audio.output.format unset or use PCMAudioFormat(). Input audio is resampled to 24 kHz before it is sent, so any audio_in_sample_rate works.
G.711 formats: The service passes audio through without converting it, and pipeline audio frames are 16-bit PCM, so use PCMAudioFormat rather than PCMUAudioFormat or PCMAAudioFormat; for telephony, the transport’s serializer converts to and from G.711.
Video support: Video frames can be sent to the model for multimodal input, at most one per second; frames in between are dropped. Control the detail level with video_frame_detail and pause/resume with set_video_input_paused().
Transcription frames: User speech transcription frames are always emitted upstream when input audio transcription is configured.
System instruction precedence: The system_instruction from service settings takes precedence over an initial system message in the LLM context. A warning is logged when both are set.
Reasoning support: The reasoning configuration is only supported by reasoning-capable models such as gpt-realtime-2. When used with non-supporting models, the reasoning configuration is automatically filtered out to prevent session errors, and a warning is logged.

Event Handlers

Event	Description
`on_conversation_item_created`	Called when a new conversation item is created in the session
`on_conversation_item_updated`	Called when a conversation item is updated or completed. `item` is `None` when a user audio transcription completes.

@llm.event_handler("on_conversation_item_created")
async def on_item_created(service, item_id, item):
    print(f"New conversation item: {item_id}")

@llm.event_handler("on_conversation_item_updated")
async def on_item_updated(service, item_id, item):
    print(f"Conversation item updated: {item_id}")

Get Started

Pipecat Server

Client SDKs

Pipecat Flows

Pipecat Cloud

CLI

Pipecat Context Hub

OpenAI Realtime

Overview

OpenAI Realtime API Reference

Example Implementation

OpenAI Documentation

OpenAI Platform

Installation

Prerequisites

OpenAI Account Setup

Required Environment Variables

Key Features

Configuration

OpenAIRealtimeLLMService

Settings

SessionProperties

AudioConfiguration

TurnDetection

Reasoning

Usage

Basic Setup

With Session Configuration

With Disabled Turn Detection (Manual Control)

With Reasoning Configuration (gpt-realtime-2)

Updating Settings at Runtime

Methods

set_audio_input_paused()

set_video_input_paused()

retrieve_conversation_item()

Notes

Event Handlers

​Overview

OpenAI Realtime API Reference

Example Implementation

OpenAI Documentation

OpenAI Platform

​Installation

​Prerequisites

​OpenAI Account Setup

​Required Environment Variables

​Key Features

​Configuration

​OpenAIRealtimeLLMService

​Settings

​SessionProperties

​AudioConfiguration

​TurnDetection

​Reasoning

​Usage

​Basic Setup

​With Session Configuration

​With Disabled Turn Detection (Manual Control)

​With Reasoning Configuration (gpt-realtime-2)

​Updating Settings at Runtime

​Methods

​set_audio_input_paused()

​set_video_input_paused()

​retrieve_conversation_item()

​Notes

​Event Handlers

Overview

Installation

Prerequisites

OpenAI Account Setup

Required Environment Variables

Key Features

Configuration

OpenAIRealtimeLLMService

Settings

SessionProperties

AudioConfiguration

TurnDetection

Reasoning

Usage

Basic Setup

With Session Configuration

With Disabled Turn Detection (Manual Control)

With Reasoning Configuration (gpt-realtime-2)

Updating Settings at Runtime

Methods

set_audio_input_paused()

set_video_input_paused()

retrieve_conversation_item()

Notes

Event Handlers