Ultravox Realtime

Overview

UltravoxRealtimeLLMService provides real-time conversational AI capabilities using Ultravox’s Realtime API. It supports both text and audio modalities with voice transcription, streaming responses, and tool usage for creating interactive AI experiences.

Ultravox Realtime API Reference

Pipecat’s API methods for Ultravox Realtime integration

Example Implementation

Complete Ultravox Realtime conversation example

Async Tool Example

Ultravox with async function calling (cancel_on_interruption=False)

Ultravox Documentation

Official Ultravox API documentation

Ultravox Console

Access Ultravox models and manage API keys

Installation

To use Ultravox Realtime services, install the required dependencies:

uv add "pipecat-ai[ultravox]"

Prerequisites

Ultravox Account Setup

Before using Ultravox Realtime services, you need:

Ultravox Account: Sign up at Ultravox Console
API Key: Generate an Ultravox API key from your account dashboard
Model Access: Ensure access to Ultravox Realtime models
Usage Limits: Configure appropriate usage limits and billing

Required Environment Variables

ULTRAVOX_API_KEY: Your Ultravox API key for authentication

Key Features

Audio-Native Model: Ultravox is an audio-native model for natural voice interactions
Real-time Streaming: Low-latency audio processing and streaming responses
Multiple Input Modes: Support for Agent, One-Shot, and Join URL input parameters
Voice Transcription: Built-in transcription with streaming output
Function Calling: Support for tool integration and API calling
Configurable Duration: Set maximum call duration limits

Configuration

UltravoxRealtimeLLMService

params

AgentInputParams | OneShotInputParams | JoinUrlInputParams

required

Configuration parameters for connecting to Ultravox. One of three input parameter types must be provided. See Input Parameter Types below.

one_shot_selected_tools

ToolsSchema | List[FunctionSchema | DirectFunction]

default:"None"

Tools to use with a one-shot call: a ToolsSchema, or a plain list of direct functions and/or FunctionSchema objects. May only be set when using OneShotInputParams.

settings

UltravoxRealtimeLLMService.Settings

default:"None"

Runtime-configurable settings. See Settings below.

Settings

Runtime-configurable settings passed via the settings constructor argument using UltravoxRealtimeLLMService.Settings(...). These can be updated mid-conversation with LLMUpdateSettingsFrame. See Service Settings for details.

Parameter	Type	Default	Description
`model`	`str`	`NOT_GIVEN`	Model identifier. (Inherited from base settings.)
`output_medium`	`str`	`NOT_GIVEN`	Output medium: `"voice"` for audio or `"text"` for text output.

NOT_GIVEN values are omitted, letting the service use its own defaults. Only parameters that are explicitly set are included.

Input Parameter Types

Ultravox supports three different ways to create or join a call:

AgentInputParams

Use a pre-configured Ultravox Agent to handle calls consistently.

Parameter	Type	Default	Description
`api_key`	`str`	required	Ultravox API key for authentication.
`agent_id`	`UUID`	required	The ID of the Ultravox agent. Create and edit agents in the Ultravox Console.
`template_context`	`Dict[str, Any]`	`{}`	Context variables for agent template instantiation.
`metadata`	`Dict[str, str]`	`{}`	Metadata to attach to the call.
`max_duration`	`timedelta`	`None`	Maximum call duration (10s to 1h). `None` uses the agent’s default.
`extra`	`Dict[str, Any]`	`{}`	Extra parameters for the agent call creation request.

OneShotInputParams

Create a one-off call with inline configuration.

Parameter	Type	Default	Description
`api_key`	`str`	required	Ultravox API key for authentication.
`system_prompt`	`str`	`None`	System prompt to guide the model’s behavior.
`temperature`	`float`	`0.0`	Sampling temperature for response generation (0.0-1.0).
`model`	`str`	`None`	Model identifier to use (e.g., `"fixie-ai/ultravox"`).
`voice`	`UUID`	`None`	Voice identifier for speech generation.
`metadata`	`Dict[str, str]`	`{}`	Metadata to attach to the call.
`max_duration`	`timedelta`	`1 hour`	Maximum call duration (10s to 1h).
`extra`	`Dict[str, Any]`	`{}`	Extra parameters for the call creation request.

JoinUrlInputParams

Join an existing Ultravox call using a join URL.

Parameter	Type	Default	Description
`join_url`	`str`	required	The join URL for the existing Ultravox Realtime call.

Usage

Pair this service with LLMContextAggregatorPair(context, realtime_service_mode=True). Realtime mode keeps context-writing correct for speech-to-speech services and adapts turn handling to the service. See Realtime (Speech-to-Speech) Services.

Basic Setup with Agent

import os
import uuid
from pipecat.services.ultravox import UltravoxRealtimeLLMService, AgentInputParams

llm = UltravoxRealtimeLLMService(
    params=AgentInputParams(
        api_key=os.getenv("ULTRAVOX_API_KEY"),
        agent_id=uuid.UUID("your-agent-id-here"),
    ),
)

One-Shot Call

from pipecat.services.ultravox import UltravoxRealtimeLLMService, OneShotInputParams

llm = UltravoxRealtimeLLMService(
    params=OneShotInputParams(
        api_key=os.getenv("ULTRAVOX_API_KEY"),
        system_prompt="You are a helpful assistant.",
        temperature=0.3,
        model="fixie-ai/ultravox",
    ),
)

One-Shot with Tools

from pipecat.services.ultravox import UltravoxRealtimeLLMService, OneShotInputParams

llm = UltravoxRealtimeLLMService(
    params=OneShotInputParams(
        api_key=os.getenv("ULTRAVOX_API_KEY"),
        system_prompt="You are a helpful assistant that can check the weather.",
    ),
    one_shot_selected_tools=tools,  # ToolsSchema instance
)

@llm.function("get_weather")
async def get_weather(function_name, tool_call_id, args, llm, context, result_callback):
    location = args.get("location", "unknown")
    await result_callback({"temperature": 72, "condition": "sunny", "location": location})

Join Existing Call

from pipecat.services.ultravox import UltravoxRealtimeLLMService, JoinUrlInputParams

llm = UltravoxRealtimeLLMService(
    params=JoinUrlInputParams(
        join_url="wss://your-ultravox-join-url",
    ),
)

Switching Output Medium at Runtime

from pipecat.frames.frames import LLMUpdateSettingsFrame
from pipecat.services.ultravox.llm import UltravoxRealtimeLLMService

# Switch to text-only output
await worker.queue_frame(
    LLMUpdateSettingsFrame(
        delta=UltravoxRealtimeLLMService.Settings(
            output_medium="text",
        )
    )
)

# Switch back to voice output
await worker.queue_frame(
    LLMUpdateSettingsFrame(
        delta=UltravoxRealtimeLLMService.Settings(
            output_medium="voice",
        )
    )
)

Notes

Audio-native model: Ultravox processes audio directly rather than relying on a separate STT step. Voice transcriptions are provided for reference but may not always align with the model’s understanding of user input.
Server-side context management: Ultravox handles conversation context server-side. The LLM context in Pipecat is only used for passing function call results back to the service.
Audio sample rate: The service uses a 48kHz sample rate. Input audio at different sample rates is automatically resampled.
Output medium: The service supports both "voice" and "text" output modes, switchable at runtime using LLMUpdateSettingsFrame.
Call duration limits: When using AgentInputParams or OneShotInputParams, you can set a maximum call duration between 10 seconds and 1 hour.
Tools with agents: When using AgentInputParams, tools are configured on the agent itself. Use one_shot_selected_tools only with OneShotInputParams.
Async tool support: Functions registered with cancel_on_interruption=False are supported. When an async function is invoked, Ultravox ships a placeholder client_tool_result immediately to unfreeze the conversation (since Ultravox’s API freezes between tool invocation and result). The actual result is injected as user-side text once the tool completes. Streamed intermediate results (is_final=False) are not supported.

Pipecat Server

Client SDKs

Pipecat Flows

Pipecat Cloud

CLI

Pipecat Context Hub

Ultravox Realtime

Overview

Ultravox Realtime API Reference

Example Implementation

Async Tool Example

Ultravox Documentation

Ultravox Console

Installation

Prerequisites

Ultravox Account Setup

Required Environment Variables

Key Features

Configuration

UltravoxRealtimeLLMService

Settings

Input Parameter Types

AgentInputParams

OneShotInputParams

JoinUrlInputParams

Usage

Basic Setup with Agent

One-Shot Call

One-Shot with Tools

Join Existing Call

Switching Output Medium at Runtime

Notes

​Overview

Ultravox Realtime API Reference

Example Implementation

Async Tool Example

Ultravox Documentation

Ultravox Console

​Installation

​Prerequisites

​Ultravox Account Setup

​Required Environment Variables

​Key Features

​Configuration

​UltravoxRealtimeLLMService

​Settings

​Input Parameter Types

​AgentInputParams

​OneShotInputParams

​JoinUrlInputParams

​Usage

​Basic Setup with Agent

​One-Shot Call

​One-Shot with Tools

​Join Existing Call

​Switching Output Medium at Runtime

​Notes

Overview

Installation

Prerequisites

Ultravox Account Setup

Required Environment Variables

Key Features

Configuration

UltravoxRealtimeLLMService

Settings

Input Parameter Types

AgentInputParams

OneShotInputParams

JoinUrlInputParams

Usage

Basic Setup with Agent

One-Shot Call

One-Shot with Tools

Join Existing Call

Switching Output Medium at Runtime

Notes