OpenAI

Overview

OpenAILLMService provides chat completion capabilities using OpenAI’s API, supporting features like streaming responses, function calling, vision input, and advanced context management.

Installation

To use OpenAI services, install the required dependencies:

pip install "pipecat-ai[openai]"

You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY

You can obtain an OpenAI API key from the OpenAI platform.

OpenAILLMService

Constructor Parameters

model

str

default:"gpt-4o"

OpenAI model identifier. See OpenAI’s docs for the latest supported models.

api_key

str

OpenAI API key (defaults to environment variable)

base_url

str

Custom API endpoint URL for alternative OpenAI-compatible services.

organization

str

OpenAI organization identifier

project

str

OpenAI project identifier

params

InputParams

Model configuration parameters (see below)

Input Parameters

The params object accepts the following configuration settings:

frequency_penalty

float

Reduces likelihood of repeating tokens based on their frequency in the output so far.

Range: [-2.0, 2.0] where higher values reduce repetition more.

presence_penalty

float

Reduces likelihood of repeating any tokens that have appeared in the output so far.

Range: [-2.0, 2.0] where higher values reduce repetition more.

temperature

float

Controls randomness/creativity in the output. Lower values are more deterministic, higher values more creative.

Range: [0.0, 2.0]

top_p

float

Controls diversity via nucleus sampling. Only tokens with top_p cumulative probability are considered.

Range: [0.0, 1.0]

max_tokens

int

Maximum number of tokens to generate. Set to limit response length.

max_completion_tokens

int

Alternative way to specify maximum completion length.

seed

int

Seed for deterministic generation. Useful for reproducible outputs.

extra

Dict[str, Any]

Additional parameters to pass to the OpenAI API.

Input Frames

The service processes the following input frames:

OpenAILLMContextFrame

Frame

Contains OpenAI-specific conversation context

LLMMessagesFrame

Frame

Contains conversation messages

VisionImageRawFrame

Frame

Contains image for vision model processing

LLMUpdateSettingsFrame

Frame

Updates model settings

UserImageRequestFrame

Frame

Requests an image from a user

UserImageRawFrame

Frame

Contains user-provided image data

Output Frames

The service produces the following output frames:

LLMFullResponseStartFrame

Frame

Indicates the start of a response

LLMFullResponseEndFrame

Frame

Indicates the end of a response

LLMTextFrame

Frame

Contains streamed completion chunks

FunctionCallInProgressFrame

Frame

Indicates start of function call

FunctionCallResultFrame

Frame

Contains function call results

ErrorFrame

Frame

Contains error information

Methods

See the LLM base class methods for additional functionality.

Context Management

OpenAI’s API requires a specific format for conversation history and tools. The OpenAILLMContext class manages this conversation state, including:

Message history (user, assistant, system messages)
Function/tool definitions
Tool choice preferences
Image and multimedia content

Constructor Parameters

messages

List[ChatCompletionMessageParam]

Initial list of conversation messages. Each message should be an object with at least a “role” (user, assistant, or system) and “content” fields.

Defaults to an empty list.

tools

List[ChatCompletionToolParam] | NotGiven | ToolsSchema

Function definitions the model can call. Use for integrating external data sources or capabilities.

Defaults to NOT_GIVEN (no functions available).

tool_choice

ChatCompletionToolChoiceOptionParam | NotGiven

Controls when the model can call functions. Options include “auto” (model decides), “required” (must call a function), or a specific function name.

Defaults to NOT_GIVEN (auto).

Creating a Context

The simplest way to create a context is:

from pipecat.adapters.schemas.function_schema import FunctionSchema
from pipecat.adapters.schemas.tools_schema import ToolsSchema
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext

# Define weather function using standardized schema
weather_function = FunctionSchema(
    name="get_weather",
    description="Get current weather information",
    properties={
        "location": {
            "type": "string",
            "description": "The city name"
        }
    },
    required=["location"]
)

# Create tools schema
tools = ToolsSchema(standard_tools=[weather_function])

# Context with function calling enabled
context = OpenAILLMContext(
    messages=[{"role": "system", "content": "You are a helpful assistant."}],
    tools=tools
)

Context Aggregators

To integrate the context into a pipeline, you need paired aggregators that handle:

Adding user messages to the context before sending to the LLM
Capturing assistant responses and adding them to the context

# Create context aggregators
context_aggregator = llm.create_context_aggregator(context)

# Access individual aggregators for the pipeline
user_aggregator = context_aggregator.user()
assistant_aggregator = context_aggregator.assistant()

The context is shared between both aggregators, so all messages are captured in conversation history automatically.

Using Context Aggregators in a Pipeline

# Create pipeline with context aggregators
pipeline = Pipeline([
    transport.input(),          # Input from user
    stt,                        # Convert speech to text
    user_aggregator,            # Process user input and update context
    llm,                        # Generate response
    tts,                        # Convert response to speech
    transport.output(),         # Send output to user
    assistant_aggregator        # Store LLM responses in context
])

Multimodal Capabilities

OpenAI’s latest models (like GPT-4o) support multimodal inputs including images:

# Vision support is handled automatically when you send a VisionImageRawFrame
if isinstance(frame, VisionImageRawFrame):
    context = OpenAILLMContext()
    context.add_image_frame_message(
        format=frame.format,
        size=frame.size,
        image=frame.image,
        text="Describe this image in detail"
    )

Function Calling

This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to:

Check current weather conditions
Query databases
Access external APIs
Perform custom actions

Function Calling Guide

Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications.

Usage Examples

Basic Usage

from pipecat.pipeline import Pipeline
from pipecat.services.openai import OpenAILLMService
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext

# Configure the service
llm = OpenAILLMService(
    model="gpt-4o",
    params=OpenAILLMService.InputParams(
        temperature=0.7,
        max_tokens=1000
    )
)

# Create context and aggregators
context = OpenAILLMContext(
    messages=[{"role": "system", "content": "You are a helpful assistant."}]
)
context_aggregator = llm.create_context_aggregator(context)

# Create pipeline
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),
    context_aggregator.assistant()
])

With Function Calling

from pipecat.services.openai import OpenAILLMService
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.adapters.schemas.function_schema import FunctionSchema
from pipecat.adapters.schemas.tools_schema import ToolsSchema
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.task import PipelineParams, PipelineTask

# Configure the service
llm = OpenAILLMService(model="gpt-4o")

# Define the weather function using standardized schema
weather_function = FunctionSchema(
    name="get_weather",
    description="Get weather information",
    properties={
        "location": {"type": "string"}
    },
    required=["location"]  # Specify required parameters
)

# Create a tools schema
tools = ToolsSchema(standard_tools=[weather_function])

# Create context with system message and tools
context = OpenAILLMContext(
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant in a voice conversation. Keep responses concise."
        }
    ],
    tools=tools
)

# Register function handlers
async def fetch_weather(function_name, tool_call_id, args, llm, context, result_callback):
    await result_callback({"conditions": "nice", "temperature": "75"})

llm.register_function("get_current_weather", fetch_weather)

# Create context aggregator for message handling
context_aggregator = llm.create_context_aggregator(context)

# Set up pipeline
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),
    context_aggregator.assistant()
])

# Create and configure task
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        allow_interruptions=True,
        enable_metrics=True,
        enable_usage_metrics=True,
    ),
)

Frame Flow

Error Handling

The service handles common API errors including:

Authentication errors
Rate limiting
Invalid requests
Network connectivity issues
API timeouts

For unexpected errors, the service emits an ErrorFrame with details.

API Reference

Services

Utilities

Frameworks

Pipeline

Base Service Classes

Overview

Installation

OpenAILLMService

Constructor Parameters

Input Parameters

Input Frames

Output Frames

Methods

Context Management

Constructor Parameters

Creating a Context

Context Aggregators

Using Context Aggregators in a Pipeline

Multimodal Capabilities

Function Calling

Function Calling Guide

Usage Examples

Basic Usage

With Function Calling

Frame Flow

Error Handling

API Reference

Services

Utilities

Frameworks

Pipeline

Base Service Classes

​Overview

​Installation

​OpenAILLMService

​Constructor Parameters

​Input Parameters

​Input Frames

​Output Frames

​Methods

​Context Management

​Constructor Parameters

​Creating a Context

​Context Aggregators

​Using Context Aggregators in a Pipeline

​Multimodal Capabilities

​Function Calling

Function Calling Guide

​Usage Examples

​Basic Usage

​With Function Calling

​Frame Flow

​Error Handling

Overview

Installation

OpenAILLMService

Constructor Parameters

Input Parameters

Input Frames

Output Frames

Methods

Context Management

Constructor Parameters

Creating a Context

Context Aggregators

Using Context Aggregators in a Pipeline

Multimodal Capabilities

Function Calling

Usage Examples

Basic Usage

With Function Calling

Frame Flow

Error Handling