​
Overview

OpenAILLMService provides chat completion capabilities using OpenAI’s API, supporting features like streaming responses, function calling, vision input, and advanced context management.

​
Installation

To use OpenAI services, install the required dependencies:

pip install "pipecat-ai[openai]"

You’ll also need to set up your OpenAI API key as an environment variable: OPENAI_API_KEY

You can obtain an OpenAI API key from the OpenAI platform.

​
OpenAILLMService

​
Constructor Parameters

​
model
str
default:"gpt-4o"

OpenAI model identifier. See OpenAI’s docs for the latest supported models.

​
api_key
str

OpenAI API key (defaults to environment variable)

​
base_url
str

Custom API endpoint URL for alternative OpenAI-compatible services.

​
organization
str

OpenAI organization identifier

​
project
str

OpenAI project identifier

​
params
InputParams

Model configuration parameters (see below)

​
Input Parameters

The params object accepts the following configuration settings:

​
frequency_penalty
float

Reduces likelihood of repeating tokens based on their frequency in the output so far.

Range: [-2.0, 2.0] where higher values reduce repetition more.

​
presence_penalty
float

Reduces likelihood of repeating any tokens that have appeared in the output so far.

Range: [-2.0, 2.0] where higher values reduce repetition more.

​
temperature
float

Controls randomness/creativity in the output. Lower values are more deterministic, higher values more creative.

Range: [0.0, 2.0]

​
top_p
float

Controls diversity via nucleus sampling. Only tokens with top_p cumulative probability are considered.

Range: [0.0, 1.0]

​
max_tokens
int

Maximum number of tokens to generate. Set to limit response length.

​
max_completion_tokens
int

Alternative way to specify maximum completion length.

​
seed
int

Seed for deterministic generation. Useful for reproducible outputs.

​
extra
Dict[str, Any]

Additional parameters to pass to the OpenAI API.

​
Input Frames

The service processes the following input frames:

​
OpenAILLMContextFrame
Frame

Contains OpenAI-specific conversation context

​
LLMMessagesFrame
Frame

Contains conversation messages

​
VisionImageRawFrame
Frame

Contains image for vision model processing

​
LLMUpdateSettingsFrame
Frame

Updates model settings

​
UserImageRequestFrame
Frame

Requests an image from a user

​
UserImageRawFrame
Frame

Contains user-provided image data

​
Output Frames

The service produces the following output frames:

​
LLMFullResponseStartFrame
Frame

Indicates the start of a response

​
LLMFullResponseEndFrame
Frame

Indicates the end of a response

​
LLMTextFrame
Frame

Contains streamed completion chunks

​
FunctionCallInProgressFrame
Frame

Indicates start of function call

​
FunctionCallResultFrame
Frame

Contains function call results

​
ErrorFrame
Frame

Contains error information

​
Methods

See the LLM base class methods for additional functionality.

​
Context Management

OpenAI’s API requires a specific format for conversation history and tools. The OpenAILLMContext class manages this conversation state, including:

  • Message history (user, assistant, system messages)
  • Function/tool definitions
  • Tool choice preferences
  • Image and multimedia content

​
Constructor Parameters

​
messages
List[ChatCompletionMessageParam]

Initial list of conversation messages. Each message should be an object with at least a “role” (user, assistant, or system) and “content” fields.

Defaults to an empty list.

​
tools
List[ChatCompletionToolParam] | NotGiven | ToolsSchema

Function definitions the model can call. Use for integrating external data sources or capabilities.

Defaults to NOT_GIVEN (no functions available).

​
tool_choice
ChatCompletionToolChoiceOptionParam | NotGiven

Controls when the model can call functions. Options include “auto” (model decides), “required” (must call a function), or a specific function name.

Defaults to NOT_GIVEN (auto).

​
Creating a Context

The simplest way to create a context is:

from pipecat.adapters.schemas.function_schema import FunctionSchema
from pipecat.adapters.schemas.tools_schema import ToolsSchema
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext

# Define weather function using standardized schema
weather_function = FunctionSchema(
    name="get_weather",
    description="Get current weather information",
    properties={
        "location": {
            "type": "string",
            "description": "The city name"
        }
    },
    required=["location"]
)

# Create tools schema
tools = ToolsSchema(standard_tools=[weather_function])

# Context with function calling enabled
context = OpenAILLMContext(
    messages=[{"role": "system", "content": "You are a helpful assistant."}],
    tools=tools
)

​
Context Aggregators

To integrate the context into a pipeline, you need paired aggregators that handle:

  1. Adding user messages to the context before sending to the LLM
  2. Capturing assistant responses and adding them to the context
# Create context aggregators
context_aggregator = llm.create_context_aggregator(context)

# Access individual aggregators for the pipeline
user_aggregator = context_aggregator.user()
assistant_aggregator = context_aggregator.assistant()

The context is shared between both aggregators, so all messages are captured in conversation history automatically.

​
Using Context Aggregators in a Pipeline

# Create pipeline with context aggregators
pipeline = Pipeline([
    transport.input(),          # Input from user
    stt,                        # Convert speech to text
    user_aggregator,            # Process user input and update context
    llm,                        # Generate response
    tts,                        # Convert response to speech
    transport.output(),         # Send output to user
    assistant_aggregator        # Store LLM responses in context
])

​
Multimodal Capabilities

OpenAI’s latest models (like GPT-4o) support multimodal inputs including images:

# Vision support is handled automatically when you send a VisionImageRawFrame
if isinstance(frame, VisionImageRawFrame):
    context = OpenAILLMContext()
    context.add_image_frame_message(
        format=frame.format,
        size=frame.size,
        image=frame.image,
        text="Describe this image in detail"
    )

​
Function Calling

This service supports function calling (also known as tool calling) which allows the LLM to request information from external services and APIs. For example, you can enable your bot to:

  • Check current weather conditions
  • Query databases
  • Access external APIs
  • Perform custom actions

Function Calling Guide

Learn how to implement function calling with standardized schemas, register handlers, manage context properly, and control execution flow in your conversational AI applications.

​
Usage Examples

​
Basic Usage

from pipecat.pipeline import Pipeline
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext

# Configure the service
llm = OpenAILLMService(
    model="gpt-4o",
    params=OpenAILLMService.InputParams(
        temperature=0.7,
        max_tokens=1000
    )
)

# Create context and aggregators
context = OpenAILLMContext(
    messages=[{"role": "system", "content": "You are a helpful assistant."}]
)
context_aggregator = llm.create_context_aggregator(context)

# Create pipeline
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),
    context_aggregator.assistant()
])

​
With Function Calling

from pipecat.services.openai.llm import OpenAILLMService
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.adapters.schemas.function_schema import FunctionSchema
from pipecat.adapters.schemas.tools_schema import ToolsSchema
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.services.llm_service import FunctionCallParams

# Configure the service
llm = OpenAILLMService(model="gpt-4o")

# Define the weather function using standardized schema
weather_function = FunctionSchema(
    name="get_weather",
    description="Get weather information",
    properties={
        "location": {"type": "string"}
    },
    required=["location"]  # Specify required parameters
)

# Create a tools schema
tools = ToolsSchema(standard_tools=[weather_function])

# Create context with system message and tools
context = OpenAILLMContext(
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant in a voice conversation. Keep responses concise."
        }
    ],
    tools=tools
)

# Register function handlers
async def fetch_weather(params: FunctionCallParams):
    await params.result_callback({"conditions": "nice", "temperature": "75"})

llm.register_function("get_current_weather", fetch_weather)

# Create context aggregator for message handling
context_aggregator = llm.create_context_aggregator(context)

# Set up pipeline
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),
    context_aggregator.assistant()
])

# Create and configure task
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        allow_interruptions=True,
        enable_metrics=True,
        enable_usage_metrics=True,
    ),
)

​
Frame Flow

​
Error Handling

The service handles common API errors including:

  • Authentication errors
  • Rate limiting
  • Invalid requests
  • Network connectivity issues
  • API timeouts

For unexpected errors, the service emits an ErrorFrame with details.