​
Overview

TogetherLLMService provides access to Together AI’s language models, including Meta’s Llama 3.1 and 3.2 models, through an OpenAI-compatible interface. It inherits from OpenAILLMService and maintains compatibility with OpenAI’s function calling format.

​
Installation

To use TogetherLLMService, install the required dependencies:

pip install pipecat-ai[together]

You’ll also need to set up your Together API key as an environment variable: TOGETHER_API_KEY

​
Configuration

​
Constructor Parameters

api_key
str
required

Your Together AI API key

base_url
str
default: "https://api.together.xyz/v1"

Together AI API endpoint

model
str
default: "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo"

Model identifier

​
Input Parameters

Inherits all input parameters from OpenAILLMService:

extra
Optional[Dict[str, Any]]

Additional parameters to pass to the model

frequency_penalty
Optional[float]

Reduces likelihood of repeating tokens based on their frequency. Range: [-2.0, 2.0]

max_completion_tokens
Optional[int]

Maximum number of tokens in the completion. Must be greater than or equal to 1

max_tokens
Optional[int]

Maximum number of tokens to generate. Must be greater than or equal to 1

presence_penalty
Optional[float]

Reduces likelihood of repeating any tokens that have appeared. Range: [-2.0, 2.0]

seed
Optional[int]

Random seed for deterministic generation. Must be greater than or equal to 0

temperature
Optional[float]

Controls randomness in the output. Range: [0.0, 2.0]

top_p
Optional[float]

Controls diversity via nucleus sampling. Range: [0.0, 1.0]

​
Input Frames

OpenAILLMContextFrame
Frame

Contains OpenAI-specific conversation context

LLMMessagesFrame
Frame

Contains conversation messages

VisionImageRawFrame
Frame

Contains image for vision model processing

LLMUpdateSettingsFrame
Frame

Updates model settings

​
Output Frames

TextFrame
Frame

Contains generated text chunks

FunctionCallInProgressFrame
Frame

Indicates start of function call

FunctionCallResultFrame
Frame

Contains function call results

​
Context Management

The Together service uses specialized context management to handle conversations and message formatting. It relies on the OpenAI base class for context management, which includes managing the conversation history, system prompts, tool calls, and converting between OpenAI and Together message formats.

​
OpenAILLMContext

The base context manager for OpenAI conversations:

context = OpenAILLMContext(
    messages=[],  # Conversation history
    tools=[],     # Available function calling tools
    system="You are a helpful assistant"  # System prompt
)

​
Context Aggregators

Context aggregators handle message format conversion and management. The service provides a method to create paired aggregators:

create_context_aggregator
static method

Creates user and assistant aggregators for handling message formatting.

@staticmethod
def create_context_aggregator(
    context: OpenAILLMContext,
    *,
    assistant_expect_stripped_words: bool = True
) -> OpenAIContextAggregatorPair

Parameters

context
OpenAILLMContext
required

The context object containing conversation history and settings

assistant_expect_stripped_words
bool
default: "True"

Controls text preprocessing for assistant responses

​
Usage Example


# 1. Create the context
context = OpenAILLMContext(
    messages=[],
    system="You are a helpful assistant"
)

# 2. Create aggregators for message handling
aggregators = OpenAILLMService.create_context_aggregator(context)

# 3. Access individual aggregators
user_aggregator = aggregators.user()      # Handles user message formatting
assistant_aggregator = aggregators.assistant()  # Handles assistant responses

# 4. Use in a pipeline
pipeline = Pipeline([
    user_aggregator,
    llm_service,
    assistant_aggregator
])

The context management system ensures proper message formatting and history tracking throughout the conversation.

​
Methods

See the LLM base class methods for additional functionality.

​
Usage Example

from pipecat.services.together import TogetherLLMService
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext

# Configure service
service = TogetherLLMService(
    api_key="your-together-api-key",
    model="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
    params=TogetherLLMService.InputParams(
        temperature=0.7,
        max_tokens=1000
    )
)

# Create context
context = OpenAILLMContext(
    messages=[
        {"role": "system", "content": "You are a helpful assistant"},
        {"role": "user", "content": "Tell me about quantum computing"}
    ]
)

# Use in pipeline
pipeline = Pipeline([
    context_manager,  # Manages conversation context
    service,          # Processes LLM requests
    text_handler      # Handles responses
])

​
Function Calling

Supports OpenAI-compatible function calling:

# Define tools
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get weather information",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string"}
            }
        }
    }
}]

# Configure context with tools
context = OpenAILLMContext(
    messages=[],
    tools=tools
)

# Register function handler
@service.function("get_weather")
async def handle_weather(location: str):
    return {"temperature": 72, "condition": "sunny"}

​
Available Models

Together AI provides access to various models, including:

Model NameDescription
meta-llama/Meta-Llama-3.1-8B-Instruct-TurboLlama 3.1 8B instruct model optimized for speed
meta-llama/Meta-Llama-3.1-34B-InstructLlama 3.1 34B instruct model
meta-llama/Meta-Llama-3.1-70B-InstructLlama 3.1 70B instruct model
meta-llama/Meta-Llama-3.1-405B-InstructLlama 3.1 405B instruct model
meta-llama/Llama-3.2-3B-Instruct-TurboLlama 3.2 3B instruct model optimized for speed
meta-llama/Llama-3.2-11B-Vision-Instruct-TurboLlama 3.2 11B vision & instruct model
meta-llama/Llama-3.2-90B-Vision-Instruct-TurboLlama 3.2 90B vision & instruct model

​
Frame Flow

Inherits the OpenAI LLM Service frame flow:

​
Metrics Support

The service collects the same metrics as OpenAILLMService:

  • Token usage (prompt and completion)
  • Processing duration
  • Time to First Byte (TTFB)
  • Function call metrics

​
Notes

  • OpenAI-compatible interface
  • Supports streaming responses
  • Handles function calling
  • Manages conversation context
  • Includes token usage tracking
  • Thread-safe processing
  • Automatic error handling
  • Inherits OpenAI service features
OpenPipeAWS Polly