LLM with a Backend

LLMWithBackend is in alpha: its API is likely to change between releases.

LLMWithBackend pairs a frontend LLM, which keeps the conversation with the user fluent, with a backend LLM that handles work needing tools or careful reasoning. It goes where the LLM goes in your pipeline and runs the backend as an agent of its own. The frontend hands work to the backend with a delegate tool and keeps talking with the user, and what the backend has to say comes back into the frontend’s conversation for it to relay to the user. It’s a good fit for a voice bot whose work may involve nontrivial tool calls, several tool calls, or real reasoning: the user keeps talking with the frontend while the backend works.

Basic Usage

Define the backend’s tools as you would for any LLM:

from pipecat.services.llm_service import FunctionCallParams


async def run_tests(params: FunctionCallParams, path: str | None = None):
    """Run the test suite, or one test file.

    Args:
        path: A test file to run alone; the whole suite when omitted.
    """
    result = await test_runner.run(path)
    await params.result_callback(result)

Give the frontend a service of its own and the backend a BackendLLMWorker holding the backend’s LLM and tools:

import os

from pipecat.pipeline.llm_with_backend import LLMWithBackend
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.services.anthropic.llm import AnthropicLLMService
from pipecat.services.openai.responses.llm import OpenAIResponsesLLMService
from pipecat.workers.llm import BackendLLMWorker

llm = LLMWithBackend(
    # By default, a frontend runs like any LLM in a pipeline: services that set a
    # reasoning default, as this one does, run a reasoning model at its lowest
    # reasoning effort, for fast replies.
    frontend=OpenAIResponsesLLMService(
        api_key=os.getenv("OPENAI_API_KEY"),
        settings=OpenAIResponsesLLMService.Settings(system_instruction=FRONTEND_INSTRUCTIONS),
    ),
    # By default, a backend runs its model at the provider's default reasoning
    # effort, since it does the reasoning the frontend hands off.
    backend=BackendLLMWorker(
        llm=AnthropicLLMService(
            api_key=os.getenv("ANTHROPIC_API_KEY"),
            settings=AnthropicLLMService.Settings(
                model="claude-opus-5-5",
                system_instruction=BACKEND_INSTRUCTIONS,
            ),
        ),
        context=LLMContext(tools=[search_codebase, read_file, apply_patch, run_tests]),
    ),
)

Put llm where the LLM goes in the pipeline:

pipeline = Pipeline([
    transport.input(),
    stt,
    aggregators.user(),
    llm,
    tts,
    transport.output(),
    aggregators.assistant(),
])

The frontend needs no tools of its own: LLMWithBackend gives it the delegate tool. LLMWithBackend is a pipeline, not an LLM service, so functions, settings updates and event handlers go on llm.frontend. The backend is added to your pipeline worker when the pipeline starts, so there is nothing else to register.

Writing the Prompts

Each model gets a system instruction from you. The frontend’s says who the assistant is and, in plain words, what work it can take on:

FRONTEND_INSTRUCTIONS = """You are Pip, the voice assistant of the Acme engineering team.
Your responses are spoken aloud, so keep them to one or two natural sentences. You can
make code changes and run the tests, research the team's internal docs, and check CI
and the open pull requests; that work takes a moment."""

The backend’s says what it does with its tools and what the user should be told:

BACKEND_INSTRUCTIONS = """You do the engineering work for the Acme team's assistant.
When asked for a code change, keep going until the tests pass, then tell the user what
you changed and the test result. Keep what you tell them short and concrete."""

Pipecat appends a pairing instruction to each, the text that tells a model how it works with the other: the frontend’s says when and how to delegate and what to do with the backend’s messages, and the backend’s says how to take requests and how to mark what the user should hear. So neither of your prompts mentions the delegate tool, the other model, or how the backend’s messages reach the user.

How a Conversation Works

When the user asks for something that needs the backend, the frontend calls delegate, which returns at once, and the frontend acknowledges: “Sure, I’ll get on that now.” The backend starts working. The user can keep talking. The frontend handles small talk and questions it can answer itself. When the user asks for more work, corrects the work in progress, or says “never mind”, the frontend delegates that too, and the backend works out what it means for its work and cancels any tool calls whose results are no longer wanted. When the backend has something the user really needs to hear, such as a result or a question it can’t go on without, the frontend relays it in its own words. Everything else the backend outputs (progress notes, function calls, reasoning) reaches the frontend silently, so when the user asks how it’s going, the frontend can answer without delegating again. To the user there is one assistant: the frontend never mentions the backend, and talks about the work as its own.

Speech-to-Speech Frontends

The frontend can be a speech-to-speech service, such as OpenAIRealtimeLLMService or GeminiLiveLLMService, in place of the STT, LLM and TTS services. Its model then words each request to the backend itself. Two can’t be a frontend: InworldRealtimeLLMService, because its model calls delegate again in place of taking in the backend’s messages, and UltravoxRealtimeLLMService, because built-in tools such as delegate don’t reach its session. OpenAILiveLLMService has the same shape built in: OpenAI’s GPT-Live model is the frontend, and in client delegation its backend is a BackendLLMWorker.

Predictive Delegation

PredictiveDelegation is in alpha: its API is likely to change between releases.

A text frontend spends a whole inference deciding to call delegate before the backend can start. Predictive delegation starts the backend sooner on turns that need it. How it works: on each user turn, a fast yes/no classifier races the frontend’s inference, asking whether the turn needs the backend. A “yes” that arrives before the frontend’s first output delegates the turn at once and cancels the inference. A “no”, a late answer, or a classifier error leaves the turn to the frontend as usual. The frontend is never slowed down waiting for the classifier.

from pipecat.classifiers.typesafe.jev.classifier import JevClassifier
from pipecat.pipeline.llm_with_backend import PredictiveDelegation

llm = LLMWithBackend(
    frontend=frontend,
    backend=backend,
    predictive_delegation=PredictiveDelegation(
        JevClassifier(api_key=os.getenv("TYPESAFE_API_KEY"))
    ),
)

The cost: a classification on every user turn, and a cancelled frontend request each time it delegates early. Predictive delegation needs a text frontend and a request strategy whose delegate call takes no arguments, such as the default TranscriptBackendRequestStrategy.

Showing Backend Work in a Client

The function calls the backend makes reach clients as RTVI function-call events, the same as calls the frontend makes, so a UI can show what the backend is doing. They are reported at the level function_call_report_level sets for them, as any function call is; raise it to show their names, arguments or results. To keep the frontend’s delegate calls from reaching the client, mark them DISABLED on your pipeline worker:

from pipecat.processors.frameworks.rtvi import RTVIFunctionCallReportLevel, RTVIObserverParams

worker = PipelineWorker(
    pipeline,
    rtvi_observer_params=RTVIObserverParams(
        function_call_report_level={
            # The frontend's delegate calls aren't reported; every other call is
            # reported at the usual level
            "delegate": RTVIFunctionCallReportLevel.DISABLED,
        },
    ),
)

While developing, turn on client_trace to send every exchange with the backend to the client:

from pipecat.pipeline.llm_with_backend import BackendConnector

llm = LLMWithBackend(
    frontend=frontend,
    backend=backend,
    connector=BackendConnector(client_trace=True),
)

Each exchange arrives as an RTVI server-message whose data.type is "llm-with-backend", which a client’s event log shows.

Running the Backend in Another Process

The frontend reaches the backend by name over the bus, so the backend can run in another process. Name the backend worker and run it under its own WorkerRunner on a network bus, set up as in Distributed Agents:

backend = BackendLLMWorker(name="engineering-backend", llm=backend_llm, context=backend_context)

runner = WorkerRunner(bus=bus)
await runner.add_workers(backend)
await runner.run()

In the frontend’s process, on the same bus, pass the name in place of the worker:

llm = LLMWithBackend(frontend=frontend_llm, backend="engineering-backend")

The frontend waits for the backend to be ready, so a backend that starts a little after the frontend is fine.

Adjusting the Defaults

BackendConnector(respond_on_delegate=False) makes the frontend delegate without replying, for an app where a quick reply on delegating isn’t wanted:

from pipecat.pipeline.llm_with_backend import BackendConnector

llm = LLMWithBackend(
    frontend=frontend,
    backend=backend,
    connector=BackendConnector(respond_on_delegate=False),
)

The pairing instructions and the way requests are worded are tuned to work across providers. The reference lists the parameters for custom overrides, and a custom override has to cover everything the default does. Say what is specific to your bot in your own prompts instead.

Other Multi-Agent Setups

LLMWithBackend sets up one multi-agent shape: a frontend, a backend, and the messages between them handled for you. When your bot needs a different shape, build it from the multi-agent pieces directly:

Multiple LLM Agents: agents that each own an LLM, tools and context
Agent Handoff: moving the conversation from one agent to another
Job Coordination: dispatching work to agents and collecting results
Distributed Agents: agents across processes and machines

What’s Next

For every parameter of LLMWithBackend, its connector and predictive delegation, see the reference:

LLMWithBackend Reference

Configuration, request strategies, predictive delegation, and frames

What's Next

Explore Fundamentals and advanced patterns

Get Started

Migration

Learning Pipecat

Fundamentals

Flows

Evals

Features

Telephony

Deployment

Examples & Recipes

LLM with a Backend

Basic Usage

Writing the Prompts

How a Conversation Works

Speech-to-Speech Frontends

Predictive Delegation

Showing Backend Work in a Client

Running the Backend in Another Process

Adjusting the Defaults

Other Multi-Agent Setups

What’s Next

LLMWithBackend Reference

What's Next

​Basic Usage

​Writing the Prompts

​How a Conversation Works

​Speech-to-Speech Frontends

​Predictive Delegation

​Showing Backend Work in a Client

​Running the Backend in Another Process

​Adjusting the Defaults

​Other Multi-Agent Setups

​What’s Next

LLMWithBackend Reference

What's Next

Basic Usage

Writing the Prompts

How a Conversation Works

Speech-to-Speech Frontends

Predictive Delegation

Showing Backend Work in a Client

Running the Backend in Another Process

Adjusting the Defaults

Other Multi-Agent Setups

What’s Next