LLMWithBackend

LLMWithBackend is in alpha: its API is likely to change between releases.

Overview

LLMWithBackend pairs a frontend LLM, which keeps the conversation with the user fluent, with a backend LLM that handles work needing tools or careful reasoning. It goes where the LLM goes in your pipeline and runs the backend as an agent of its own. The frontend hands work to the backend with a delegate tool and keeps talking with the user, and what the backend has to say comes back into the frontend’s conversation for it to relay to the user. Composite services hold one or more services and stand in a pipeline where a single service would. LLMWithBackend holds the frontend service; the backend is a BackendLLMWorker. For a walkthrough, see LLM with a Backend.

Example

An engineering assistant with text and speech-to-speech frontends

BackendLLMWorker

The worker that runs the backend LLM

from pipecat.pipeline.llm_with_backend import (
    BackendConnector,
    ExplicitBackendRequestStrategy,
    LLMWithBackend,
    PredictiveDelegation,
    TranscriptBackendRequestStrategy,
)

LLMWithBackend is a Pipeline, not an LLMService: functions, settings updates, event handlers and a FlowManager go on its frontend. The backend’s tools go in the backend’s own context.

Configuration

LLMService

required

The conversational LLM service, text or speech-to-speech.

BackendLLMWorker | str

required

The backend worker, or the name of one registered elsewhere: in the app, or in another process on a shared bus. A worker given here is added to the pipeline worker when the pipeline starts.

BackendConnector | None

default:"None"

How delegation works. A default BackendConnector when None.

PredictiveDelegation | None

default:"None"

A PredictiveDelegation, to start the backend sooner on turns that need it. Text frontends only.

Raises ValueError when:

The frontend is InworldRealtimeLLMService, UltravoxRealtimeLLMService or OpenAILiveLLMService, which can’t be a frontend (see Notes).
The frontend is speech-to-speech and the request strategy is TranscriptBackendRequestStrategy, which reads the frontend’s context.
predictive_delegation is given with a speech-to-speech frontend, or with ExplicitBackendRequestStrategy, whose delegate call takes a request argument.

Properties

Property	Type	Description
`frontend`	`LLMService`	The frontend LLM service.
`backend`	`BackendLLMWorker \| str`	The backend worker, or the name of one running elsewhere, such as in another process.
`connector`	`BackendConnector`	The connector joining the two.
`predictive_delegation`	`PredictiveDelegation \| None`	The predictive delegation that starts the backend sooner, or `None`.
`tool`	`FunctionSchema`	The `delegate` tool installed on the frontend.

BackendConnector

BackendConnector manages the interaction between the frontend and the backend. It defines the delegate tool, holds the session with the backend for the pipeline’s life, and turns what the backend produces into messages in the frontend’s conversation.

BackendRequestStrategy | None

default:"None"

How a delegate call becomes the backend’s request. When None, TranscriptBackendRequestStrategy for a text frontend and ExplicitBackendRequestStrategy for a speech-to-speech one.

str

default:"FRONTEND_PAIRING_INSTRUCTION"

How the frontend’s model works with a backend, appended to its system instruction. A custom override has to cover everything the default does: when and how to delegate, that to the user there is one assistant, what to do after delegating, and what to do with the backend’s messages. The request strategy’s guidance goes where PAIRING_INSTRUCTION_STRATEGY_PLACEHOLDER stands, or after the text when it has none.

bool

default:"True"

Whether the frontend replies as soon as a delegation is sent, which is where its acknowledgement comes from. Set False to delegate without a reply.

float | None

default:"30"

How long the backend may take to acknowledge a request, including the wait for it to become ready.

bool

default:"False"

Whether to send every exchange with the backend to the client as an RTVI server-message, for debugging. See Frames.

connector = BackendConnector(respond_on_delegate=False)

Request Strategies

A request strategy defines the delegate tool’s parameters and description, and builds the request the backend receives from each delegate call.

TranscriptBackendRequestStrategy

The frontend’s model calls delegate with no arguments, and the request is the transcript of the user and assistant turns since the previous delegation (the whole conversation the first time). The default for a text frontend. Its delegate call needs no arguments, so it works with predictive delegation. It reads the frontend’s context, so a speech-to-speech frontend refuses it.

str | None

default:"TRANSCRIPT_FRONTEND_INSTRUCTION"

The strategy’s guidance to the frontend’s model, placed in the pairing instruction.

str

default:"TRANSCRIPT_TOOL_DESCRIPTION"

How the delegate tool is described.

str

default:"TRANSCRIPT_REQUEST_INSTRUCTION"

What the backend should do with the transcript, placed after it.

ExplicitBackendRequestStrategy

The frontend’s model words the request itself, in the delegate tool’s request argument. The default for a speech-to-speech frontend, which replies to the user’s audio before that audio’s transcript reaches its context.

str | None

default:"EXPLICIT_FRONTEND_INSTRUCTION"

The strategy’s guidance to the frontend’s model, placed in the pairing instruction.

str

default:"EXPLICIT_TOOL_DESCRIPTION"

How the delegate tool is described.

str

default:"EXPLICIT_REQUEST_INSTRUCTION"

What the backend should do with the request, placed after it.

PredictiveDelegation

PredictiveDelegation is in alpha: its API is likely to change between releases.

PredictiveDelegation starts the backend sooner on turns that need it. How it works: on each user turn, a yes/no classifier races the frontend’s inference, asking whether the turn needs the backend. A “yes” that arrives before the frontend’s first output delegates the turn at once and cancels the inference. A “no”, a late answer, or a classifier error leaves the turn to the frontend as usual. The frontend is never slowed down waiting for the classifier. The classifier reads what the frontend’s model reads: its system instruction, pairing instruction included, the delegate tool’s description, and the latest messages of the conversation.

BaseClassifier

required

The yes/no classifier to ask, such as JevClassifier.

from pipecat.classifiers.typesafe.jev.classifier import JevClassifier

llm = LLMWithBackend(
    frontend=frontend,
    backend=backend,
    predictive_delegation=PredictiveDelegation(
        JevClassifier(api_key=os.getenv("TYPESAFE_API_KEY"))
    ),
)

It needs a text frontend and a request strategy whose delegate call takes no arguments, such as the default TranscriptBackendRequestStrategy. It leaves speculative turns (eager end of turn) to the frontend, and every turn while the frontend filters incomplete user turns. The classifier’s metrics are pushed as MetricsFrames.

Constants

The constants are in pipecat.pipeline.llm_with_backend.prompts, except the request instructions, which are in pipecat.workers.llm.backend_llm_worker, and DELEGATE_TOOL_NAME, which is in pipecat.pipeline.llm_with_backend.connector. Read the source for the default texts’ wording.

Constant	What it is
`FRONTEND_PAIRING_INSTRUCTION`	The default `frontend_pairing_instruction`.
`PAIRING_INSTRUCTION_STRATEGY_PLACEHOLDER`	`"{strategy}"`: where a pairing instruction takes the request strategy’s guidance.
`BACKEND_MESSAGE_RELAY_INSTRUCTION`	Appended to each `Backend:` message for the user: relay it now, once, even if the conversation has moved on.
`TRANSCRIPT_FRONTEND_INSTRUCTION`	`TranscriptBackendRequestStrategy`’s default `frontend_instruction`.
`TRANSCRIPT_TOOL_DESCRIPTION`	`TranscriptBackendRequestStrategy`’s default `tool_description`.
`TRANSCRIPT_REQUEST_INSTRUCTION`	`TranscriptBackendRequestStrategy`’s default `request_instruction`.
`EXPLICIT_FRONTEND_INSTRUCTION`	`ExplicitBackendRequestStrategy`’s default `frontend_instruction`.
`EXPLICIT_TOOL_DESCRIPTION`	`ExplicitBackendRequestStrategy`’s default `tool_description`.
`EXPLICIT_REQUEST_INSTRUCTION`	`ExplicitBackendRequestStrategy`’s default `request_instruction`.
`DELEGATE_TOOL_NAME`	`"delegate"`.

Frames

Pushed by

Frame	Fields	Effect
`ExternalFunctionCall*Frame`	function name, call ID, arguments, result	One per phase of each function call the backend makes. `RTVIObserver` reports them to the client as function-call events.
`RTVIServerMessageFrame`	`data`	With `client_trace=True`, one per exchange with the backend. `data["type"]` is `"llm-with-backend"`, and `data["event"]` is `"request"`, `"output"`, `"tool_call"`, `"error"` or `"idle"`.

Notes

Backend messages in the frontend’s context: What the backend produces is appended to the frontend’s context as developer messages. A message for the user starts with Backend: and ends with BACKEND_MESSAGE_RELAY_INSTRUCTION, and the rest are silent: Backend (note): for text not meant for the user, Backend (working): for a function call the backend is making, and Backend (thinking): for a reasoning summary.
The delegate tool: The frontend gets delegate as a built-in tool: it is sent with every inference beside the frontend’s own tools and never enters the context’s tool list.
Services that can’t be a frontend:
- InworldRealtimeLLMService: its model calls delegate again in place of taking in the backend’s messages.
- UltravoxRealtimeLLMService: built-in tools such as delegate don’t reach its session.
- OpenAILiveLLMService: it is already a frontend with a backend of its own, set up with its delegation modes. It can’t be a backend either.

Get Started

Standards

Pipecat Server

Client SDKs

Pipecat Flows

Pipecat Cloud

CLI

Pipecat Context Hub

LLMWithBackend

Overview

Example

BackendLLMWorker

LLMWithBackend

Configuration

Properties

BackendConnector

Request Strategies

TranscriptBackendRequestStrategy

ExplicitBackendRequestStrategy

PredictiveDelegation

Constants

Frames

Pushed by

Notes

​Overview

Example

BackendLLMWorker

​LLMWithBackend

​Configuration

​Properties

​BackendConnector

​Request Strategies

​TranscriptBackendRequestStrategy

​ExplicitBackendRequestStrategy

​PredictiveDelegation

​Constants

​Frames

​Pushed by

​Notes

Overview

LLMWithBackend

Configuration

Properties

BackendConnector

Request Strategies

TranscriptBackendRequestStrategy

ExplicitBackendRequestStrategy

PredictiveDelegation

Constants

Frames

Pushed by

Notes