# Pipecat > Pipecat is an open source ecosystem for building voice and multimodal AI > agents. Build with the Python framework, connect users with the client SDKs, > structure conversations with Pipecat Flows, and deploy to production on > Pipecat Cloud. ## About Pipecat Pipecat is an open source framework and hosted platform for real-time voice and multimodal AI agents. Its primary components: - **Pipecat framework**: An open source Python framework that orchestrates AI services (STT, LLM, TTS, and more) into real-time pipelines. Start with the [quickstart](https://docs.pipecat.ai/pipecat/get-started/quickstart.md). - **Pipecat client SDKs**: JavaScript, React, React Native, iOS, Android, and C++ SDKs that connect users to agents over WebRTC or WebSockets. See the [client introduction](https://docs.pipecat.ai/client/introduction.md). - **Pipecat Flows**: A framework for structured conversations - define conversation paths as nodes with functions and actions. See the [Flows introduction](https://docs.pipecat.ai/pipecat-flows/introduction.md). - **Pipecat Cloud**: Managed infrastructure for deploying and scaling agents, run by the Pipecat team. See the [Cloud introduction](https://docs.pipecat.ai/pipecat-cloud/introduction.md). - **Pipecat CLI**: Scaffold projects, run evals, and deploy from the terminal. See the [CLI overview](https://docs.pipecat.ai/api-reference/cli/overview.md). Server API reference lives under `api-reference/server/`: services (STT, TTS, LLM, transports, serializers), pipelines, frames, workers, and utilities. # Pipecat Documentation Source: https://docs.pipecat.ai/overview/introduction.md Documentation for the Pipecat ecosystem: the open source framework, client SDKs, Pipecat Flows, and Pipecat Cloud hosting. Pipecat is an open source ecosystem for building voice and multimodal AI agents. It provides everything you need to create, deploy, and scale real-time AI applications that can see, hear, and speak. The framework is free to use under the BSD-2 license, and works with any AI provider and any hosting environment — you pay only for the services you choose. ## The Pipecat Ecosystem Open source Python framework for building voice and multimodal AI pipelines. Orchestrate 100+ AI services with ultra-low latency. Client SDKs for JavaScript, React, React Native, iOS, Android, and C++. Connect users to your agents via web and mobile. Build structured conversations with defined paths and state management. Break complex tasks into focused steps for better LLM accuracy. Managed hosting platform for deploying and scaling Pipecat agents in production with built-in infrastructure. ## How It All Fits Together A typical Pipecat application has a **client** and a **server**. The client connects users via browser, mobile app, or phone. The server runs a Pipecat pipeline that processes audio, runs LLMs, and generates speech in real-time. Your hosting provider — Pipecat Cloud or self-hosted — manages deployment and scales instances to handle concurrent sessions. ![Pipecat architecture](/images/pipecat-architecture.png) ## Getting Started Follow the [Quickstart](/pipecat/get-started/quickstart) to create a voice AI bot in 5 minutes. Work through the [Learning Pipecat](/pipecat/learn/overview) guide to understand pipelines, processors, and transports. Connect users to your agent with a [Client SDK](/client/introduction) for web or mobile. Ship to production with [Pipecat Cloud](/pipecat-cloud/introduction) or [self-host](/pipecat/deployment/overview) on your own infrastructure. ## Community Connect with other developers, share projects, and get support. Explore the source code, open issues, and contribute. ## Enterprise Support Pipecat is built and maintained by Daily. If your team is taking voice agents to production, we're here to help. [Learn more about enterprise support](/enterprise-support) # Pipecat Open Source Framework Source: https://docs.pipecat.ai/overview/pipecat.md Pipecat is an open source Python framework for voice and multimodal AI agents, orchestrating AI services and transports. Pipecat is an open source Python framework for building voice and multimodal AI agents. It orchestrates AI services, network transports, and audio processing to enable ultra-low latency conversations that feel natural and responsive. Want to dive right in? Build and run your first Pipecat application ## What You Can Build Natural, real-time conversations with AI using speech recognition and synthesis Connect to your agent via phone for support, intake, and customer service interactions Applications that combine voice, video, images, and text for rich interactions Storytelling experiences and social companions that engage users Voice-controlled games and interactive experiences with real-time AI responses Build structured conversations with Pipecat Flows to complete tasks and improve LLM accuracy ## How It Works Pipecat orchestrates AI services in a **pipeline**, which is a series of processors that handle real-time audio, text, and video frames with ultra-low latency. Here's what happens in a typical voice conversation: 1. **Transport** receives audio from the user (browser, phone, etc.) 2. **Speech Recognition** converts speech to text in real-time 3. **LLM** generates intelligent responses based on context 4. **Speech Synthesis** converts responses back to natural speech 5. **Transport** streams audio back to the user In most cases, the entire round-trip interaction happens between 500-800ms, creating a natural conversation experience for the user. Pipecat Overview ## A Multi-Agent System A typical bot is a single agent, but Pipecat is a multi-agent system: you can coordinate many agents that talk to each other over a shared message **bus**. The same `WorkerRunner` you use for one bot coordinates many agents, so any Pipecat app is multi-agent ready. - The **`WorkerRunner`** owns the bus and manages every agent's lifecycle and discovery. - Each agent is a **worker**. `BaseWorker` is the foundation and coordinates purely over the bus with no pipeline of its own; `PipelineWorker` and `LLMWorker` add a Pipecat pipeline. - Any agent can **start other agents** at runtime: a worker adds child workers (or the runner does), and they immediately join the shared bus. - Agents coordinate in more than one way. **Handoff** transfers control of the conversation to another agent, while **jobs** and **job groups** dispatch work to one or many agents in parallel and collect their results. Agents can run together in one process over an in-process bus, or across separate processes and machines over a Redis or Postgres bus, without changing your agent code. Learn how agents, the runner, and the bus work together ## Ready to Build? Build and run your first Pipecat application Learn about pipelines, processors, transports, and context management Browse the complete list of 100+ AI service integrations Deploy to Pipecat Cloud or self-host on your own infrastructure # Pipecat Client SDKs Source: https://docs.pipecat.ai/overview/clients.md The Pipecat client SDK family connects users to your agents from web and mobile apps, handling the real-time media layer. Pipecat Clients are a family of SDKs that connect users to your Pipecat agents through web and mobile applications. They handle real-time audio/video transport, session management, and provide messaging and events for building responsive voice AI interfaces. ## Supported Platforms Web applications with vanilla JavaScript React applications with hooks and components Cross-platform mobile apps Native iOS applications Native Android applications High-performance native applications ## What the SDKs Provide All Pipecat Client SDKs include: - **Transport management** — WebRTC and WebSocket connections to your Pipecat pipeline - **Media handling** — Microphone, camera, and speaker device management - **Session lifecycle** — Connect, disconnect, reconnect, and error handling - **Messaging** — Send custom messages to your bot and receive responses - **Events** — Callbacks for bot state changes, transcriptions, and more ## Next Steps Introduction to client-side development with Pipecat. Pre-built React components for voice AI interfaces. # Pipecat Flows Source: https://docs.pipecat.ai/overview/flows.md Pipecat Flows adds structured conversations to your voice agents: define conversation paths as nodes with functions and actions. Pipecat Flows is a framework for building structured conversations in your AI applications. It lets you define conversation paths as a graph of nodes, where each node focuses the LLM on a single task with only the tools it needs. This approach solves a common problem: monolithic prompts with many tools lead to hallucinations and lower accuracy. Pipecat Flows breaks complex tasks into focused steps with clear, specific instructions. ## When to Use Pipecat Flows Pipecat Flows is best suited for use cases where: - **You need precise control** over how a conversation progresses through specific steps - **Your bot handles complex tasks** that can be broken down into smaller, manageable pieces - **You want to improve LLM accuracy** by focusing the model on one specific task at a time instead of managing multiple responsibilities simultaneously ## How Pipecat Flows Builds on the Pipeline A Pipecat **pipeline** provides your bot's core mechanics — receiving audio, transcribing input, running LLM completions, converting responses to audio, and sending audio back to the user. **Pipecat Flows** builds on that pipeline to structure the conversation, managing context and tools as it moves from one state to the next. This keeps your conversation logic cleanly separated from the pipeline mechanics. ## Ready to Build? Build your first conversation flow in minutes Complete reference docs and technical details Explore real-world examples and use cases Source code, issues, and contributions # Pipecat Cloud Source: https://docs.pipecat.ai/overview/cloud.md Pipecat Cloud is the managed hosting platform for deploying and scaling Pipecat agents, from the team behind the framework. [Pipecat Cloud](https://pipecat.daily.co) is a managed platform for deploying and scaling Pipecat agents in production. It handles infrastructure, scaling, and operations so you can focus on building your agent. ## Key Capabilities - **One-command deploy**: Package and deploy agents with `pipecat cloud deploy` - **Auto-scaling**: Scale from zero to thousands of concurrent sessions - **Built-in WebRTC**: Daily WebRTC transport included, no separate infrastructure needed - **Secrets management**: Securely store and inject API keys and credentials - **Session management**: Start, stop, and monitor agent sessions via REST API or SDK - **Logging & monitoring**: Built-in logging with Datadog integration support - **Global regions**: Deploy close to your users for lowest latency ## How It Works Write a Pipecat pipeline as you normally would, using any supported services. Use the CLI to build and deploy your agent image to Pipecat Cloud. ```bash pipecat cloud deploy ``` Use the REST API or Python SDK to start agent sessions on demand. ```bash curl --request POST \ --url https://api.pipecat.daily.co/v1/public/{agentName}/start \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ ``` ## Next Steps Set up your account and deploy your first agent. Learn about accounts, agent images, secrets, and scaling. # Enterprise Support Source: https://docs.pipecat.ai/enterprise-support/index.md Enterprise support for Pipecat and Pipecat Cloud from Daily: architecture guidance, production readiness, and compliance. Pipecat is open source and free to use under the BSD-2 license. For teams running voice agents in production, [Daily](https://www.daily.co) — the team behind Pipecat and [Pipecat Cloud](/pipecat-cloud/introduction) — offers enterprise support. ## What enterprise support covers - **Architecture guidance** — designing pipelines, reviewing benchmarks, choosing services, and tuning latency for your use case. - **Production readiness** — [capacity planning](/pipecat-cloud/guides/capacity-planning), observability, and evals. - **Enterprise Pipecat Cloud** — on premises and VPC deployment of Pipecat Cloud, data residency, and compliance requirements. - **Forward Deployed Engineering** — work directly with senior Pipecat engineers in FDE engagements. ## Get in touch Talk to Daily about enterprise support, limits, and pricing. Reach Daily directly at help@daily.co. Not an enterprise customer? Community support is available in the [Pipecat Discord](https://discord.gg/pipecat), and issues can be filed on [GitHub](https://github.com/pipecat-ai/pipecat/issues). # Introduction to Pipecat Source: https://docs.pipecat.ai/pipecat/get-started/introduction.md Start here with the Pipecat framework: what it is, how pipelines orchestrate AI services, and where to go next. Pipecat is an open source Python framework for building voice and multimodal AI agents. It orchestrates AI services, network transports, and audio processing to enable ultra-low latency conversations that feel natural and responsive. The framework is free to use under the BSD-2 license, and you choose your own stack: any speech-to-text, LLM, and text-to-speech provider, hosted on your own infrastructure or [Pipecat Cloud](https://www.daily.co/pricing/pipecat-cloud) — you pay only for the providers you choose. Want to dive right in? Build and run your first Pipecat application ## What You Can Build Natural, real-time conversations with AI using speech recognition and synthesis Connect to your agent via phone for support, intake, and customer service interactions Applications that combine voice, video, images, and text for rich interactions Storytelling experiences and social companions that engage users Voice-controlled games and interactive experiences with real-time AI responses Build structured conversations with Pipecat Flows to complete tasks and improve LLM accuracy ## How It Works Pipecat orchestrates AI services in a **pipeline**, which is a series of processors that handle real-time audio, text, and video frames with ultra-low latency. Here's what happens in a typical voice conversation: 1. **Transport** receives audio from the user (browser, phone, etc.) 2. **Speech Recognition** converts speech to text in real-time 3. **LLM** generates intelligent responses based on context 4. **Speech Synthesis** converts responses back to natural speech 5. **Transport** streams audio back to the user In most cases, the entire round-trip interaction happens between 500-800ms, creating a natural conversation experience for the user. Pipecat Overview ## A Multi-Agent System The bot above is a single agent. Pipecat is also a multi-agent system: the same `WorkerRunner` can coordinate many agents that communicate over a shared message **bus**. Most agents run their own pipeline (`PipelineWorker`, `LLMWorker`), while a coordinator can be a plain `BaseWorker` with no pipeline at all. A normal bot is just the one-agent case, so every Pipecat app is multi-agent ready. Specialized agents that transfer control seamlessly during a conversation. Dispatch work to multiple agents in parallel and collect their results. Run agents across separate processes or machines on a shared bus. ## Installation Pipecat requires Python 3.11 or later and is published on PyPI as [`pipecat-ai`](https://pypi.org/project/pipecat-ai/). Install the base package with optional **extras** for each transport, AI service, and utility you need: ```bash uv add "pipecat-ai[daily,deepgram,openai,cartesia,silero]" ``` Each [service page](/api-reference/server/services/supported-services) lists the exact extra to install. To upgrade to the latest version: ```bash uv lock --upgrade-package pipecat-ai uv sync ``` ## Ready to Build? Build and run your first Pipecat application Learn about pipelines, processors, transports, and context management Browse the complete list of 100+ AI service integrations Deploy to Pipecat Cloud or self-host on your own infrastructure # Pipecat Quickstart Source: https://docs.pipecat.ai/pipecat/get-started/quickstart.md Build and run your first Pipecat voice AI bot in under 5 minutes: a simple conversational agent you can talk to in a browser. This quickstart guide will help you build and deploy your first Pipecat voice AI bot. You'll create a simple conversational agent that you can talk to in real-time, then deploy it to production on Pipecat Cloud. **Two steps**: Local Development (5 min) → Production Deployment (5 min) ## Step 1: Local Development ### Prerequisites **Environment** - Python 3.11 or later - [uv](https://docs.astral.sh/uv/getting-started/installation/) package manager installed Adding Pipecat to an existing project instead of scaffolding a new one? See [Installation](/pipecat/get-started/introduction#installation) for manual setup. **AI Service API Keys** This quickstart uses three AI services working together in a pipeline. You'll need API keys from each service: Create an account and generate your API key for real-time speech recognition. Create an account and generate an API key for intelligent conversation responses. Sign up and generate your API key for natural voice synthesis. Have these API keys ready. You'll add them to your environment file in the next section. ### Setup 1. Install the Pipecat CLI and scaffold the quickstart project ```bash # Install the Pipecat CLI uv tool install "pipecat-ai[cli]" # Scaffold the quickstart project (also writes AGENTS.md + CLAUDE.md) pipecat init quickstart # Change to the project directory cd pipecat-quickstart ``` Planning to build with a coding agent? Co-install the [Pipecat Context Hub](/api-reference/context-hub) — `uv tool install "pipecat-ai[cli]" --with pipecat-ai-context-hub` — so your agent queries current Pipecat APIs instead of its training data. It's a larger download, so skip it if you just want the quickstart running. 2. Configure your API keys Create your environment file: ```bash cp .env.example .env ``` Open the `.env` file in your text editor and add your API keys: ```ini DEEPGRAM_API_KEY=your_deepgram_api_key OPENAI_API_KEY=your_openai_api_key CARTESIA_API_KEY=your_cartesia_api_key ``` 3. Set up virtual environment and install dependencies ```bash uv sync ``` ### Run your bot locally Now you're ready to run your bot! Start it with ```bash uv run bot.py ``` You should see output similar to this: ``` 🚀 WebRTC server starting at http://localhost:7860/client Open this URL in your browser to connect! ``` Open http://localhost:7860/client in your browser and click **Connect** to start talking to your bot. **First run note**: The initial startup may take ~20 seconds as Pipecat downloads required models and imports. Subsequent runs will be much faster. 🎉 **Success!** Your bot is running locally. Now let's deploy it to production so others can use it. --- ## Step 2: Deploy to Production Transform your local bot into a production-ready service. Pipecat Cloud handles scaling, monitoring, and global deployment. ### Prerequisites 1. Sign up for Pipecat Cloud [Create your Pipecat Cloud account](https://pipecat.daily.co/sign-up) to deploy and manage your bots. 2. Pipecat CLI Log in to Pipecat Cloud using the CLI to get started: ```bash pipecat cloud auth login ``` Select `Allow` to authenticate your CLI session in the browser page that opens. ### Your deployment configuration The `pcc-deploy.toml` file tells Pipecat Cloud how to build and deploy your bot: ```toml agent_name = "pipecat-quickstart" secret_set = "pipecat-quickstart-secrets" [scaling] min_agents = 1 ``` **Understanding the configuration:** - `agent_name`: Your bot's name in Pipecat Cloud - `secret_set`: Where your API keys are stored securely - `min_agents`: Number of bot instances to keep ready (1 = instant start) ### Configure secrets Upload your API keys to Pipecat Cloud's secure storage: ```bash pipecat cloud secrets set pipecat-quickstart-secrets --file .env ``` This creates a secret set called `pipecat-quickstart-secrets` (matching your TOML file) and uploads all your API keys from `.env`. ### Build and Deploy Build and deploy your bot to Pipecat Cloud: ```bash pipecat cloud deploy ``` The CLI automatically builds your image using the pcc-deploy.toml file and Dockerfile in your project and deploys it to Pipecat Cloud — no container registry needed. Want to use your own container registry? See the [container registries guide](/pipecat-cloud/guides/container-registries/overview) for advanced deployment options. ### Connect to your agent 1. Open your [Pipecat Cloud dashboard](https://pipecat.daily.co/) 2. Select your `pipecat-quickstart` agent → **Sandbox** 3. Allow microphone access and click **Connect** 🎉 **Your bot is now live in production!** Explore advanced Pipecat Cloud features like scaling, monitoring, secrets management, and production best practices. --- ## Understanding the Quickstart Bot Let's walk through the generated `bot.py` to understand what's happening. When you speak to your bot, here's the real-time pipeline that processes your conversation: 1. **Audio Capture**: Your browser captures microphone audio and sends it via WebRTC 2. **Voice Activity Detection**: Silero VAD detects when you start and stop speaking 3. **Speech Recognition**: Deepgram converts your speech to text in real-time 4. **Language Processing**: OpenAI's GPT model generates an intelligent response 5. **Speech Synthesis**: Cartesia converts the response text back to natural speech 6. **Audio Playback**: The generated audio streams back to your browser Each step happens with minimal latency, typically completing the full round-trip in under one second. ### AI Services Your bot uses three AI services, each configured with API keys from your `.env` file: ```python # Create AI Services stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY")) tts = CartesiaTTSService( api_key=os.getenv("CARTESIA_API_KEY"), settings=CartesiaTTSService.Settings( voice=os.getenv("CARTESIA_VOICE_ID", "71a7ad14-091c-4e8e-a314-022ece01c121"), ), ) llm = OpenAIResponsesLLMService( api_key=os.getenv("OPENAI_API_KEY"), settings=OpenAIResponsesLLMService.Settings( model=os.getenv("OPENAI_MODEL", "gpt-4.1"), system_instruction="You are a helpful assistant in a voice conversation. Your responses will be spoken aloud, so avoid emojis, bullet points, or other formatting that can't be spoken. Respond to what the user said in a creative, helpful, and brief way.", ), ) ``` Pipecat supports many different AI services. You can swap out Deepgram for Soniox, OpenAI for Anthropic, or Cartesia for ElevenLabs without changing the rest of your code. See the [supported services documentation](/api-reference/server/services/supported-services) for all available options. ### Context and Messages Your bot maintains conversation history using a context object, enabling multi-turn interactions where the bot remembers what was said earlier. The context is initialized and used to store the conversation history by the context aggregators: ```python context = LLMContext() user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, user_params=LLMUserAggregatorParams( vad_analyzer=SileroVADAnalyzer(), ), ) ``` ### RTVI Protocol When building web or mobile clients, you can use [Pipecat's client SDKs](/client/introduction) that communicate with your bot via the [RTVI (Real-Time Voice Interaction) protocol](/client/rtvi-standard). Pipecat comes with RTVI enabled by default, allowing your the Pipecat server and client to exchange messages and events in real-time. ### Pipeline Configuration The core of your bot is a Pipeline that processes data through a series of processors: ```python # Create the pipeline with the processors pipeline = Pipeline([ transport.input(), # Receive audio from browser stt, # Speech-to-text (Deepgram) user_aggregator, # Add user message to context llm, # Language model (OpenAI) tts, # Text-to-speech (Cartesia) transport.output(), # Send audio back to browser assistant_aggregator, # Add bot response to context ]) ``` Data flows through the pipeline as "frames", objects containing audio, text, or other data types. The ordering is crucial: audio must be transcribed before it can be processed by the LLM, and text must be synthesized before it can be played back. The pipeline is managed by a PipelineWorker: ```python # Create a PipelineWorker to manage the pipeline execution worker = PipelineWorker( pipeline, params=PipelineParams( enable_metrics=True, enable_usage_metrics=True, ), ) ``` The worker handles pipeline execution, collects metrics, and manages RTVI events through [observers](/api-reference/server/utilities/observers/observer-pattern). ### Event Handlers Event handlers manage the bot's lifecycle and user interactions: ```python # Kick off the conversation when the client signals it's ready @worker.rtvi.event_handler("on_client_ready") async def on_client_ready(rtvi): context.add_message({"role": "developer", "content": "Start by concisely introducing yourself."}) await worker.queue_frames([LLMRunFrame()]) # Event handler for when a client connects @transport.event_handler("on_client_connected") async def on_client_connected(transport, client): logger.info("Client connected") # Event handler for when a client disconnects @transport.event_handler("on_client_disconnected") async def on_client_disconnected(transport, client): logger.info("Client disconnected") # Cancel the worker when the client disconnects # This stops the pipeline and all processors, cleaning up resources await worker.cancel() ``` When the client signals it's ready over the RTVI protocol, the bot adds a greeting instruction and queues an `LLMRunFrame` to start the conversation. When the client disconnects, the worker is cancelled to clean up resources. ### Running the Pipeline Finally, the pipeline is executed by a WorkerRunner: ```python # Create a WorkerRunner to run the worker runner = WorkerRunner(handle_sigint=False) # Finally, add the worker and run it # This will start the pipeline and begin processing frames await runner.add_workers(worker) await runner.run() ``` The runner manages the pipeline's execution lifecycle. Note that `handle_sigint=False` because the main runner handles system signals. ### Bot Entry Point The quickstart uses Pipecat's runner system: ```python async def bot(runner_args: RunnerArguments): """Main bot entry point.""" # Configure transport parameters for different environments transport_params = { "daily": lambda: DailyParams( audio_in_enabled=True, audio_out_enabled=True, ), "webrtc": lambda: TransportParams( audio_in_enabled=True, audio_out_enabled=True, ), } transport = await create_transport(runner_args, transport_params) await run_bot(transport, runner_args) if __name__ == "__main__": from pipecat.runner.run import main main() ``` This runner automatically handles WebRTC connection setup and management, making it easy to get started with minimal configuration. The same code works for both local development and production deployment. **Production ready**: This bot pattern is fully compatible with Pipecat Cloud, meaning you can deploy your bot without any code changes. ## Troubleshooting - **Browser permissions**: Make sure to allow microphone access when prompted by your browser. - **Connection issues**: If the WebRTC connection fails, first try a different browser. If that fails, make sure you don't have a VPN or firewall rules blocking traffic. WebRTC uses UDP to communicate. - **Audio issues**: Check that your microphone and speakers are working and not muted. ## Next Steps Congratulations! You've built and deployed your first Pipecat bot. Here's what to do next based on your goals: ### 🚀 Ready to Build Your Own Application? The quickstart gave you a working example, but the Pipecat CLI helps you scaffold production-ready projects with your choice of platform (phone vs web/mobile), transport providers, and AI services—all tailored to your specific use case. Use the CLI to scaffold phone or web/mobile projects customized for your needs ### 🧠 Want to Understand How It Works? Dive deeper into Pipecat's architecture and learn how to build custom solutions. Master pipelines, processors, transports, and context management 30+ production-ready examples for inspiration # Build Your Next Bot Source: https://docs.pipecat.ai/pipecat/get-started/build-your-next-bot.md Scaffold a new Pipecat project and build it with a coding agent or by hand Ready to build your own bot? The Pipecat CLI takes you from zero to a runnable project, whether you build with an AI coding agent or scaffold it by hand. **New to Pipecat?** We recommend completing the [Quickstart](/pipecat/get-started/quickstart) first to understand Pipecat basics before scaffolding your own project. ## Install the Pipecat CLI Install the CLI globally with [uv](https://docs.astral.sh/uv/): ```bash uv tool install "pipecat-ai[cli]" ``` Verify installation: ```bash pipecat --version ``` ## Start with `pipecat init` `pipecat init` is the starting point. It writes the Pipecat coding-agent guide (`AGENTS.md` + `CLAUDE.md`) so your coding agent works well with Pipecat, then asks how you want to build: ```bash pipecat init ``` - **Build with a coding agent (recommended).** Let an AI coding assistant (Claude Code, Codex, …) do the building. Continue with [Build with a coding agent](#build-with-a-coding-agent) below. - **Scaffold a runnable bot now.** Prefer to drive the CLI yourself? Skip to [Scaffold it yourself](#scaffold-it-yourself). ## Build with a coding agent AI coding tools like Claude Code and Codex write your agent code. The agent follows the `AGENTS.md` guide that `init` wrote, so it uses Pipecat conventions, scaffolds the app for you, and verifies its own work. This path also writes `GETTING_STARTED.md`, a short guide to driving the agent well. ### Add the Pipecat Context Hub The Context Hub indexes Pipecat docs, examples, and API source into a local database, so your agent queries live context instead of stale training data. Co-install it with the CLI, then let it set itself up: ```bash uv tool install "pipecat-ai[cli]" --with pipecat-ai-context-hub pipecat context-hub install ``` `install` registers the MCP server with each coding agent it finds and builds the index — a few minutes the first time, since it downloads local models. MCP servers load at session start, so do this before opening your coding session. Full setup, the tool list, CLI lookups, and MCP config for Cursor and VS Code ### Start a coding session Open Claude Code or Codex in your project and prompt it to build. The agent follows `AGENTS.md`, queries the Context Hub for accurate APIs, and scaffolds and iterates on your bot. Prefer to feed context by hand? Pipecat docs support the [llms.txt standard](https://llmstxt.org/): [`llms.txt`](https://docs.pipecat.ai/llms.txt) is a structured index of all pages, and [`llms-full.txt`](https://docs.pipecat.ai/llms-full.txt) is the full documentation in a single file, useful for tools that ingest docs in bulk. ## Scaffold it yourself Prefer to drive the CLI directly? When you pick **Scaffold a runnable bot now**, `init` runs an interactive setup wizard that walks you through your platform (phone or web/mobile), transport provider, AI services (STT, LLM, TTS), and deployment target, then scaffolds the project in place alongside `AGENTS.md` + `CLAUDE.md`. The generated README includes Pipecat Context Hub setup, so the project is ready for a coding agent too. Once scaffolding completes, the CLI provides specific next steps for your generated project: - How to configure your API keys - Installing dependencies - Running your bot locally - Customizing the bot logic and behavior ## Deploy to Production Ready to deploy your bot? Choose between managed cloud hosting or self-hosted infrastructure. Deploy and manage your agents with the CLI - purpose-built for Pipecat Deploy to Fly.io, Modal, AWS, or your own infrastructure ## Learn More Understand pipelines, processors, and transports 30+ production-ready examples for inspiration # Continue Learning Source: https://docs.pipecat.ai/pipecat/get-started/next-steps.md Where to go after the Pipecat quickstart: learning paths, examples, and guides for building production voice agents. Now that you've run your first Pipecat bot, here's how to continue learning and build your own applications. ## Choose Your Path Understand pipelines, processors, transports, and how to build custom AI applications from the ground up. Dive into complete examples including multimodal bots, creative applications, and enterprise integrations. ## What's Your Goal? Choose your path based on what you want to accomplish: ### 🚀 Build a Production Application Ready to create your own bot? Use the CLI to scaffold a project tailored to your needs. Scaffold phone or web/mobile projects with the Pipecat CLI ### 🧠 Understand How Pipecat Works Master the fundamentals to build custom solutions and debug effectively. Learn about pipelines, processors, transports, and context management Function calling, audio recording, transcripts, and custom processors ### 💡 Get Inspired Explore real-world examples and recipes for common use cases. 30+ production-ready examples including multimodal bots and games Common patterns and solutions for specific use cases ### 🚀 Deploy to Production Scale your bot with managed hosting or self-hosted infrastructure. Managed hosting with auto-scaling and built-in WebRTC Deploy on Fly.io, Modal, Cerebrium, or your own infrastructure ## Need Help? Get support and share projects with other developers Report bugs or request features # Migrating to Pipecat 1.0 Source: https://docs.pipecat.ai/pipecat/migration/migration-1.0.md Upgrade a Pipecat application from 0.0.x to 1.0: removed deprecated APIs, their replacements, and the migration path. Pipecat 1.0 removes many deprecated APIs from the 0.0.x series. If you were already using the non-deprecated replacements, most of these changes won't affect you. Before upgrading, search your codebase for the deprecated imports and patterns listed below to identify what needs to change. Pipecat 1.0 requires **Python 3.11 or later**. Support for Python 3.10 has been dropped. Python 3.11 through 3.14 are supported. **Removed parameters are silently ignored, not errored.** `TransportParams` and `PipelineParams` are Pydantic models that drop unknown fields, so leftover 0.0.x params like `vad_analyzer`, `turn_analyzer`, and `allow_interruptions` are accepted with no error or warning. Your app can start cleanly and look upgraded while turn detection is actually broken (missed user speech, no bot responses). Don't rely on the app starting: delete these params from your transport/pipeline config and move them to `LLMUserAggregatorParams` as shown below. ## 1. Universal LLMContext `LLMContext` is a universal context that works with all LLM providers. It allows you to dynamically switch `LLMService` at runtime, passing a compatible context between services. Pipecat maintains adapters for each LLM service, which are applied automatically at the time of inference. Use the standard messages format along with [`FunctionSchema` and `ToolsSchema`](/pipecat/learn/function-calling) to define tools in a provider-agnostic way. ### Before (1.0) ```python from pipecat.services.openai.llm import OpenAILLMService from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext # Service-specific context context = OpenAILLMContext(messages, tools) # Service-specific aggregator factory context_aggregator = llm.create_context_aggregator(context) pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ]) ``` ### After (1.0) ```python from pipecat.services.openai.llm import OpenAILLMService from pipecat.processors.aggregators.llm_context import LLMContext from pipecat.processors.aggregators.llm_response_universal import LLMContextAggregatorPair from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Universal context — works with any LLM provider tools = ToolsSchema(standard_tools=[ FunctionSchema( name="get_weather", description="Get current weather", properties={"location": {"type": "string", "description": "City name"}}, required=["location"], ) ]) context = LLMContext(messages=messages, tools=tools) # Universal aggregator user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context) pipeline = Pipeline([ transport.input(), stt, user_aggregator, llm, tts, transport.output(), assistant_aggregator, ]) ``` ### What was removed | Removed | Replacement | | ---------------------------------------- | -------------------------------------- | | `OpenAILLMContext` | `LLMContext` | | `OpenAILLMContextFrame` | `LLMContextFrame` | | `AnthropicLLMContext` | `LLMContext` | | `AWSBedrockLLMContext` | `LLMContext` | | `GatedOpenAILLMContextAggregator` | `GatedLLMContextAggregator` | | `llm.create_context_aggregator(context)` | `LLMContextAggregatorPair(context)` | | `VisionImageFrame` | `LLMContext.add_image_frame_message()` | ## 2. Turn Management Turn management was unified into the user context aggregator configuration. Strategies for interruptions, muting, and idle detection are now configured via `LLMUserAggregatorParams`. ### Before (1.0) ```python from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.processors.user_idle_processor import UserIdleProcessor # Interruptions configured on the pipeline task task = PipelineTask( pipeline, params=PipelineParams(allow_interruptions=True), ) # Separate idle processor idle = UserIdleProcessor(timeout=5.0, callback=handle_idle) # STTMuteFilter stt_mute_filter = STTMuteFilter( config=STTMuteConfig(strategies={STTMuteStrategy.FIRST_SPEECH}) ) pipeline = Pipeline([ transport.input(), stt, stt_mute_filter, idle, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ]) ``` ### After (1.0) ```python from pipecat.processors.aggregators.llm_response_universal import ( LLMContextAggregatorPair, LLMUserAggregatorParams, ) from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3 from pipecat.turns.user_start import ( TranscriptionUserTurnStartStrategy, VADUserTurnStartStrategy, ) from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy from pipecat.turns.user_turn_strategies import UserTurnStrategies # Everything configured on the aggregator user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, user_params=LLMUserAggregatorParams( user_turn_strategies=UserTurnStrategies( start=[ VADUserTurnStartStrategy(), TranscriptionUserTurnStartStrategy(), ], stop=[ TurnAnalyzerUserTurnStopStrategy( turn_analyzer=LocalSmartTurnAnalyzerV3() ), ], ), user_mute_strategies=[ # Mute strategies ], user_turn_stop_timeout=5.0, user_idle_timeout=8.0, vad_analyzer=SileroVADAnalyzer(), ), ) ``` **Where did `allow_interruptions` go?** The `allow_interruptions` parameter was a legacy parameter used for non-voice AI applications. If you're looking to prevent an interruption from occurring, consider using a user mute strategy instead. See the [User Mute Strategies reference](/api-reference/server/utilities/turn-management/user-mute-strategies) for more information. ### What was removed | Removed | Replacement | | ---------------------------------------- | ------------------------------------------------------------ | | `PipelineParams.allow_interruptions` | `user_turn_strategies` on `LLMUserAggregatorParams` | | `PipelineParams.interruption_strategies` | `user_turn_strategies` on `LLMUserAggregatorParams` | | `UserResponseAggregator` | `LLMUserAggregator` (created via `LLMContextAggregatorPair`) | | `UserIdleProcessor` | `user_idle_timeout` on `LLMUserAggregatorParams` | | `STTMuteFilter` | `user_mute_strategies` on `LLMUserAggregatorParams` | | `MinWordsInterruptionStrategy` | `MinWordsUserTurnStartStrategy` | | `TranscriptionUserTurnStopStrategy` | `SpeechTimeoutUserTurnStopStrategy` | ## 3. VAD & Turn Analyzer Configuration VAD and turn detection configuration moved from transport params to the user aggregator. Additionally, the `VADParams` default `stops_secs` changed from 0.8 to 0.2. This ensures that STT services create transcripts with optimal latency and sufficient silence padding. The `UserTurnStrategies` (`stop`) now control how long to wait before the user's turn has stopped: - `SpeechTimeoutUserTurnStopStrategy` has a `user_speech_timeout` parameter that you can configure to control how long to wait before the user's turn has stopped. - `TurnAnalyzerUserTurnStopStrategy` uses the `smart-turn` model to dynamically modify the wait time. No configuration is required. ### Before (1.0) ```python from pipecat.transports.services.daily import DailyParams, DailyTransport from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.audio.vad.vad_analyzer import VADParams transport = DailyTransport( room_url, token, "Bot", DailyParams( vad_enabled=True, vad_audio_passthrough=True, vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)), turn_analyzer=LocalSmartTurnAnalyzerV3(params=SmartTurnParams()), ), ) ``` ### After (1.0) By default, the `TurnAnalyzerUserTurnStopStrategy` stop strategy is used, which includes the `LocalSmartTurnAnalyzerV3` turn analyzer. ```python from pipecat.transports.daily.transport import DailyTransport, DailyParams from pipecat.audio.vad.silero import SileroVADAnalyzer transport = DailyTransport( room_url, token, "Bot", DailyParams( audio_in_enabled=True, audio_out_enabled=True, ), ) # VAD is now configured on the aggregator user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, user_params=LLMUserAggregatorParams( vad_analyzer=SileroVADAnalyzer(), ), ) ``` ### What was removed | Removed | Replacement | | --------------------------------------- | --------------------------------------------------------------------------------- | | `TransportParams.vad_analyzer` | `LLMUserAggregatorParams.vad_analyzer` | | `TransportParams.vad_enabled` | Removed (VAD is active when analyzer is set) | | `TransportParams.vad_audio_passthrough` | Removed | | `TransportParams.turn_analyzer` | `UserTurnStrategies(stop=[TurnAnalyzerUserTurnStopStrategy()])` (now the default) | | `VADParams.stop_secs` default (0.8) | Now defaults to 0.2; turn timing is controlled by stop strategies | ## 4. Service Import Paths All flat service imports have been removed. Services now use submodule imports. ### Pattern change ```python # Before (1.0) from pipecat.services.openai import OpenAILLMService # After (1.0) from pipecat.services.openai.llm import OpenAILLMService ``` ### Renamed/moved service modules | Old module | New module | | ----------------------------------------- | ------------------------------------- | | `pipecat.services.gemini_multimodal_live` | `pipecat.services.google.gemini_live` | | `pipecat.services.aws_nova_sonic` | `pipecat.services.aws.nova_sonic` | | `pipecat.services.openai_realtime` | `pipecat.services.openai.realtime` | | `pipecat.services.riva` (STT) | `pipecat.services.nvidia.stt` | | `pipecat.services.riva` (TTS) | `pipecat.services.nvidia.tts` | | `pipecat.services.nim` | `pipecat.services.nvidia.llm` | ### Removed service classes These service have been removed. | Removed class | Replacement class | | ------------------------------ | -------------------------- | | `PollyTTSService` | `AWSPollyTTSService` | | `OpenAIRealtimeBetaLLMService` | `OpenAIRealtimeLLMService` | | `GoogleLLMOpenAIBetaService` | `GoogleLLMService` | ## 5. Transport Import Paths Transport imports moved from `services` subdirectory to transport-specific subdirectories. | Old import | New import | | ---------------------------------------------- | -------------------------------------- | | `pipecat.transports.services.daily` | `pipecat.transports.daily.transport` | | `pipecat.transports.services.livekit` | `pipecat.transports.livekit.transport` | | `pipecat.transports.network.websocket_server` | `pipecat.transports.websocket.server` | | `pipecat.transports.network.websocket_client` | `pipecat.transports.websocket.client` | | `pipecat.transports.network.fastapi_websocket` | `pipecat.transports.websocket.fastapi` | | `pipecat.sync` | `pipecat.utils.sync` | ## 6. Frames Several frame classes were renamed or removed. Update any direct references in your code. | Removed | Replacement | | ------------------------------------ | --------------------------------------------------------------- | | `TransportMessageFrame` | `OutputTransportMessageFrame` | | `TransportMessageUrgentFrame` | `OutputTransportMessageUrgentFrame` | | `InputTransportMessageUrgentFrame` | `InputTransportMessageFrame` | | `KeypadEntryFrame` | `DTMFFrame` | | `StartInterruptionFrame` | `InterruptionFrame` | | `BotInterruptionFrame` | `InterruptionWorkerFrame` | | `TranscriptionMessage` | Use events: `on_user_turn_stopped`, `on_assistant_turn_stopped` | | `TranscriptionUpdateFrame` | Use events: `on_user_turn_stopped`, `on_assistant_turn_stopped` | | `DailyTransportMessageFrame` | `DailyOutputTransportMessageFrame` | | `DailyTransportMessageUrgentFrame` | `DailyOutputTransportMessageUrgentFrame` | | `LiveKitTransportMessageFrame` | `LiveKitOutputTransportMessageFrame` | | `LiveKitTransportMessageUrgentFrame` | `LiveKitOutputTransportMessageUrgentFrame` | ## 7. Service-Specific Parameter Changes Individual services renamed or removed configuration parameters. Find your service below to see what changed. | Service | Removed param | Replacement | | -------------------------- | ---------------------------- | -------------------------------------- | | `DeepgramSTTService` | `url` | `base_url` | | `DeepgramSTTService` | `vad_events` | Removed | | `FishAudioTTSService` | `model` | `reference_id` | | `GladiaSTTService` | `InputParams.language` | Removed | | `GladiaSTTService` | `InputParams.confidence` | Removed | | `GeminiTTSService` | `api_key` | Removed | | `GeminiLiveLLMService` | `base_url` | `http_options` | | `GoogleVertexLLMService` | `InputParams` | Removed (`project_id` now required) | | `MiniMaxHttpTTSService` | `english_normalization` | `text_normalization` | | `SimliVideoService` | `simli_config` | `api_key` and `face_id` (required str) | | `AnthropicLLMService` | `enable_prompt_caching_beta` | `enable_prompt_caching` | | `AWSNovaSonicLLMService` | `send_transcription_frames` | Removed | | `OpenAIRealtimeLLMService` | `send_transcription_frames` | Removed | ## 8. Transport Parameters All `camera_*` transport parameters were renamed to `video_*`. VAD-related transport params were removed — VAD is now configured on the user aggregator (see [section 3](#3-vad-&-turn-analyzer-configuration)). | Removed | Replacement | | ---------------------------------------- | -------------------------------------- | | `camera_in_enabled` | `video_in_enabled` | | `camera_in_is_live` | `video_in_is_live` | | `camera_in_width` / `camera_in_height` | `video_in_width` / `video_in_height` | | `camera_out_enabled` | `video_out_enabled` | | `camera_out_is_live` | `video_out_is_live` | | `camera_out_width` / `camera_out_height` | `video_out_width` / `video_out_height` | | `camera_out_bitrate` | `video_out_bitrate` | | `camera_out_framerate` | `video_out_framerate` | | `camera_out_codec` | `video_out_codec` | | `vad_enabled` | Removed | | `vad_audio_passthrough` | Removed | ## 9. Other API Changes Smaller breaking changes across the rest of the API. ### Pipeline | Removed | Replacement | | ------------------------------------ | ---------------------------------------------- | | `PipelineParams.observers` | Pass `observers` to `PipelineTask` constructor | | `PipelineTask.on_pipeline_ended` | `on_pipeline_finished` | | `PipelineTask.on_pipeline_cancelled` | `on_pipeline_finished` | | `PipelineTask.on_pipeline_stopped` | `on_pipeline_finished` | | `DailyRunner.configure_with_args()` | Use `WorkerRunner` with `RunnerArguments` | ### TTS service | Removed | Replacement | | ---------------------------- | ---------------------- | | `TTSService.say()` | Push a `TTSSpeakFrame` | | `text_aggregator` init param | Use `LLMTextProcessor` | | `text_filter` init param | `text_filters` | ### Processors | Removed | Replacement | | --------------------------------------------- | --------------------------------------------------------------- | | `AudioBufferProcessor.user_continuous_stream` | `user_audio_passthrough` | | `UserBotLatencyLogObserver` | `UserBotLatencyObserver` (use `on_latency_measured` event) | | `TranscriptProcessor` | Use events: `on_user_turn_stopped`, `on_assistant_turn_stopped` | | `add_pattern_pair()` | `add_pattern()` | ### RTVI - Deprecated RTVI models (`RTVIConfig`, `RTVIServiceConfig`) removed - `RTVIActionFrame` removed - `RTVIProcessor.handle_function_call` and `handle_function_call_start` removed ### Other | Removed | Replacement | | ----------------------------------------------------------- | -------------------------------------------- | | `expect_stripped_words` from `LLMAssistantAggregatorParams` | Removed | | `context` field from `UserImageRequestFrame` | Removed | | Old multi-parameter `on_push_frame` observer signature | Use new signature | | `pipecat.utils.tracing.class_decorators` | Removed | | `NoisereduceFilter` | Use `KrispVivaFilter` or other audio filters | | `KrispFilter` | Use `KrispVivaFilter` | | `pipecat.turns.mute` | `pipecat.turns.user_mute` | # Overview of Pipecat Source: https://docs.pipecat.ai/pipecat/learn/overview.md Learn the foundational concepts of Pipecat's architecture for building voice AI agents ## What You'll Learn This comprehensive guide will teach you how to build real-time voice AI agents with Pipecat. By the end, you'll be equipped with the knowledge to create custom applications—from simple voice assistants to complex multimodal bots that can see, hear, and speak. **Prerequisites**: Basic Python knowledge is recommended. The guide takes approximately 45-60 minutes to complete, with hands-on examples throughout. ## Why Voice AI is Challenging Building responsive voice AI applications involves coordinating multiple AI services in real-time: - **Speech recognition** must transcribe audio as users speak - **Language models** need to process context and generate responses - **Speech synthesis** has to convert text back to natural audio - **Network transports** must handle streaming audio with minimal delay Doing this manually means managing complex timing, buffering, error handling, and service coordination. Most developers end up rebuilding the same orchestration logic repeatedly. ## Pipecat's Solution Pipecat solves this orchestration problem with a **pipeline architecture** that handles the complexity for you. Instead of managing individual API calls and timing, you define a flow of processing steps that work together automatically. Here's what makes Pipecat different: Typical voice interactions complete in 500-800ms for natural conversations Swap AI providers, add features, or customize behavior without rewriting code Stream processing eliminates waiting for complete responses at each step Built-in error handling, logging, and scaling considerations ## Core Architecture Concepts Before diving into how voice AI works, let's understand Pipecat's four foundational concepts: ### Frames Think of frames as **data packages** moving through your application. Each frame contains a specific type of information: - Audio data from a microphone - Transcribed text from speech recognition - Generated responses from an LLM - Synthesized audio for playback ### Frame Processors Frame processors are **specialized building blocks** that handle specific tasks: - A speech-to-text processor converts audio frames into text frames - An LLM processor takes text frames and produces response frames - A text-to-speech processor converts response frames into audio frames ### Pipelines Pipelines **connect processors together**, creating a path for frames to flow through your application. They handle the orchestration automatically. ### Workers A **worker** runs a pipeline. A worker that owns a pipeline is an **agent** in your application -- a standalone voice bot is a single worker (a `PipelineWorker`). Pipecat is a multi-agent system, so you can run several workers that coordinate over a shared bus. The `WorkerRunner` starts your workers and manages their lifecycle. ## Voice AI Processing Flow Now let's see how these concepts work together in a typical voice AI interaction: User speaks → Transport receives streaming audio → Creates audio frames STT processor receives audio frames → Transcribes speech in real-time → Outputs text frames Context processor aggregates text frames with conversation history → Creates formatted input for LLM LLM processor receives context → Generates streaming response → Outputs text frames TTS processor receives text frames → Converts to speech → Outputs audio frames Transport receives audio frames → Streams to user's device → User hears response The key insight: **everything happens in parallel**. While the LLM is generating later parts of a response, earlier parts are already being converted to speech and played back to the user. ## Pipeline Architecture Here's how this flow translates into a Pipecat pipeline: Pipecat Pipeline Architecture Each processor in the pipeline: 1. Receives specific frame types as input 2. Performs its specialized task (transcription, language processing, etc.) 3. Outputs new frames for the next processor 4. Passes through frames it doesn't handle While frames can flow upstream or downstream, most data flows downstream as shown above. We'll discuss pushing frames in later sections. ## What's Next In the following sections, we'll build a complete agent and explore each component in detail: - Building and running your first agent - How to initialize sessions and connect users - Configuring different transport options (Daily, WebRTC, Twilio, etc.) - Setting up speech recognition and synthesis services - Managing conversation context and LLM integration - Handling the complete pipeline lifecycle - Coordinating multiple agents that share a message bus Each section includes practical examples and configuration options to help you build production-ready voice AI applications. Let's build and run your first agent # Your First Agent Source: https://docs.pipecat.ai/pipecat/learn/your-first-agent.md Build and run a single agent: a PipelineWorker, the WorkerRunner, and the pipeline lifecycle. In Pipecat, an **agent** is a worker that runs a pipeline. A standalone voice bot is a single agent. Pipecat is a multi-agent system, so once you have one agent you can add more that coordinate over a shared bus -- but let's start with one. This page shows the whole shape; the sections that follow break down each piece (transports, speech-to-text, the LLM, text-to-speech, and more). ## Agent types Pipecat provides built-in worker types, each building on the previous: | Type | Purpose | | ------------------ | --------------------------------------------------------------------------------------------------- | | `BaseWorker` | Foundation for all agents. Connects to the bus, manages lifecycle, coordinates jobs. | | `PipelineWorker` | Extends `BaseWorker` to run a Pipecat pipeline. A standalone agent is a single `PipelineWorker`. | | `LLMWorker` | Extends `PipelineWorker` with an LLM pipeline and automatic `@tool` registration. | | `LLMContextWorker` | Extends `LLMWorker` with a built-in `LLMContext` and aggregator pair (its own or a shared context). | | `UIWorker` | Extends `LLMContextWorker` to read and drive a client GUI over the RTVI UI channel. | If you've built a Pipecat bot before, you've already used these. `PipelineTask` is a deprecated alias for `PipelineWorker`, and `PipelineRunner` for `WorkerRunner`, so existing `PipelineRunner().run(task)` code keeps working unchanged. New code should use `PipelineWorker` and `WorkerRunner` directly. `LLMWorker` gives an agent its own LLM and tools. It's covered in [Multiple LLM Agents](/pipecat/learn/multiple-llm-agents); this page focuses on building and running a single agent with a `PipelineWorker`. ## The WorkerRunner The `WorkerRunner` is the entry point. It: - Creates and manages the message bus - Starts agents and manages their lifecycle - Tracks agent readiness through a registry - Coordinates graceful shutdown When you don't provide a bus, the runner creates an `AsyncQueueBus` automatically -- an in-process bus backed by asyncio queues. For distributed setups, you can pass a network bus such as `RedisBus` or `PgmqBus` instead. Agents get their bus from the runner. You never pass a `bus=` argument to a worker constructor; you register agents with `runner.add_workers(...)`, which attaches them to the runner's bus and registry. Inside a running agent you can read `self.bus`. ## Building the agent A single agent is one `PipelineWorker` wrapping a complete pipeline -- the same `transport -> STT -> LLM -> TTS -> transport` flow you'd build for a standalone bot. The LLM runs inline in the pipeline. ```python import os from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline.pipeline import Pipeline from pipecat.workers.runner import WorkerRunner from pipecat.pipeline.worker import PipelineParams, PipelineWorker from pipecat.processors.aggregators.llm_context import LLMContext from pipecat.processors.aggregators.llm_response_universal import ( LLMContextAggregatorPair, LLMUserAggregatorParams, ) from pipecat.runner.types import RunnerArguments from pipecat.runner.utils import create_transport from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.services.deepgram.stt import DeepgramSTTService from pipecat.services.openai.llm import OpenAILLMService from pipecat.transports.base_transport import BaseTransport, TransportParams transport_params = { "webrtc": lambda: TransportParams( audio_in_enabled=True, audio_out_enabled=True, ), } async def run_bot(transport: BaseTransport, runner_args: RunnerArguments): runner = WorkerRunner(handle_sigint=runner_args.handle_sigint) stt = DeepgramSTTService(api_key=os.environ["DEEPGRAM_API_KEY"]) llm = OpenAILLMService( api_key=os.environ["OPENAI_API_KEY"], settings=OpenAILLMService.Settings( system_instruction="You are a helpful voice assistant. Keep responses brief.", ), ) tts = CartesiaTTSService( api_key=os.environ["CARTESIA_API_KEY"], settings=CartesiaTTSService.Settings( voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc", ), ) context = LLMContext() aggregators = LLMContextAggregatorPair( context, user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()), ) pipeline = Pipeline( [ transport.input(), stt, aggregators.user(), llm, tts, transport.output(), aggregators.assistant(), ] ) agent = PipelineWorker( pipeline, name="assistant", params=PipelineParams(enable_metrics=True, enable_usage_metrics=True), ) @transport.event_handler("on_client_disconnected") async def on_client_disconnected(transport, client): await runner.cancel() await runner.add_workers(agent) await runner.run() async def bot(runner_args: RunnerArguments): transport = await create_transport(runner_args, transport_params) await run_bot(transport, runner_args) if __name__ == "__main__": from pipecat.runner.run import main main() ``` That's a complete agent: one worker running one pipeline. ## Running the agent `runner.add_workers(...)` attaches each agent to the runner's bus and registry and starts it. `runner.run()` then blocks until one of these happens: - The session ends normally (the agent calls `end()`) - A signal is received (`SIGINT`/`SIGTERM`, when `handle_sigint=True`) - You call `runner.cancel()` - An unhandled error occurs In the example above, the runner is cancelled when the client disconnects. ## Configuring execution `PipelineParams` controls how the agent runs -- audio sample rates, metrics, and more: ```python params = PipelineParams( audio_in_sample_rate=16000, audio_out_sample_rate=24000, enable_metrics=True, enable_usage_metrics=True, ) ``` The full list of execution parameters ## Adding more agents This is a single agent. Because Pipecat is a multi-agent system, you can run several agents that each handle part of a conversation and coordinate over the shared bus. Later in this guide you'll see how: giving an agent its own LLM and tools, transferring control between agents with handoff, and dispatching work as jobs. ## What's next Now you've seen the shape of an agent. Next, let's connect users to it, starting with session initialization. Connect users and set up a session # Session Initialization Source: https://docs.pipecat.ai/pipecat/learn/session-initialization.md Learn how to set up connections between users and your Pipecat voice AI bot Before your voice AI bot can start processing audio and generating responses, you need to establish a connection between the user and your bot. This process is called **session initialization** - it's how users and bots find each other and establish a communication channel for real-time audio exchange. ## Understanding the Architecture Session initialization involves multiple components working together: - **Runner**: A FastAPI server that handles incoming connection requests and manages session setup - **Pipecat Bot**: Your voice AI application running as a separate server-side service - **Client Application**: The user-facing app (web browser, mobile app, etc.) The runner acts as the coordinator, setting up the necessary resources and starting bot instances, while the Pipecat bot handles the actual voice AI processing. ## Development Runner For most development and many production use cases, Pipecat provides a **development runner** that handles all the session initialization complexity for you. Instead of building FastAPI servers and managing WebRTC connections yourself, you focus on your bot logic while the runner handles the infrastructure. ### Using the Development Runner Your bot needs a single entry point function that the runner will call: ```python from pipecat.runner.types import RunnerArguments async def bot(runner_args: RunnerArguments): """Main bot entry point called by the development runner.""" # Create your transport based on the runner arguments transport = SmallWebRTCTransport( params=TransportParams( audio_in_enabled=True, audio_out_enabled=True, ), webrtc_connection=runner_args.webrtc_connection, ) # Run your bot logic await run_bot(transport) if __name__ == "__main__": from pipecat.runner.run import main main() ``` Then start your bot with different connection types: ```bash # P2P WebRTC (opens browser interface) python bot.py -t webrtc # Daily room-based WebRTC python bot.py -t daily # Telephony (requires ngrok or similar) python bot.py -t twilio -x your_domain.ngrok.io ``` where `-t` specifies the transport type (e.g., `webrtc`, `daily`, `twilio`) and `-x` is the optional proxy domain for telephony. `-t` is optional — omit it to serve all transports from one server and let the client choose via the `/start` request; pass it to restrict the runner to a single transport. The development runner automatically: - Creates the FastAPI server - Sets up the appropriate endpoints - Handles connection management - Starts your bot instances - Provides a web interface (for WebRTC) Learn more about building with the development runner in the [runner guide](/api-reference/server/utilities/runner/guide). ## Connection Types Under the Hood While the development runner handles the complexity, understanding the three connection patterns helps you choose the right approach and debug issues: ### 1. P2P WebRTC Connections **What happens:** 1. Runner serves a web interface at `http://localhost:7860/client` 2. When you open the page and connect, browser creates a WebRTC offer 3. Runner receives the offer, establishes connection, starts your bot 4. Browser and bot communicate directly via WebRTC **When to use:** Direct client connections, embedded applications, local development ### 2. Room-Based WebRTC (Daily) **What happens:** User visits the client application and clicks to start a session Runner calls Daily's API to create a room and tokens using `pipecat.runner.daily.configure()` Both user's browser and your bot join the same Daily room Once media streams are established, browser sends `client_ready` message Your bot receives the event and starts the conversation **When to use:** Video calls, group sessions, production deployments Room-based WebRTC can also be used for SIP or PSTN connections, which require different connection patterns. Refer to the [telephony guide](/pipecat/telephony/overview) for details. ### 3. WebSocket Connections (Telephony) **What happens:** 1. Telephony provider (Twilio, etc.) receives a phone call 2. Provider connects to your runner's webhook endpoint 3. Runner accepts WebSocket connection and parses telephony-specific messages 4. Your bot starts immediately with the parsed connection data **When to use:** Phone bots, telephony integrations ## Starting Conversations How and when your bot begins talking depends on the connection type: ### Immediate Start (P2P WebRTC, WebSocket) These connections are ready immediately, so you can start talking right after connection: ```python @transport.event_handler("on_client_connected") async def on_client_connected(transport, client): logger.info("Client connected - starting conversation") messages.append({ "role": "developer", "content": "Say hello and introduce yourself." }) await worker.queue_frames([LLMRunFrame()]) ``` ### Handshake Required (Client/Server Room-based WebRTC) For client/server applications using room-based WebRTC, a handshake ensures both sides are ready and the client won't miss the opening message: ```python @rtvi.event_handler("on_client_ready") async def on_client_ready(rtvi): await rtvi.set_bot_ready() # Confirm readiness to client # Start the conversation await worker.queue_frames([LLMRunFrame()]) ``` For client/server room-based connections, waiting for `on_client_ready` is crucial - starting too early can cause the client to miss part of the initial message. ## Process Isolation Each session runs its own dedicated bot instance for: - **Resource Management**: Dedicated CPU and memory per session - **Error Isolation**: One session crash doesn't affect others - **Clean Cleanup**: Resources automatically freed when sessions end The development runner handles this process management automatically. ## Custom Runners: When You Need More Control The development runner works for most cases, but sometimes you need custom behavior - specific authentication, custom endpoints, or integration with existing systems. For these cases, you can create your own FastAPI runner. The development runner source code ([available on GitHub](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/runner/run.py)) provides excellent examples for: **Daily Integration Example:** ```python from pipecat.runner.daily import configure @app.post("/start") async def start_bot(background_tasks: BackgroundTasks): async with aiohttp.ClientSession() as session: room_url, token = await configure(session) # Start bot instance background_tasks.add_task(run_bot, room_url, token) return {"room_url": room_url, "token": token} ``` **WebSocket Telephony Example:** ```python from pipecat.runner.utils import parse_telephony_websocket @app.websocket("/ws") async def websocket_endpoint(websocket: WebSocket): await websocket.accept() # Parse provider-specific messages transport_type, call_data = await parse_telephony_websocket(websocket) # Start bot with parsed data await run_telephony_bot(websocket, transport_type, call_data) ``` Refer to the development runner source code to understand these patterns before building custom runners. It handles many edge cases and provides battle-tested implementations. ## Key Takeaways - **Start with the development runner** for fastest development and learning - **Understand connection types** to choose the right approach for your use case - **Handle startup timing correctly** - immediate start vs. handshake patterns matter - **Plan for process isolation** - one bot instance per session is the recommended pattern - **Reference the source code** when building custom runners for production ## What's Next Now that you understand session initialization, let's explore the different transport options and how to configure them for your specific needs. Learn how Pipecat's pipeline architecture orchestrates frame processing for voice AI applications # Pipeline & Frame Processing Source: https://docs.pipecat.ai/pipecat/learn/pipeline.md Learn how Pipecat's pipeline architecture orchestrates frame processing for voice AI applications The **Pipeline** is the core orchestration component in Pipecat that connects frame processors together, creating a structured path for data to flow through your voice AI application. ## Basic Pipeline Structure A Pipeline takes a list of frame processors and connects them in sequence. Here's a simple voice AI pipeline that matches the voice AI agent architecture we discussed earlier: ```python pipeline = Pipeline([ transport.input(), # Receives user audio stt, # Speech-to-text conversion context_aggregator.user(), # Collect user responses llm, # Language model processing tts, # Text-to-speech conversion transport.output(), # Sends audio to user context_aggregator.assistant(), # Collect assistant responses ]) ``` ## Understanding Frames and Frame Processing Before diving deeper into pipelines, it's important to understand how data moves through them using **frames** and **frame processors**. ### What Are Frames? **Frames** are data containers that carry information through your pipeline. Think of them as packages on a conveyor belt. Each frame contains a specific type of data that processors can examine and act upon. Frames automatically receive unique identifiers and names (like `TranscriptionFrame#1`) that help with debugging and tracking data flow through your pipeline. Common frame types include: - **Audio frames**: Raw audio data from users or generated by TTS - **Text frames**: Transcriptions, LLM responses, or other text content - **Image frames**: Visual data for multimodal applications - **System frames**: Control signals for pipeline management - **Context frames**: Conversation history and state information Frames flow through your pipeline from processor to processor, carrying the data your voice AI application needs to operate. ### What Are Frame Processors? **Frame Processors** are the workers in your pipeline. Each processor has a specific job - like converting speech to text, generating responses, or playing audio. They: - **Receive frames** from the previous processor in the pipeline - **Process the data** (transcribe audio, generate text, etc.) - **Create new frames** with their output - **Pass frames along** to the next processor Frame processors are modular and reusable. You can swap out different STT services or LLM providers without changing the rest of your pipeline. ### Frame Types Frames in Pipecat have different base classes that determine how they're processed: ```python @dataclass class SystemFrame(Frame): """System frames are queued with high priority.""" pass @dataclass class DataFrame(Frame): """Data frames are queued and processed in order.""" pass @dataclass class ControlFrame(Frame): """Control frames are queued and processed in order.""" pass ``` **Key differences:** - **SystemFrames**: High-priority and ordered with other SystemFrames; interruptions do not discard them (interruptions, pipeline control, user input) - **DataFrames & ControlFrames**: Queued and processed in order (audio output, text, images) **Examples by type:** ```python # SystemFrames (high-priority and ordered with other SystemFrames) InputAudioRawFrame # User audio input UserStartedSpeakingFrame # Speech detection events InterruptionFrame # Interruption control ErrorFrame # Error notifications # DataFrames (queued and ordered) OutputAudioRawFrame # Audio for playback TextFrame # Text content TranscriptionFrame # Speech-to-text results LLMTextFrame # LLM responses AggregatedTextFrame # Text aggregated into a describable unit TTSTextFrame # Text-to-speech text output # ControlFrames (queued and ordered) EndFrame # Pipeline shutdown TTSStartedFrame # TTS response boundaries LLMFullResponseStartFrame # LLM response boundaries ``` ### Frame Processing Order **Frames are processed in guaranteed order within their processing lane**, even across ParallelPipelines. SystemFrames are queued in a high-priority lane and ordered with other SystemFrames; DataFrames and ControlFrames are queued together in the non-system lane and ordered with each other. This enables reliable sequencing. For example, you can push two non-system frames in order and the order will be respected. Additionally, the corresponding processing will finish before allowing the next non-system frame to be processed. Let's look at an example where we push two frames—`TTSSpeakFrame` and `EndFrame`—in order to say goodbye then end the pipeline: ```python from pipecat.frames.frames import EndFrame, TTSSpeakFrame # These will execute in order: speak first, then end pipeline await worker.queue_frames([ TTSSpeakFrame("Goodbye!"), EndFrame() ]) ``` ### How Frame Processors Work Every frame processor follows the same pattern with two key methods: ```python class TranscriptionLogger(FrameProcessor): async def process_frame(self, frame: Frame, direction: FrameDirection): # Always call parent first await super().process_frame(frame, direction) # Handle specific frame types if isinstance(frame, TranscriptionFrame): print(f"Transcription: {frame.text}") # Push frame to next processor await self.push_frame(frame, direction) ``` **Key methods:** - **`process_frame()`**: Inspect and handle incoming frames - **`push_frame()`**: Send frames upstream or downstream Learn how to build your own frame processors ## How Data Flows Through Pipelines Understanding data flow is crucial for building effective pipelines: ### Frame Processing Order **Order matters**: Processors must be arranged so that each receives the frame types it needs: 1. `transport.input()` creates `InputAudioRawFrame`s from user audio 2. `stt` receives audio frames and outputs `TranscriptionFrame`s 3. `llm` processes text and generates `LLMTextFrame`s 4. `tts` converts text frames to `TTSAudioRawFrame`s, `AggregatedTextFrame`s, and `TTSTextFrame`s 5. `transport.output()` creates `OutputAudioRawFrame`s and sends audio back to user - Note: An `LLMTextProcessor` can sit between the `llm` and `tts` to pre-aggregate `LLMTextFrame`s into `AggregatedTextFrame`s. This simply moves the aggregation step out of the TTS. ### Frame Propagation **Processors always push frames**: Processors don't consume frames, they pass them along: ```python # This pipeline allows multiple processors to use the same audio pipeline = Pipeline([ transport.input(), # Creates InputAudioRawFrame stt, # Uses audio → creates TranscriptionFrame # ... # Other processors can use the same audio tts, # Uses various text frames → creates TTSAudioRawFrame transport.output(), # Uses TTSAudioRawFrame → sends audio to user audio_buffer_processor, # Also uses the same user and assistant audio for recording ]) ``` This design allows multiple processors to operate on the same data stream without interfering with each other. ### Parallel Processing Patterns Use `ParallelPipeline` to create branches where each branch receives all upstream frames. Frames are collected and pushed individually from each branch: ```python pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, ParallelPipeline([ # English branch [FunctionFilter(english_filter), english_tts], # Spanish branch [FunctionFilter(spanish_filter), spanish_tts], ]), transport.output(), context_aggregator.assistant(), ]) ``` In this example: - Both TTS branches receive all LLM output - Each branch can filter and process frames independently - Results from both branches flow to `transport.output()` ParallelPipelines are traditionally paired with filters or gates to control which frames go where, allowing for complex conditional logic. ### Frame Queuing and Processing Frame processors have internal queues that ensure ordered processing: - **SystemFrames use a high-priority input queue** and are processed in order with other SystemFrames (interruptions, errors, input audio) - **DataFrames and ControlFrames use the non-system process queue** and are processed in order with each other - **Queuing is managed automatically** by the pipeline infrastructure - **Order is guaranteed within each lane** even across complex pipeline structures Learn more about frame flow patterns in the [Custom Frame Processor Guide](/pipecat/fundamentals/custom-frame-processor). ## Key Takeaways - **Order matters** - arrange processors so each gets the frames it needs - **Processors push frames** - processors pass frames downstream, not consume them - **Frame types determine processing** - SystemFrames use the high-priority input queue, while DataFrames and ControlFrames use the non-system process queue - **Queuing ensures reliability** - frames are processed in guaranteed order within their processing lane - **Parallel processing** enables conditional logic and multi-modal handling ## What's Next Now that you understand how pipelines orchestrate processing, let's explore the different transport options that connect your pipeline to users. Learn about the different ways users can connect to your voice AI pipeline # Transports Source: https://docs.pipecat.ai/pipecat/learn/transports.md Learn about the different ways users can connect to your Pipecat voice AI bot **Transports** are the communication layer between users and your Pipecat bot. They handle receiving and sending audio, video, and data, serving as the media interface that enables real-time interaction. ## Available Transport Types Pipecat supports multiple transport types to fit different use cases and deployment scenarios: WebRTC-based transport using Daily's infrastructure for video calls and conferencing WebSocket transport for telephony providers and custom WebSocket connections Specialized transport for HeyGen LiveAvatar video generation and streaming WebRTC transport using LiveKit's real-time communication platform Direct peer-to-peer WebRTC connections without cloud infrastructure Specialized transport for Tavus video generation and streaming General-purpose WebSocket transport for custom implementations ## Pipeline Integration Transports provide two key components for your pipeline: `input()` and `output()` methods. These methods define how the transport interacts with the pipeline: ### Transport Input and Output ```python pipeline = Pipeline([ transport.input(), # Receives user audio/video stt, context_aggregator.user(), llm, tts, transport.output(), # Sends bot audio/video context_aggregator.assistant(), # Processes after output ]) ``` **Key points about transport placement:** - **`transport.input()`** typically goes first in the pipeline to receive user input - **`transport.output()`** doesn't always go last - you may want processors after it - **Post-output processing** enables synchronized actions like: - Recording with word-level accuracy - Displaying subtitles synchronized to audio - Capturing context information precisely timed to output ## Transport Modularity Transports are modular components in your Pipeline, allowing you to flexibly change how users connect to your bot depending on the context. This modularity enables you to: - **Switch environments easily**: Use P2P WebRTC for development, Daily for production - **Support multiple connection types**: Same bot logic works across different transports - **Optimize for use case**: Choose the best transport for your specific requirements ## Transport Configuration All transports are configured using `TransportParams`, which provides common settings across transport types: ```python from pipecat.transports.base_transport import TransportParams params = TransportParams( # Audio settings audio_in_enabled=True, audio_out_enabled=True, # Video settings video_in_enabled=False, video_out_enabled=False, # Video stream configuration video_out_width=1024, video_out_height=576, video_out_bitrate=800000, video_out_framerate=30, ) ``` Each transport may have its own specialized parameters class that extends TransportParams with transport-specific options. Check the individual transport documentation for details. For advanced turn detection (like Smart Turn), configure [User Turn Strategies](/api-reference/server/utilities/turn-management/user-turn-strategies) on the context aggregator instead of using the transport's turn_analyzer parameter. Complete reference for all transport configuration options ## Telephony Integration Telephony services (phone calls) use WebSocket connections with specialized serialization: ### Supported Telephony Providers Media Streams over WebSocket with TwilioFrameSerializer Real-time media streaming with TelnyxFrameSerializer Voice streaming API with PlivoFrameSerializer Voice streaming integration with ExotelFrameSerializer ### Telephony Transport Setup Telephony requires a `FrameSerializer` to handle provider-specific message formats: ```python # Create provider-specific serializer serializer = TwilioFrameSerializer( stream_sid=stream_sid, call_sid=call_sid, account_sid=os.getenv("TWILIO_ACCOUNT_SID", ""), auth_token=os.getenv("TWILIO_AUTH_TOKEN", ""), ) # Configure transport with serializer transport = FastAPIWebsocketTransport( websocket=websocket_client, params=FastAPIWebsocketParams( audio_in_enabled=True, audio_out_enabled=True, add_wav_header=False, serializer=serializer, # Provider-specific serialization ), ) ``` The development runner automatically detects and configures the appropriate serializer when using `parse_telephony_websocket()`. ## Conditional Transport Selection The development runner provides a pattern for conditionally selecting transports based on the environment: ```python async def bot(runner_args: RunnerArguments): """Main bot entry point compatible with Pipecat Cloud.""" transport = None if isinstance(runner_args, DailyRunnerArguments): from pipecat.transports.daily.transport import DailyParams, DailyTransport transport = DailyTransport( runner_args.room_url, runner_args.token, "Pipecat Bot", params=DailyParams( audio_in_enabled=True, audio_out_enabled=True, ), ) elif isinstance(runner_args, SmallWebRTCRunnerArguments): from pipecat.transports.base_transport import TransportParams from pipecat.transports.network.small_webrtc import SmallWebRTCTransport transport = SmallWebRTCTransport( params=TransportParams( audio_in_enabled=True, audio_out_enabled=True, ), webrtc_connection=runner_args.webrtc_connection, ) else: logger.error(f"Unsupported runner arguments type: {type(runner_args)}") return if transport is None: logger.error("Failed to create transport") return await run_bot(transport) ``` This pattern allows you to run the same bot code across different environments with different connection types. ## WebRTC vs WebSocket Considerations Understanding when to use each connection type is crucial for building effective voice AI applications: ### WebRTC (Recommended for Client Applications) **Best for:** Browser apps, mobile apps, real-time conversations **Advantages:** - **Low latency**: Optimized for real-time media with minimal delay - **Built-in resilience**: Handles packet loss and network variations - **Advanced audio processing**: Echo cancellation, noise reduction, automatic gain control - **Quality monitoring**: Detailed performance and media quality statistics - **Automatic timestamping**: Simplifies interruption and playout logic - **Robust reconnection**: Built-in connection management **Use WebRTC when:** - Building client-facing applications (web, mobile) - Conversational latency is critical - Users are on potentially unreliable networks - You need built-in audio processing features ### WebSocket (Good for Server-to-Server) **Best for:** Telephony integration, server-to-server communication, prototyping **Limitations for real-time media:** - **TCP-based**: Subject to head-of-line blocking - **Network sensitivity**: Less resilient to packet loss and jitter - **Manual implementation**: Requires custom logic for reconnection, timestamping - **Limited observability**: Harder to monitor connection quality **Use WebSocket when:** - Integrating with telephony providers (Twilio, Telnyx, etc.) - Building server-to-server connections - Prototyping or latency isn't critical - Working within existing WebSocket infrastructure ## Key Takeaways - **Transports are modular** - swap them without changing bot logic - **Choose based on use case** - WebRTC for clients, WebSocket for telephony - **Configuration is standardized** - TransportParams work across transport types - **Pipeline placement matters** - consider what processing happens after output - **Development runner helps** - provides patterns for multi-transport bots ## What's Next Now that you understand how transports connect users to your bot, let's explore how to configure speech recognition to convert user audio into text. Learn how to configure speech recognition in your voice AI pipeline # Speech Input & Turn Detection Source: https://docs.pipecat.ai/pipecat/learn/speech-input.md Learn how Pipecat detects user turns using VAD, transcriptions, and turn detection models A key to natural conversations is properly detecting when the user starts and stops speaking. This is more nuanced than simply detecting audio; a brief pause doesn't always mean the user is done talking. ## Overview Pipecat uses [user turn strategies](/api-reference/server/utilities/turn-management/user-turn-strategies) to determine when user turns start and end. These strategies can use different techniques: **For detecting turn start:** - Voice Activity Detection (VAD): triggers when speech is detected - Transcription-based (fallback): triggers when transcription is received but VAD didn't detect speech - Minimum words: waits for a minimum number of spoken words before triggering **For detecting turn end (default: Smart Turn):** - Turn detection model (default): uses AI to understand if the user has finished their thought - Speech timeout: waits for silence after transcription to determine when the user is done Custom strategies can also be implemented for specific use cases. By combining these techniques, you can create responsive yet natural conversations that don't interrupt users mid-sentence or wait too long after they've finished. ## Voice Activity Detection (VAD) ### What VAD Does VAD is responsible for detecting when a user starts and stops speaking. Pipecat includes [Silero VAD](https://github.com/snakers4/silero-vad), an open-source model that runs locally on CPU with minimal overhead. [Krisp VIVA VAD](/api-reference/server/services/vad/krisp-viva-vad-analyzer) is also available for applications requiring support for higher sample rates. **Silero VAD performance characteristics:** - Processes 30+ms audio chunks in less than 1ms - Runs on a single CPU thread - Minimal system resource impact ### VAD Configuration VAD is configured through `VADParams` in the `LLMContextAggregatorPair`: ```python from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.audio.vad.vad_analyzer import VADParams from pipecat.processors.aggregators.llm_response_universal import ( LLMContextAggregatorPair, LLMUserAggregatorParams, ) vad_analyzer = SileroVADAnalyzer( params=VADParams( confidence=0.7, # Minimum confidence for voice detection start_secs=0.2, # Time to wait before confirming speech start stop_secs=0.2, # Time to wait before confirming speech stop min_volume=0.6, # Minimum volume threshold ) ) user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, user_params=LLMUserAggregatorParams(vad_analyzer=vad_analyzer), ) ``` VAD is configured on the user aggregator because its speech start/stop signals feed the [user turn strategies](/api-reference/server/utilities/turn-management/user-turn-strategies) that decide when a turn begins and ends. In the vast majority of cases, the default values will work well. Only adjust these parameters if you have specific audio conditions that require it. ### Key Parameters **`start_secs` (default: 0.2)** - How long a user must speak before VAD confirms speech has started - Lower values = more responsive, but may trigger on brief sounds - Higher values = less sensitive, but may miss quick utterances, like "yes", "no", or "ok" **`stop_secs` (default: 0.2)** - How much silence must be detected before confirming speech has stopped - Critical for turn-taking behavior - A short value (0.2s) allows STT services to finalize sooner, improving transcription speed - **Important**: Built-in STT P99 latency values are measured with `stop_secs=0.2`. If you change this value, re-run the [stt-benchmark](https://github.com/pipecat-ai/stt-benchmark) with your settings and pass the measured latency to your STT service via `ttfs_p99_latency` **`confidence` and `min_volume`** - Generally work well with defaults - Only adjust after extensive testing with your specific audio conditions Changing confidence and min_volume requires careful profiling to ensure optimal performance across different audio environments and use cases. `confidence` and `min_volume` only raise the bar for what counts as speech — blunt instruments for noisy audio. If your bot reacts to background voices or your STT transcribes noise, it's usually better to remove the noise upstream with an input audio filter — [Krisp VIVA](/pipecat/features/krisp-viva), [ai-coustic](/api-reference/server/services/audio-filters/aic-filter), or [RNNoise](/api-reference/server/services/audio-filters/rnnoise-filter) — before the audio reaches VAD and STT. ## User Turn Detection While VAD detects speech vs. silence, it can't understand linguistic context. A pause doesn't mean the user is done. User turn strategies interpret VAD signals and transcriptions to determine actual turn boundaries. ### How It Works 1. **Turn Start**: When VAD detects speech (or transcription arrives), the start strategy emits `UserStartedSpeakingFrame` and optionally triggers an interruption 2. **Turn End**: When the stop strategy determines the user is done, it emits `UserStoppedSpeakingFrame` VAD also emits its own frames (`VADUserStartedSpeakingFrame`, `VADUserStoppedSpeakingFrame`) which indicate raw speech/silence detection. These are inputs to the turn strategies, not the final turn decisions. ### Detecting Turn End Turn end detection determines when the user has finished speaking and expects a response: **Smart Turn Model (Default)**: Uses an AI model to analyze audio and determine if the user has finished their thought. This is the default turn stop strategy. ```python from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3 from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy stop_strategy = TurnAnalyzerUserTurnStopStrategy( turn_analyzer=LocalSmartTurnAnalyzerV3() ) ``` **Speech Timeout**: Waits for a configurable timeout after VAD detects silence and a transcript is received. Useful as a simpler alternative to Smart Turn. ```python from pipecat.turns.user_stop import SpeechTimeoutUserTurnStopStrategy stop_strategy = SpeechTimeoutUserTurnStopStrategy(user_speech_timeout=0.6) ``` Complete reference for start and stop strategies Smart Turn model implementation guide ### Interruptions When a user turn starts with interruptions enabled (the default, via the `enable_interruptions` parameter on start strategies), the bot immediately stops speaking, in-flight work is cancelled, and the pipeline is ready for the new user input. Keep interruptions enabled for natural conversations. For how the cancellation works, what ends up in the LLM context, and how to tune, disable, or trigger interruptions yourself, see [Interruptions](/pipecat/fundamentals/interruptions). ## Best Practices ### Optimal Configuration **For most voice AI use cases, the defaults work well:** ```python from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.processors.aggregators.llm_response_universal import ( LLMContextAggregatorPair, LLMUserAggregatorParams, ) transport = YourTransport( params=TransportParams(), ) # Default configuration: Smart Turn detection + VAD with stop_secs=0.2 user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()), ) worker = PipelineWorker(pipeline) ``` The defaults use `TurnAnalyzerUserTurnStopStrategy` with `LocalSmartTurnAnalyzerV3` for turn detection and `SileroVADAnalyzer` with `stop_secs=0.2` for voice activity detection. ### Performance Considerations - **Use local VAD**: 150-200ms faster than remote VAD services - **Tune for your use case**: Test with real audio conditions - **Monitor CPU usage**: VAD adds minimal overhead but monitor in production - **Consider turn detection**: Improves conversation quality but adds complexity ## Key Takeaways - **VAD detects speech activity** but turn detection understands conversation context - **Configuration affects user experience** - tune parameters for your specific use case - **System frames coordinate behavior** - enable interruptions and natural turn-taking - **Local processing is faster** - Silero VAD provides low-latency speech detection - **Turn detection improves quality** - but requires careful VAD configuration ## What's Next Now that you understand how speech input is detected and processed, let's explore how that audio gets converted to text through speech recognition. Learn how to configure speech recognition in your voice AI pipeline # Speech to Text Source: https://docs.pipecat.ai/pipecat/learn/speech-to-text.md Learn how to configure speech recognition to convert user audio into text in your Pipecat pipeline **Speech to Text (STT)** services are responsible for converting user audio into text transcriptions. They receive audio input from users and provide real-time transcriptions that your bot can process and respond to. ## Pipeline Placement STT processors must be positioned correctly in your pipeline to receive and process audio frames: ```python pipeline = Pipeline([ transport.input(), # Creates InputAudioRawFrames stt, # Processes audio → creates TranscriptionFrames context_aggregator.user(), # Uses transcriptions for context llm, tts, transport.output(), ]) ``` **Placement requirements:** - **After `transport.input()`**: STT needs `InputAudioRawFrame`s from the transport - **Before context processing**: Transcriptions must be available for context aggregation - **Before LLM processing**: Text must be ready for language model input ## STT Service Types Pipecat provides two types of STT services based on how they process audio: ### 1. STTService (Streaming) **How it works:** - Establishes a WebSocket connection to the STT provider - Continuously streams audio for real-time transcription - Lower latency due to persistent connection ### 2. SegmentedSTTService (HTTP-based) **How it works:** - Uses local VAD (Voice Activity Detection) to chunk speech - Sends audio segments to STT service as wav files - Higher latency due to segmentation and HTTP POST requests STT services are modular and can be swapped out with no additional overhead. You can easily switch between streaming and segmented services based on your needs. ## Supported STT Services Pipecat supports a wide range of STT providers to fit different needs and budgets: View the complete list of supported speech-to-text providers Popular options include: Fast, accurate streaming STT with excellent real-time performance Real-time streaming STT with strong multilingual support and language hints AI-powered transcription with speaker diarization and sentiment analysis Advanced speech recognition with strong accent and dialect handling Low-latency real-time transcription from the Cartesia voice platform Real-time streaming STT with a low-latency transcription API ## STT Configuration ### Service-Specific Configuration Each STT service has its own customization options. Refer to specific service documentation for details: Explore configuration options for each supported STT provider For example, let's look at configuring the **DeepgramSTTService** using the `LiveOptions` class: ```python from deepgram import LiveOptions from pipecat.services.deepgram.stt import DeepgramSTTService from pipecat.transcriptions.language import Language # Configure using LiveOptions for full control live_options = LiveOptions( model="nova-2", language=Language.EN_US, interim_results=True, # Enable interim transcripts punctuate=True, # Add punctuation profanity_filter=True, # Filter profanity vad_events=False, # Use pipeline VAD instead ) stt = DeepgramSTTService( api_key=os.getenv("DEEPGRAM_API_KEY"), live_options=live_options, ) ``` ### STTService Base Class Configuration All STT services inherit from the STTService base class. The base class has base configuration options which are set with smart defaults: ```python stt = YourSTTService( # Service-specific options... audio_passthrough=True, # Pass audio frames downstream (recommended) sample_rate=16000, # Audio sample rate (better set in PipelineParams) ) ``` **Key options:** - **`audio_passthrough=True`**: Allows audio frames to continue downstream to other processors (like audio recording) - **`sample_rate`**: Audio sampling rate - best practice is to **set the `audio_in_sample_rate` in `PipelineParams` for consistency** Setting `audio_passthrough=False` will stop audio frames from being passed downstream, which may break audio recording or other audio-dependent processors. ### Pipeline-Level Audio Configuration Instead of setting sample rates on individual services, configure them pipeline-wide: ```python worker = PipelineWorker( pipeline, params=PipelineParams( audio_in_sample_rate=16000, # All input processors use this rate audio_out_sample_rate=24000, # All output processors use this rate ), ) ``` This ensures all audio processors use consistent sample rates without manual configuration. Always set audio sample rates in `PipelineParams` to avoid mismatches between different audio processors. This simplifies configuration and ensures consistent audio quality across your pipeline. ## Multilingual Transcription Many STT services in Pipecat default to `Language.EN` (English). If you need to transcribe speech in other languages or let the model auto-detect the spoken language, you can enable multilingual support. However, providers implement this differently: **`language=None`** — Whisper-based services (Groq, OpenAI, local Whisper) and ElevenLabs support automatic language detection when no language is specified: ```python from pipecat.services.groq.stt import GroqSTTService stt = GroqSTTService( api_key=os.getenv("GROQ_API_KEY"), settings=GroqSTTService.Settings( language=None, # Auto-detect language ), ) ``` **`language="multi"`** — Deepgram uses a special `"multi"` language code to enable multilingual transcription: ```python from pipecat.services.deepgram.stt import DeepgramSTTService stt = DeepgramSTTService( api_key=os.getenv("DEEPGRAM_API_KEY"), settings=DeepgramSTTService.Settings( language="multi", # Enable multilingual mode ), ) ``` **Language array** — Google Cloud STT accepts a list of languages for multi-language recognition. See the [Google STT docs](/api-reference/server/services/stt/google) for details. Some services have additional multilingual features. For example, Soniox supports language hints, AssemblyAI offers a dedicated multilingual model, and Speechmatics supports bilingual transcription. See individual service docs for details. ## Best Practices ### Enable Interim Results When available, enable interim transcripts for better user experience: ```python stt = DeepgramSTTService( api_key=os.getenv("DEEPGRAM_API_KEY"), live_options=LiveOptions( interim_results=True, ) ) ``` **Benefits:** - Notifies context aggregation that more text is coming - Prevents premature LLM completions - Enables interruption detection - Improves conversation flow ### Enable Punctuation and Formatting Use punctuation when available for better LLM comprehension: ```python stt = DeepgramSTTService( api_key=os.getenv("DEEPGRAM_API_KEY"), live_options=LiveOptions( punctuate=True, # Adds punctuation profanity_filter=True, # Optional content filtering ) ) ``` **Benefits:** - Professional-looking transcripts - Better LLM comprehension - Eliminates post-processing needs - Improved context understanding ### Use Local VAD While many STT services provide Voice Activity Detection, use Pipecat's local Silero VAD for better performance: ```python from pipecat.audio.vad.silero import SileroVADAnalyzer # Configure in context aggregator user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, user_params=LLMUserAggregatorParams( vad_analyzer=SileroVADAnalyzer(), ), ) ``` **Advantages:** - **150-200ms faster** speech detection (no network round trip) - More responsive conversation flow - Better interruption handling - Reduced latency overall ### Tune STT Latency Each STT service has a measured P99 latency for delivering final transcripts after the user stops speaking. This value is used by turn stop strategies to decide how long to wait before ending the user's turn. If you notice the bot responding too early (cutting off the user) or too late (long pauses), tuning this value can help. Learn about TTFS latency, see default values for every STT service, and how to measure and override for your deployment ## Key Takeaways - **Pipeline placement matters** - STT must come after transport input, before context processing - **Service types differ** - streaming services have lower latency than segmented - **Services are modular** - easily swap providers without code changes - **Best practices improve performance** - use interim results, formatting, and local VAD - **Configuration affects quality** - proper setup significantly impacts transcription accuracy ## What's Next Now that you understand speech recognition, let's explore how to manage conversation context and memory in your voice AI bot. Learn how to handle conversation history and context in your pipeline # Context Management Source: https://docs.pipecat.ai/pipecat/learn/context-management.md Work with Pipecat's LLM context and context aggregators: how conversation history is built and shared in a pipeline. ## What is Context in Pipecat? In Pipecat, **context** refers to the conversation history that the LLM uses to generate responses. The context consists of user/assistant messages representing the conversation history, and can also include **developer messages** for task-specific instructions to the LLM. ```python # Example context structure messages = [ {"role": "developer", "content": "Keep responses under two sentences."}, {"role": "user", "content": "Hello!"}, {"role": "assistant", "content": "Hi there! How can I help you?"}, # Context aggregators automatically add new messages here ] ``` The system prompt is typically set via `system_instruction` in the LLM service's [Settings](/pipecat/fundamentals/service-settings), not as a message in the context. See [System Instruction, Developer Messages, and System Messages](#system-instruction-developer-messages-and-system-messages) below for details. Since Pipecat is a real-time voice AI framework, context management happens automatically as the conversation flows, but you can also control it manually when needed. ## How Context Updates During Conversations Context updates happen automatically as frames flow through your pipeline: **User Messages:** 1. User speaks → `InputAudioRawFrame` → STT Service → `TranscriptionFrame` 2. `context_aggregator.user()` receives `TranscriptionFrame` and adds user message to context **Assistant Messages:** 1. LLM generates response → `LLMTextFrame` → TTS Service → `TTSTextFrame` 2. `context_aggregator.assistant()` receives `TTSTextFrame` and adds assistant message to context **Frame types that update context:** - **`TranscriptionFrame`**: Contains user speech converted to text by STT service - **`LLMTextFrame`**: Contains LLM-generated responses - **`TTSTextFrame`**: Contains bot responses converted to text by TTS service (represents what was actually spoken) The TTS service processes `LLMTextFrame`s but outputs `TTSTextFrame`s, which represent the actual spoken text returned by the TTS provider. This ensures context matches what users actually hear. ## Setting Up Context Management Pipecat includes a context aggregator that creates and manages context for both user and assistant messages: ### 1. Create the Context and Context Aggregator ```python # Create LLM service with system instruction llm = OpenAILLMService( api_key=os.getenv("OPENAI_API_KEY"), settings=OpenAILLMService.Settings( model="gpt-4o", system_instruction="You are a helpful voice assistant.", ), ) # Create context (no system message needed — system_instruction handles it) context = LLMContext() # Create context aggregator instance user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context) ``` The context aggregator also supports configuring [user turn strategies](/api-reference/server/utilities/turn-management/user-turn-strategies) and [user mute strategies](/api-reference/server/utilities/turn-management/user-mute-strategies) via `LLMUserAggregatorParams`. **About LLMContext:** `LLMContext` is Pipecat's universal context container that stores conversation messages, tool definitions, and tool choice settings. It uses an OpenAI-compatible format that works across all LLM services through automatic adapter translation. Key properties: - **`messages`**: List of conversation messages (user, assistant, developer, tool) - **`tools`**: Optional available functions — a list of direct functions and/or `FunctionSchema` objects (or a `ToolsSchema`) - **`tool_choice`**: Optional strategy for tool selection ### 2. Context with Function Calling Context can also include [tools](/pipecat/learn/function-calling#1-define-a-tool) (function definitions) that the LLM can call during conversations: ```python from pipecat.services.llm_service import FunctionCallParams # A direct function: schema is auto-derived from the signature and docstring async def get_current_weather(params: FunctionCallParams, location: str, format: str): """Get the current weather. Args: location: The city and state, e.g. "San Francisco, CA". format: The temperature unit to use. Must be either "celsius" or "fahrenheit". """ await params.result_callback({"conditions": "sunny", "temperature": "75"}) # Create context with both messages and tools context = LLMContext(messages, tools=[get_current_weather]) user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context) ``` Function call results are also automatically stored in the context, maintaining a complete conversation history including tool interactions. We'll cover function calling in detail in an upcoming section. The context aggregator handles function call storage automatically. ### 3. Add Context Aggregators to Your Pipeline ```python pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), # User context aggregator llm, tts, transport.output(), context_aggregator.assistant(), # Assistant context aggregator ]) ``` ## System Instruction, Developer Messages, and System Messages Pipecat provides three ways to give instructions to your LLM, each suited for a different purpose. ### Using `system_instruction` (recommended) Set the bot's personality and core behavior via the LLM service's Settings. The service automatically prepends it to the context messages on each request: ```python llm = OpenAILLMService( api_key=os.getenv("OPENAI_API_KEY"), settings=OpenAILLMService.Settings( model="gpt-4o", system_instruction="You are a helpful voice assistant.", ), ) # Context only needs conversation messages context = LLMContext() ``` This approach is recommended because: - **Survives context updates**: The system prompt is always prepended, even after `LLMMessagesUpdateFrame` replaces the context or context summarization compresses older messages. - **Shared context**: Multiple LLM services can share a single `LLMContext` while each provides its own system instruction. - **Runtime updates**: You can change the system prompt mid-conversation via `LLMUpdateSettingsFrame`. ### Using developer messages in context **Developer messages** (`"role": "developer"`) are task-specific instructions placed directly in the context. Use them for supplementary guidance that applies to a particular phase of the conversation rather than defining the bot's overall personality. ```python messages = [ {"role": "developer", "content": "Keep responses under two sentences. Use metric units."}, ] context = LLMContext(messages) ``` Key characteristics: - **Task-specific**: Use for instructions like response format constraints, domain rules, or workflow steps. The bot's personality belongs in `system_instruction`. - **Part of normal context flow**: Developer messages participate in context like any other message. They are included in the summarization range and may be compressed or dropped during [context summarization](/pipecat/fundamentals/context-summarization). - **Cross-provider translation**: Pipecat's adapters automatically convert developer messages for providers that don't support the role natively (for example, Anthropic receives them as user messages). ### Using a context system message (legacy) You can also include a system message directly in the context messages: ```python messages = [ {"role": "system", "content": "You are a helpful voice assistant."}, ] context = LLMContext(messages) ``` This works but has limitations: the system message can be lost during full context replacement, and it cannot be shared across multiple LLM services with different system prompts. Context summarization does preserve `messages[0]` when it's a system message, but `system_instruction` is still more reliable because it sits outside the context entirely. For new projects, prefer `system_instruction` for personality and developer messages for task-specific instructions. If both `system_instruction` and a system message in the context are set, `system_instruction` takes precedence and a warning is logged. Avoid using both at the same time. ## Context Aggregator Placement The placement of context aggregator instances in your pipeline is **crucial** for proper operation: ### User Context Aggregator Place the user context aggregator **downstream from the STT service**. Since the user's speech results in `TranscriptionFrame` objects pushed by the STT service, the user aggregator needs to be positioned to collect these frames. ### Assistant Context Aggregator Place the assistant context aggregator **after `transport.output()`**. This positioning is important because: - The TTS service outputs `TTSTextFrame`s in addition to audio - The assistant aggregator must be downstream to collect those frames - It ensures context updates happen word-by-word for specific services (e.g. Cartesia, ElevenLabs, and Rime) - Your context stays updated at the word level in case an interruption occurs Always place the assistant context aggregator **after** `transport.output()` to ensure proper word-level context updates during interruptions. See [Interruptions](/pipecat/fundamentals/interruptions#what-ends-up-in-the-context) for what the context contains when the bot is cut off mid-sentence. ## Manual Context Control You can programmatically add new messages to the context by pushing or queueing specific frames: ### Adding Messages - **`LLMMessagesAppendFrame`**: Appends a new message to the existing context - **`LLMMessagesUpdateFrame`**: Completely replaces the existing context with new messages - **`LLMMessagesTransformFrame`**: Edits the existing context in place using a transform function ```python # Add a new user message to context and trigger a response new_message = {"role": "user", "content": "Tell me about your capabilities."} await worker.queue_frames([ LLMMessagesAppendFrame([new_message], run_llm=True), # Optionally trigger bot response, too ]) ``` #### Adding a message silently All three frames take a `run_llm` argument that controls whether the change also prompts a bot response. Pass `run_llm=True` to respond; the default (`None`, which behaves like `False`) updates the context silently. This is useful when you collect information in the background and don't want the bot to react every time: ```python # Add a message to context without triggering a bot response note = {"role": "user", "content": "Caller's name is Maria. Account verified."} await worker.queue_frames([ LLMMessagesAppendFrame([note], run_llm=False), # Silent: no response ]) ``` #### Editing or removing specific messages To surgically edit or remove individual messages without rebuilding the whole list, push an `LLMMessagesTransformFrame`. It takes a function that receives the current list of messages and returns a modified list. Use it to drop stale instructions, remove an offensive turn, or rewrite content: ```python from pipecat.frames.frames import LLMMessagesTransformFrame def remove_language_instructions(messages): # Drop any message that contains a per-turn language instruction return [m for m in messages if "LANGUAGE INSTRUCTION" not in str(m.get("content", ""))] await worker.queue_frames([ LLMMessagesTransformFrame(transform=remove_language_instructions, run_llm=False), ]) ``` To make the bot **speak** specific text and have it recorded in context (for example, a greeting the LLM did not generate), use [`TTSSpeakFrame` with `append_to_context=True`](/pipecat/learn/text-to-speech). That path is for driving speech output; the frames above are for editing the context directly. ### Retrieving Current Context The context aggregator provides a `context` property for getting the current context: ```python context = context_aggregator.user().context ``` ## Context Summarization In long-running conversations, context grows with every exchange, increasing token usage and potentially hitting context window limits. Pipecat includes built-in context summarization that automatically compresses older conversation history while preserving recent messages. Enable it by setting `enable_auto_context_summarization=True` when creating your context aggregators (default: `False`): ```python user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, assistant_params=LLMAssistantAggregatorParams( enable_auto_context_summarization=True, ), ) ``` Learn how to configure summarization triggers, customize behavior, and control what gets preserved. ## Triggering Bot Responses You may want to manually trigger the bot to speak in two scenarios: 1. **Starting a pipeline** where the bot should speak first 2. **After editing the context** using `LLMMessagesAppendFrame` or `LLMMessagesUpdateFrame` ```python # Example: Bot speaks first when pipeline starts @transport.event_handler("on_client_connected") async def on_client_connected(transport, client): # Trigger a response await worker.queue_frames([LLMRunFrame()]) ``` ```python # Example: Bot speaks after context is edited new_message = {"role": "user", "content": "Tell me a fun fact."} await worker.queue_frames([ LLMMessagesAppendFrame([new_message], run_llm=True), # Trigger bot response ]) ``` This gives you fine-grained control over when and how the bot responds during the conversation flow. ## Key Takeaways - **Context is conversation history** - automatically maintained as users and bots exchange messages - **Use `system_instruction` for system prompts** - set it in LLM Settings rather than as a context message for reliability across context updates and summarization - **Use developer messages for task instructions** - place task-specific guidance in context with `"role": "developer"` rather than overloading the system prompt - **Frame types matter** - `TranscriptionFrame` for users, `TTSTextFrame` for assistants - **Placement matters** - user aggregator after STT, assistant aggregator after transport output - **Tools are included** - function definitions and results are stored in context - **Manual control available** - use frames to append messages or trigger responses when needed - **Word-level precision** - proper placement ensures context accuracy during interruptions - **Automatic summarization** - enable context summarization to manage long conversations efficiently and reduce token costs ## What's Next Now that you understand context management, let's explore how to configure the LLM services that process this context to generate intelligent responses. Learn how to configure language models in your voice AI pipeline # LLM Inference Source: https://docs.pipecat.ai/pipecat/learn/llm.md Learn how to configure language models to generate intelligent responses in your voice AI pipeline **LLM services** are responsible for chat completions and tool calling based on the provided context (conversation history). The LLM responds by streaming tokens via `LLMTextFrame`s, which are used by subsequent processors to create audio output for the bot. ## Pipeline Placement The LLM instance should be placed after the user context aggregator and before any downstream services that depend on the LLM's output stream: ```python pipeline = Pipeline([ transport.input(), stt, # Creates TranscriptionFrames context_aggregator.user(), # Processes user context → creates LLMContextFrame llm, # Processes context → streams LLMTextFrames tts, # Processes LLMTextFrames → creates TTSAudioRawFrames transport.output(), context_aggregator.assistant(), ]) ``` **Frame flow:** - **Input**: Receives `LLMContextFrame` containing conversation history - **Processing**: - Analyzes context and generates streaming response - Handles function calls if tools are available - Tracks token usage for metrics - **Output**: - Denotes the start of the streaming response by pushing an `LLMFullResponseStartFrame` - Streams `LLMTextFrame`s containing response tokens to downstream processors (enables real-time TTS processing) - Ends with an `LLMFullResponseEndFrame` to mark the completion of the response - Output frames can be configured to [skip TTS](/pipecat/learn/text-to-speech#skipping-tts-output) via `LLMConfigureOutputFrame(skip_tts=True)`, allowing text to flow through the pipeline without being spoken - **Function calls:** - `FunctionCallsStartedFrame`: Indicates function execution beginning - `FunctionCallInProgressFrame`: Indicates a function is currently executing - `FunctionCallResultFrame`: Contains results from executed functions ## Supported LLM Services Pipecat supports a wide range of LLM providers to fit different needs, performance requirements, and budgets: ### Text-Based LLMs Most LLM services are built on the OpenAI chat completion specification for compatibility: GPT models with the original chat completion API Claude models with advanced reasoning capabilities Multimodal capabilities with competitive performance Enterprise-grade hosting for various foundation models **Compatible APIs**: Any OpenAI-spec compatible service can be used via the `base_url` parameter. ### Speech-to-Speech Models For lower latency, some providers offer direct speech-to-speech models: - **OpenAI Realtime**: Direct speech input/output with GPT models - **Gemini Live**: Real-time speech conversations with Gemini - **AWS Nova Sonic**: Speech-optimized models on Bedrock For a complete list of supported LLM services, see the Supported Services page: View the complete list of supported language model providers ## LLM Service Architecture ### BaseOpenAILLMService Many LLM services use the OpenAI chat completion specification. Pipecat provides a `BaseOpenAILLMService` that most providers extend, enabling easy switching between compatible services: ```python from pipecat.services.openai.llm import OpenAILLMService # Native OpenAI llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY")) # OpenAI-compatible service via base_url llm = OpenAILLMService( api_key=os.getenv("OTHER_API_KEY"), base_url="https://api.other-provider.com/v1" # Custom endpoint ) ``` This architecture allows you to quickly plug in different LLM services without changing your pipeline code. ## LLM Configuration ### Service-Specific Configuration Each LLM service has its own configuration options. For example, configuring OpenAI with various parameters: ```python from pipecat.services.openai.llm import OpenAILLMService llm = OpenAILLMService( api_key=os.getenv("OPENAI_API_KEY"), settings=OpenAILLMService.Settings( model="gpt-4.1", system_instruction="You are a helpful voice assistant.", temperature=0.7, # Response creativity (0.0-2.0) max_completion_tokens=150, # Maximum response length frequency_penalty=0.5, # Reduce repetition (0.0-2.0) presence_penalty=0.5, # Encourage topic diversity (0.0-2.0) ), ) ``` `system_instruction` defines the bot's personality and core behavior. For task-specific instructions (response format constraints, domain rules, workflow steps), use developer messages in context instead. See [Context Management](/pipecat/learn/context-management#system-instruction-developer-messages-and-system-messages) for how these interact. For detailed configuration options specific to each provider: Explore configuration options for each supported LLM provider ### Base Class Configuration All LLM services inherit from the `LLMService` base class with shared configuration options: ```python llm = YourLLMService( # Service-specific options... run_in_parallel=True, # Whether function calls run in parallel (default: True) ) ``` **Key options:** - **`run_in_parallel`**: Controls whether function calls execute simultaneously or sequentially - `True` (default): Faster execution when multiple functions are called - `False`: Sequential execution for dependent function calls ## Event Handlers LLM services provide event handlers for monitoring completion lifecycle: ```python @llm.event_handler("on_completion_timeout") async def on_completion_timeout(service): logger.warning("LLM completion timed out") # Handle timeout (retry, fallback, etc.) @llm.event_handler("on_function_calls_started") async def on_function_calls_started(service, function_calls): logger.info(f"Starting {len(function_calls)} function calls") # Optionally notify user that bot is "thinking" await tts.queue_frame(TTSSpeakFrame("Let me check on that.")) ``` **Available events:** - **`on_completion_timeout`**: Triggered when LLM requests timeout - **`on_function_calls_started`**: Triggered when function calls are initiated These handlers enable you to provide user feedback and implement error recovery strategies. ## Function Calling LLMs can call external functions to access real-time data and perform actions beyond their training data. This enables capabilities like checking weather, querying databases, or controlling external APIs. Function calls and their results are automatically stored in the conversation context by the context aggregator. Learn how to enable LLMs to interact with external services and APIs ## Key Takeaways - **Pipeline placement matters** - LLM goes after user context, before TTS - **Token streaming enables real-time responses** - no waiting for complete generation - **OpenAI compatibility** enables easy provider switching - **Function calling extends capabilities** beyond training data - **Configuration affects behavior** - tune temperature, penalties, and limits - **Services are modular** - swap providers without changing pipeline code ## What's Next Now that you understand LLM configuration, let's explore how function calling enables your bot to interact with external services and real-time data. Learn how to enable LLMs to interact with external services and APIs # Function Calling Source: https://docs.pipecat.ai/pipecat/learn/function-calling.md Enable LLMs to interact with external services and APIs in your voice AI pipeline **Function calling** (also known as tool calling) allows LLMs to request information from external services and APIs during conversations. This extends your voice AI bot's capabilities beyond its training data to access real-time information and perform actions. ## Pipeline Integration Function calling works seamlessly within your existing pipeline structure. The LLM service handles function calls automatically when they're needed: ```python pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), # Collects user transcriptions llm, # Processes context, calls functions when needed tts, transport.output(), context_aggregator.assistant(), # Collects function results and responses ]) ``` **Function call flow:** 1. User asks a question requiring external data 2. LLM recognizes the need and calls appropriate function 3. Your function handler executes and returns results 4. LLM incorporates results into its response 5. Response flows to TTS and user as normal **Context integration:** Function calls and their results are automatically stored in conversation context by the context aggregators, maintaining complete conversation history. ## Understanding Function Calling Function calling allows your bot to access real-time data and perform actions that aren't part of its training data. For example, you could give your bot the ability to: - Check current weather conditions - Look up stock prices - Query a database - Control smart home devices - Schedule appointments Here's how it works: 1. You define functions the LLM can use and make them available to the LLM service used in your pipeline 2. When needed, the LLM requests a function call 3. Your application executes any corresponding functions 4. The result is sent back to the LLM 5. The LLM uses this information in its response ## Implementation ### 1. Define a tool A tool needs two things: a handler — the code to run when the LLM calls the tool — and a schema that describes the tool to the LLM (its name, what it does, and its parameters) so the model knows it exists and how to call it. The preferred way to define a tool is with a **direct function**: a single async function that is _both_ the handler and the schema. Pipecat auto-derives the tool's metadata — name, description, parameter properties (with their descriptions), and which parameters are required — from the function's signature and docstring. The first parameter is always `params` (a `FunctionCallParams`); the tool's own arguments follow. Document each argument in a Google-style docstring. ```python from pipecat.services.llm_service import FunctionCallParams async def get_current_weather(params: FunctionCallParams, location: str, format: str): """Get the current weather. Args: location: The city and state, e.g. "San Francisco, CA". format: The temperature unit to use. Must be either "celsius" or "fahrenheit". Infer this from the user's location. """ weather_data = {"conditions": "sunny", "temperature": "75"} await params.result_callback(weather_data) ``` The direct-function schema generator doesn't yet map `Literal` types to a JSON-schema `enum`. Express enum-like constraints in the docstring prose instead (e.g. _'Must be either "celsius" or "fahrenheit"'_), as shown above. If you need a strict `enum` in the schema, use the [verbose `FunctionSchema`](#advanced-defining-tools-with-functionschema) pattern. ### 2. Add the tool to the context List your direct functions in `LLMContext(tools=[...])`: ```python from pipecat.processors.aggregators.llm_context import LLMContext from pipecat.processors.aggregators.llm_response_universal import LLMContextAggregatorPair context = LLMContext(tools=[get_current_weather, get_restaurant_recommendation]) user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context) ``` The bot's personality (e.g. "You are a helpful assistant") is set via [`system_instruction`](/pipecat/learn/context-management#using-system_instruction-recommended) in the LLM service's Settings, not as a context message. Tools are automatically converted to the correct format for your LLM provider through adapters. ### 3. Create the pipeline Include your LLM service in the pipeline: ```python # Create the pipeline pipeline = Pipeline([ transport.input(), # Input from the transport stt, # STT processing user_aggregator, # User context aggregation llm, # LLM processing tts, # TTS processing transport.output(), # Output to the transport assistant_aggregator, # Assistant context aggregation ]) ``` ## Per-Tool Options with `@tool_options` By default, a direct function is cancelled if the user interrupts, and it uses the LLM service's global timeout. To override either, decorate the function with `@tool_options`. The decorator only attaches call options — the schema is still auto-derived — so decorated functions can stay at module level. ```python import asyncio from pipecat.adapters.schemas.direct_function import tool_options from pipecat.services.llm_service import FunctionCallParams @tool_options(cancel_on_interruption=False, timeout_secs=30) async def get_current_weather(params: FunctionCallParams, location: str, format: str): """Get the current weather. Args: location: The city and state, e.g. "San Francisco, CA". format: The temperature unit to use. Must be either "celsius" or "fahrenheit". Infer this from the user's location. """ # Simulate a long-running API call. await asyncio.sleep(20) await params.result_callback({"conditions": "nice", "temperature": "75"}) ``` **Options:** - **`cancel_on_interruption`** (default `True`): When `True`, the call is cancelled if the user interrupts. When `False`, the call is treated as **asynchronous** — see below. - **`timeout_secs`** (default `None`): Per-tool timeout in seconds. Overrides the global `function_call_timeout_secs` for this function. Use a longer timeout for slow operations (e.g. database queries) or a shorter one for quick lookups. `@tool_options` also sets call options on the handler of a [`FunctionSchema`](#advanced-defining-tools-with-functionschema) tool, not just a direct function. On an [`LLMWorker`](/api-reference/server/workers/llm-context-worker), mark tool methods with `@tool` instead. It applies the same options _and_ marks the method for automatic collection as one of the worker's own tools. ### Synchronous vs. asynchronous calls With `cancel_on_interruption=True` (the default), the call is **synchronous**: the LLM waits for the result before generating its next response. This ensures the LLM has complete information before responding. With `cancel_on_interruption=False`, the call is **asynchronous**: the LLM continues the conversation immediately without waiting. Once the result returns, it's injected back into the context as a developer message, triggering a new LLM inference at that point. This enables truly non-blocking calls where the conversation proceeds while the function runs in the background. Async calls can also send [intermediate updates](#intermediate-results-for-async-functions) before their final result. #### Async function call cancellation For async functions (`cancel_on_interruption=False`), you can also enable model-directed cancellation: ```python llm = OpenAILLMService( api_key="your-api-key", enable_async_tool_cancellation=True, ) ``` When `enable_async_tool_cancellation=True` and at least one async function is available, Pipecat automatically adds the built-in `cancel_async_tool_call` tool and supporting system instructions. The LLM can call that tool to cancel a stale in-progress async function call — for example, when the user changes their request before a long-running lookup completes. ## Parallel and Multiple Tool Calls When the LLM calls more than one tool in a single turn, two `LLMService` constructor options control how those calls run and when the LLM responds: ```python llm = OpenAILLMService( api_key="your-api-key", run_in_parallel=True, # default group_parallel_tools=True, # default ) ``` Whether multiple tool calls in one turn run in parallel or one after another. When `True`, all tool calls in a batch are grouped so the LLM is triggered **exactly once** after every call in the batch completes. When `False`, each function call result triggers the LLM independently as it arrives. If multiple tools firing in one turn produce **duplicate or repeated responses** (the bot answers once per tool instead of once total), check that `group_parallel_tools` is `True`. With it disabled, each result re-triggers the LLM, so the model responds once for every tool call in the batch. ## Changing Tools Mid-Conversation To change the set of tools the LLM can use during a session, push an `LLMSetToolsFrame`. Its `tools` field takes the same things as `LLMContext(tools=[...])` — a list of direct functions and/or `FunctionSchema` objects. Whatever you pass becomes the LLM's new tool set. ```python from pipecat.frames.frames import LLMSetToolsFrame from pipecat.processors.aggregators.llm_context import NOT_GIVEN # Make get_current_weather the only tool the LLM can call await worker.queue_frame(LLMSetToolsFrame(tools=[get_current_weather])) # Clear all tools await worker.queue_frame(LLMSetToolsFrame(tools=NOT_GIVEN)) ``` ## Tools Across Service Switches When you use an [`LLMSwitcher`](/api-reference/server/utilities/service-switchers/llm-switcher) to swap LLM providers mid-session, the tools you list in `LLMContext(tools=[...])` are available on whichever provider is active. You define them once for the whole switcher. ```python from pipecat.pipeline.llm_switcher import LLMSwitcher llm_switcher = LLMSwitcher(llms=[llm_openai, llm_google]) # Tools in the context are available on whichever provider is active context = LLMContext(tools=[get_current_weather, get_restaurant_recommendation]) user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context) ``` ## Advanced: Defining Tools with `FunctionSchema` Direct functions cover most cases. Reach for the verbose `FunctionSchema` pattern when you need explicit control over the schema — for example, a strict `enum` constraint (which the direct-function generator doesn't yet emit) — or when the tool's handler isn't shaped like a direct function. A `FunctionSchema` spells out the tool's name, description, and parameters by hand. Pass the `handler` that runs when the LLM calls the tool as the schema's `handler`, then list the schema in `LLMContext(tools=[...])` — exactly as you would a direct function. ```python from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.services.llm_service import FunctionCallParams async def fetch_weather_from_api(params: FunctionCallParams): weather_data = {"conditions": "sunny", "temperature": "75"} await params.result_callback(weather_data) weather_function = FunctionSchema( name="get_current_weather", description="Get the current weather in a location", properties={ "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA", }, "format": { "type": "string", # A strict enum — the kind of explicit control a direct function can't yet express. "enum": ["celsius", "fahrenheit"], "description": "The temperature unit to use.", }, }, required=["location", "format"], handler=fetch_weather_from_api, # bundle the handler on the schema ) # List the schema in the context, just like a direct function. context = LLMContext(tools=[weather_function]) ``` To override the handler's default call options, decorate it with [`@tool_options`](#per-tool-options-with-@tool_options) — the same decorator direct functions use, with the same [synchronous vs. asynchronous](#synchronous-vs-asynchronous-calls) semantics: ```python from pipecat.adapters.schemas.direct_function import tool_options @tool_options(cancel_on_interruption=False, timeout_secs=30) async def fetch_weather_from_api(params: FunctionCallParams): ... ``` These schemas behave just like direct functions everywhere else in this guide — swap them mid-conversation with an `LLMSetToolsFrame`, and they keep working across an `LLMSwitcher`'s providers. ### Registering a handler manually Bundling the handler on the schema (above) is the recommended approach. If you'd rather keep the handler separate, list a handler-free `FunctionSchema` in the context as usual and register its handler by name: ```python # weather_function here is the same schema, just defined without handler=. context = LLMContext(tools=[weather_function]) llm.register_function("get_current_weather", fetch_weather_from_api) ``` If the handler carries [`@tool_options`](#per-tool-options-with-@tool_options), `register_function` honors it the same way bundling does — or pass `cancel_on_interruption` / `timeout_secs` to `register_function` directly to override. To remove the tool, un-advertise it with an [`LLMSetToolsFrame`](#changing-tools-mid-conversation); call `llm.unregister_function(...)` only afterward, since unregistering a still-advertised tool leaves the LLM able to call a handler that's no longer there. This is uncommon — bundling keeps a tool and its handler together — but the option is there when you need to manage registration directly. ### Provider-Specific Custom Tools For normal function calling, prefer `standard_tools` with `FunctionSchema` or direct functions so Pipecat can convert them to each provider's native format. When a provider has tools that don't fit Pipecat's standard function schema, add those provider-native definitions through `ToolsSchema.custom_tools`. These custom tools are passed only to the matching adapter and are appended to the converted standard tools. ```python OpenAI-family adapter from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import AdapterType, ToolsSchema # Standard function converted by Pipecat weather_function = FunctionSchema( name="get_current_weather", description="Get the current weather", properties={"location": {"type": "string"}}, required=["location"], ) # Provider-native tool appended only for OpenAI-family adapters. # This object must match the target OpenAI API you are using. provider_tool = {"type": "tool_search"} tools = ToolsSchema( standard_tools=[weather_function], custom_tools={AdapterType.OPENAI: [provider_tool]}, ) ``` ```python Gemini from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import AdapterType, ToolsSchema # Standard function converted by Pipecat weather_function = FunctionSchema( name="get_current_weather", description="Get the current weather", properties={"location": {"type": "string"}}, required=["location"], ) # Provider-native tool appended only for Gemini adapters. # This object must match the Gemini API you are using. gemini_search_tool = {"google_search": {}} tools = ToolsSchema( standard_tools=[weather_function], custom_tools={AdapterType.GEMINI: [gemini_search_tool]}, ) ``` Raw provider-native tool lists are not the normal `LLMContext` path. Some lower-level adapter code still preserves non-`ToolsSchema` tools for legacy or direct provider-specific paths, but `LLMContext(tools=...)` validates tools as a `ToolsSchema`. Use `custom_tools` as the provider-specific escape hatch while staying in the universal context flow. For normal callable functions, use direct functions or `FunctionSchema` instead of provider-native function definitions. Today, `custom_tools` is supported for OpenAI-family adapters and Gemini. Anthropic standard functions should be represented with `FunctionSchema`. ## Function Handler Details ### FunctionCallParams Every function handler receives a `FunctionCallParams` object containing all the information needed for execution: ```python @dataclass class FunctionCallParams: function_name: str # Name of the called function tool_call_id: str # Unique identifier for this call arguments: Mapping[str, Any] # Arguments from the LLM llm: LLMService # Reference to the LLM service context: LLMContext # Current conversation context result_callback: FunctionCallResultCallback # Return results here app_resources: Any # Application-defined resources shared across tool calls ``` **Using the parameters:** ```python async def example_function_handler(params: FunctionCallParams): # Access function details print(f"Called function: {params.function_name}") print(f"Call ID: {params.tool_call_id}") # Extract arguments location = params.arguments["location"] # Access LLM context for conversation history messages = params.context.messages # Access shared resources (database, API clients, etc.) if params.app_resources: db = params.app_resources.database user_id = params.app_resources.current_user_id # Use LLM service for additional operations await params.llm.push_frame(TTSSpeakFrame("Looking up weather data...")) # Return results await params.result_callback({"conditions": "nice", "temperature": "75"}) ``` See the [API reference](https://reference-server.pipecat.ai/en/latest/api/pipecat.services.llm_service.html#pipecat.services.llm_service.FunctionCallParams) for complete details. `params.tool_resources` is a deprecated alias for `params.app_resources`. Use `app_resources` in new code. ### Handler Structure Your function handler should: 1. Receive necessary arguments, either: - From `params.arguments` - Directly from function arguments, if using [direct functions](#1-define-a-tool) 2. Process data or call external services 3. Return results via `params.result_callback(result)` ```python Non-Direct Function async def fetch_weather_from_api(params: FunctionCallParams): try: # Extract arguments location = params.arguments.get("location") format_type = params.arguments.get("format", "celsius") # Call external API api_result = await weather_api.get_weather(location, format_type) # Return formatted result await params.result_callback({ "location": location, "temperature": api_result["temp"], "conditions": api_result["conditions"], "unit": format_type }) except Exception as e: # Handle errors await params.result_callback({ "error": f"Failed to get weather: {str(e)}" }) ``` ```python Direct Function async def get_current_weather(params: FunctionCallParams, location: str, format: str): """Get the current weather. Args: location: The city and state, e.g. "San Francisco, CA". format: The temperature unit to use. Must be either "celsius" or "fahrenheit". """ try: # Call external API api_result = await weather_api.get_weather(location, format) # Return formatted result await params.result_callback({ "location": location, "temperature": api_result["temp"], "conditions": api_result["conditions"], "unit": format }) except Exception as e: # Handle errors await params.result_callback({ "error": f"Failed to get weather: {str(e)}" }) ``` ### Sharing Resources with app_resources When function handlers need access to shared resources like database connections, API clients, or application state, you can pass them via `app_resources` when creating the `PipelineWorker`. These resources are then accessible in every function handler via `params.app_resources`. ```python from dataclasses import dataclass from pipecat.pipeline.worker import PipelineWorker from pipecat.services.llm_service import FunctionCallParams # Define your application resources @dataclass class AppResources: database: DatabaseConnection api_client: WeatherAPIClient user_id: str # Create your resources resources = AppResources( database=db_connection, api_client=weather_client, user_id="user-123" ) # Pass resources to the pipeline worker worker = PipelineWorker( pipeline, app_resources=resources ) # Access resources in function handlers async def query_user_preferences(params: FunctionCallParams): # Access shared resources db = params.app_resources.database user_id = params.app_resources.user_id # Query database with shared connection prefs = await db.query("SELECT * FROM preferences WHERE user_id = ?", user_id) await params.result_callback(prefs) ``` **Key points:** - Resources are **passed by reference** — the caller retains their handle and can read mutations after the task finishes - The framework **never copies or clears** the `app_resources` object - All function handlers in the pipeline share the same `app_resources` instance - Useful for database connections, API clients, caches, or any shared state `PipelineWorker(tool_resources=...)` and `FunctionCallParams.tool_resources` are deprecated aliases retained for compatibility. Prefer `PipelineWorker(app_resources=...)` and `params.app_resources`. ## Advanced: Controlling Function Call Behavior When returning results from a function handler, you can control how the LLM processes those results using a `FunctionCallResultProperties` object passed to the result callback. ### Properties `FunctionCallResultProperties` provides fine-grained control over LLM execution: ```python @dataclass class FunctionCallResultProperties: run_llm: bool | None = None # Whether to run LLM after this result on_context_updated: Callable | None = None # Callback when context is updated is_final: bool = True # Whether this is the final result ``` **Property options:** - **`run_llm=True`**: Run LLM after function call (default behavior) - **`run_llm=False`**: Don't run LLM after function call (useful for chained calls) - **`on_context_updated`**: Async callback executed after the function result is added to context - **`is_final=False`**: Treat this as an intermediate result for an async function call. Only use this for async functions (`cancel_on_interruption=False`) Skip LLM execution (`run_llm=False`) when you have back-to-back function calls. If you skip a completion, you must manually trigger one from the context aggregator. See the [API reference](https://reference-server.pipecat.ai/en/latest/api/pipecat.frames.frames.html#pipecat.frames.frames.FunctionCallResultProperties) for complete details. ### Example Usage ```python from pipecat.frames.frames import FunctionCallResultProperties from pipecat.services.llm_service import FunctionCallParams async def fetch_weather_from_api(params: FunctionCallParams): # Fetch weather data weather_data = {"conditions": "sunny", "temperature": "75"} # Don't run LLM after this function call properties = FunctionCallResultProperties(run_llm=False) await params.result_callback(weather_data, properties=properties) async def query_database(params: FunctionCallParams): # Query database results = await db.query(params.arguments["query"]) async def on_update(): await notify_system("Database query complete") # Run LLM after function call and notify when context is updated properties = FunctionCallResultProperties( run_llm=True, on_context_updated=on_update ) await params.result_callback(results, properties=properties) ``` ### Intermediate Results for Async Functions Async function calls can send progress updates before their final result. Make the function async with `@tool_options(cancel_on_interruption=False)`, then call `params.result_callback(..., properties=FunctionCallResultProperties(is_final=False))` for each intermediate update. Finish with a normal `params.result_callback(...)`. ```python from pipecat.adapters.schemas.direct_function import tool_options from pipecat.frames.frames import FunctionCallResultProperties from pipecat.services.llm_service import FunctionCallParams @tool_options(cancel_on_interruption=False) async def track_delivery(params: FunctionCallParams): """Track a delivery, reporting each status update until it arrives.""" await params.result_callback( {"status": "picked_up"}, properties=FunctionCallResultProperties(is_final=False), ) await params.result_callback( {"status": "nearby"}, properties=FunctionCallResultProperties(is_final=False), ) await params.result_callback({"status": "delivered"}) ``` Intermediate results are injected into the LLM context as async-tool developer messages. They do not close the function call; the call remains in progress until the final result is sent. ## Key Takeaways - **Function calling extends LLM capabilities** beyond training data to real-time information - **Context integration is automatic** - function calls and results are stored in conversation history - **Direct functions are the preferred approach** - one async function is both schema and handler; list it in `LLMContext(tools=[...])` or add it via `LLMSetToolsFrame` to make it available. When you need explicit schema control, use a `FunctionSchema` with its `handler` bundled in - **Async function calls are opt-in** - set `cancel_on_interruption=False` for deferred results, intermediate updates, and optional async-tool cancellation - **Pipeline integration is seamless** - functions work within your existing voice AI architecture - **Advanced control available** - fine-tune LLM execution and monitor function call lifecycle ## What's Next Now that you understand function calling, let's explore how to configure text-to-speech services to convert your LLM's responses (including function call results) into natural-sounding speech. Learn how to configure speech synthesis in your voice AI pipeline # Text to Speech Source: https://docs.pipecat.ai/pipecat/learn/text-to-speech.md Learn how to configure speech synthesis to convert text into natural-sounding audio in your voice AI pipeline **Text to Speech (TTS)** services are responsible for converting text into natural-sounding speech audio. They receive text input from LLMs and other sources, then generate audio output that users can hear through their connected devices. ## Pipeline Placement TTS processors must be positioned correctly in your pipeline to receive text and generate audio frames: ```python pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, # Generates LLMTextFrames tts, # Processes text → creates TTSAudioRawFrames transport.output(), # Sends audio to user context_aggregator.assistant(), # Processes TTSTextFrames for context ]) ``` **Placement requirements:** - **After LLM processing**: TTS needs `LLMTextFrame`s from language model responses - **Before transport output**: Audio must be generated before sending to user - **Before assistant context aggregator**: Ensures spoken text is captured in conversation history ### Frame Processing Flow **TTS generates speech through two primary mechanisms:** 1. **Streamed LLM tokens** via `LLMTextFrame`s: - By default, TTS aggregates streaming tokens into complete sentences before synthesis (`TextAggregationMode.SENTENCE`) - Set `text_aggregation_mode=TextAggregationMode.TOKEN` to stream tokens directly for lower latency - Audio bytes stream back and play immediately - End-to-end latency often under 200ms 2. **Direct speech requests** via `TTSSpeakFrame`s: - Bypasses LLM for immediate audio generation - Optionally appends text to conversation context via `append_to_context` parameter - Useful for developer messages, greetings, or injected speech **Frame output:** - `TTSAudioRawFrame`s: Raw audio data for playback - `TTSTextFrame`s: Text that was actually spoken (for context updates) - `TTSStartedFrame`/`TTSStoppedFrame`: Speech boundary markers ## Supported TTS Services Pipecat supports a wide range of TTS providers with different capabilities and performance characteristics: View the complete list of supported text-to-speech providers ### Service Categories **WebSocket-Based Services (Recommended):** - **Cartesia**: Ultra-low latency with word timestamps - **ElevenLabs**: High-quality voices with emotion control - **Rime**: Ultra-realistic voices with advanced features **HTTP-Based Services:** - **OpenAI TTS**: High-quality synthesis with multiple voices - **Azure Speech**: Enterprise-grade with extensive language support - **Google Text-to-Speech**: Reliable with WaveNet voices **Advanced Features:** - **Word timestamps**: Enable word-level accuracy for context and subtitles - **Voice cloning**: Custom voice creation from samples - **Emotion control**: Dynamic emotional expression - **SSML support**: Fine-grained pronunciation control WebSocket services typically provide the lowest latency, while HTTP services may have intermittent higher latency due to their request/response nature. ## TTS Configuration ### Service-Specific Configuration Each TTS service has its own configuration options. Here's an example with Cartesia: ```python from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.transcriptions.language import Language tts = CartesiaTTSService( api_key=os.getenv("CARTESIA_API_KEY"), settings=CartesiaTTSService.Settings( model="sonic-3.5", voice="voice-id-here", language=Language.EN, # Speech language ), # Word timestamps automatically enabled for precise context updates ) ``` **Word timestamps:** Services like Cartesia, ElevenLabs, and Rime provide word-level timestamps that enable precise context updates during interruptions and better synchronization with other pipeline components. For example, if an interruption occurs while the bot is speaking, the word timestamps allow you to accurately capture which words were spoken up to that point, enabling better context management and user experience. Additionally, transcription events streamed from server to client can be done in sync with the audio output, allowing for real-time subtitles or captions. Explore configuration options for each supported TTS provider ### Pipeline-Level Audio Configuration Set consistent audio settings across your entire pipeline: ```python worker = PipelineWorker( pipeline, params=PipelineParams( audio_in_sample_rate=16000, # Input audio quality audio_out_sample_rate=24000, # Output audio quality (TTS) ), ) ``` Set the `audio_out_sample_rate` to match your TTS service's requirements for optimal quality. This is preferred to setting the sample_rate directly in the TTS service as the PipelineParam ensures that all output sample_rates match. ## Text Processing and Filtering ### Custom Text Aggregation By default, TTS services have a built-in text aggregator that collects streaming text into sentences before passing them to the underlying service. However, you can customize this behavior by inserting an [`LLMTextProcessor`](/api-reference/server/utilities/frame/llm-text-processor) with a different text aggregator before the TTS in your pipeline. This allows the ability to categorize and structure text into logical units beyond simple sentences, such as code blocks, URLs, or custom tags. You can then configure the TTS to handle these different text types appropriately, such as skipping code blocks or transforming them in a just-in-time manner before speaking. ### Skipping Text Aggregations To skip certain text aggregations (e.g., code snippets or URLs) and keep them from being spoken, use a custom text aggregator like [`PatternPairAggregator`](/api-reference/server/utilities/text/pattern-pair-aggregator) within an [`LLMTextProcessor`](/api-reference/server/utilities/frame/llm-text-processor), and configure it to identify and handle specific patterns in the text stream. With this, you can then pass any aggregated types you want to skip (like "code") to the TTS service's `skip_aggregator_types` parameter. ```python # Create pattern aggregator pattern_aggregator = PatternPairAggregator() # Add pattern for JSON data pattern_aggregator.add_pattern( type="code", start_pattern="", end_pattern="", action=MatchAction.AGGREGATE ) # Set the aggregator on an LLMTextProcessor llm_text_processor = LLMTextProcessor(text_aggregator=pattern_aggregator) # Initialize TTS service, and don't speak JSON data tts = CartesiaTTSService( api_key=os.getenv("CARTESIA_API_KEY"), skip_aggregator_types=["code"], # The strings here should match the types defined in the PatternPairAggregator ) # add the llm_text_processor to your pipeline after the llm and before the tts # llm -> llm_text_processor -> tts ``` ### Text Transforms For TTS-specific text preprocessing, you can provide custom text transforms that modify text in a just-in-time manner before sending the text off to the TTS service. This is useful for handling special text segments that need to be altered for better pronunciation or clarity, such as spelling out phone numbers, removing URLs, or expanding abbreviations. These text transforms can be mapped to a specific text aggregation type, like with `skip_aggregator_types`, or applied globally to all text using `'*'` as the type. Text transforms are registered directly on the TTS service instance via the `add_text_transformer()` method or during initialization using the `text_transforms` parameter. The intentions of text transforms are meant to be TTS-specific modifications that do not affect the underlying LLM text or context. That said, since the context aggregator attempts to base its context on what was actually spoken, for services that support word timestamps, like Cartesia, ElevenLabs, and Rime,these transforms will modify the context as they modify what is spoken. ```python # Create pattern aggregator pattern_aggregator = PatternPairAggregator() # Add patterns for different parts of an explanation pattern_aggregator.add_pattern( type="phone_number", start_pattern="", end_pattern="", action=MatchAction.AGGREGATE ) # Set the aggregator on an LLMTextProcessor llm_text_processor = LLMTextProcessor(text_aggregator=pattern_aggregator) # Text-to-Speech service tts = CartesiaTTSService( api_key=os.getenv("CARTESIA_API_KEY"), ) # Text transformers for TTS # This will insert Cartesia's spell tags around the provided text. async def spell_out_text(text: str, type: str) -> str: # CartesiaTTSService provides a helper for this along with other common transforms return CartesiaTTSService.SPELL(text) async def replace_acronyms(text: str, type: str) -> str: # Replace "SEC" with "Southeastern Conference" return text.replace(" SEC ", " Southeastern Conference ") # Setup the text transformers in TTS to spell out phone numbers and replace # acronyms. The string below matches the type defined in the PatternPairAggregator # above so that whenever those segments are encountered, this transform # is applied tts.add_text_transformer(spell_out_text, "phone_number") tts.add_text_transformer(replace_acronyms, "*") # Apply to all text # add the llm_text_processor to your pipeline after the llm and before the tts # llm -> llm_text_processor -> tts ``` ### Text Filters Text filters are no longer the preferred method for text preprocessing and will be deprecated in future releases. Instead, you should use one of the methods described above. Apply preprocessing to text before synthesis: ```python from pipecat.utils.text.markdown_text_filter import MarkdownTextFilter tts = YourTTSService( # ... other options text_filters=[ MarkdownTextFilter(), # Remove markdown formatting CustomTextFilter(), # Your custom processing ], ) ``` **Common filters:** - **MarkdownTextFilter**: Strips markdown formatting from LLM responses - **Custom filters**: Implement your own text preprocessing logic ## Skipping TTS Output Sometimes you want text from the LLM to flow through the pipeline—updating the conversation context, reaching observers, or being processed by custom frame processors—without being spoken by the TTS service. Pipecat provides a `skip_tts` attribute on text and response frames for this purpose. When `skip_tts` is `True` on a frame, the TTS service passes it through without generating audio, but the text still reaches downstream processors like the assistant context aggregator. ### Configuring All LLM Output Use `LLMConfigureOutputFrame` to tell the LLM service to mark **all** subsequent output frames (`LLMTextFrame`, `LLMFullResponseStartFrame`, `LLMFullResponseEndFrame`) with `skip_tts`: ```python from pipecat.frames.frames import LLMConfigureOutputFrame # Tell the LLM to skip TTS for all output await worker.queue_frame(LLMConfigureOutputFrame(skip_tts=True)) # ... LLM responses will not be spoken ... # Re-enable TTS await worker.queue_frame(LLMConfigureOutputFrame(skip_tts=False)) ``` This is useful when you want to toggle TTS on or off for an entire stretch of conversation, such as switching between voice and text input modes. ### Setting skip_tts on Individual Frames For more granular control, set `skip_tts=True` directly on individual text frames. This is useful when building custom frame processors that selectively silence certain parts of the LLM output: ```python from pipecat.frames.frames import LLMTextFrame # In a custom frame processor frame = LLMTextFrame(text) frame.skip_tts = True await self.push_frame(frame) ``` The `skip_tts` attribute is available on `TextFrame` and all its subclasses (`LLMTextFrame`, `AggregatedTextFrame`, `TTSTextFrame`, etc.), as well as `LLMFullResponseStartFrame` and `LLMFullResponseEndFrame`. ### Common Use Cases **Encoding structured output from the LLM.** You can instruct the LLM to include markers or metadata in its response that should be processed by pipeline logic but not spoken. For example, Pipecat's [turn completion detection](/api-reference/server/utilities/turn-management/filter-incomplete-turns) uses this approach — the LLM outputs completion markers (`✓`, `○`, `◐`) that are pushed with `skip_tts=True` so they update the context but aren't spoken. **Switching between voice and text input.** When a client sends text input instead of speech, you may want the bot to respond with text only. The client SDKs support this via `sendText()` with `audio_response: false`, which uses `LLMConfigureOutputFrame` internally to temporarily disable TTS for that response. **Testing without audio.** When building test pipelines, you can use `LLMConfigureOutputFrame(skip_tts=True)` to bypass audio generation entirely while still exercising the rest of the pipeline. ## Advanced TTS Features ### Direct Speech Commands Use `TTSSpeakFrame` for immediate speech synthesis: ```python from pipecat.frames.frames import TTSSpeakFrame # Make bot speak directly (added to context by default) await tts.queue_frame(TTSSpeakFrame("Hello, how can I help you?")) # Explicitly append spoken text to conversation context await tts.queue_frame( TTSSpeakFrame("Welcome! Let's begin.", append_to_context=True) ) # Speak without adding to context await tts.queue_frame( TTSSpeakFrame("Processing...", append_to_context=False) ) ``` The `append_to_context` parameter controls whether the spoken text is added to the conversation history. When `append_to_context=True`, the text is automatically committed to the context after being spoken, making it useful for bot greetings and injected speech that should be part of the conversation flow. As of Pipecat v1.4.0, `append_to_context` defaults to `True`. A plain `TTSSpeakFrame("...")` **is** added to the conversation context after it is spoken; pass `append_to_context=False` to speak without recording it. (`None` was the previous default and is no longer supported.) ### Dynamic Settings Updates Update TTS settings during conversation using typed settings objects: ```python from pipecat.frames.frames import TTSUpdateSettingsFrame from pipecat.services.cartesia.tts import CartesiaTTSSettings # Change voice speed during conversation await worker.queue_frames([ TTSUpdateSettingsFrame(delta=CartesiaTTSSettings(speed="fast")), TTSSpeakFrame("I'm speaking faster now!") ]) ``` ## Key Takeaways - **Pipeline placement matters** - TTS must come after LLM, before transport output - **Service types differ** - WebSocket services provide lower latency than HTTP - **Text processing affects quality** - use aggregation and filters for better results - **Word timestamps enable precision** - better interruption handling and context accuracy - **Configuration impacts performance** - balance quality, latency, and bandwidth needs - **Services are modular** - easily swap providers without changing pipeline code ## What's Next You've now learned how to build a complete voice AI pipeline! Let's explore some additional topics to enhance your implementation. Learn how to terminate your voice AI pipeline at the end of a conversation # Pipeline Termination Source: https://docs.pipecat.ai/pipecat/learn/pipeline-termination.md Learn how to properly terminate Pipecat pipelines for clean shutdown and resource management **Pipeline termination** ensures your voice AI applications shut down cleanly without resource leaks or hanging processes. Understanding the different termination methods helps you handle various scenarios from natural conversation endings to unexpected disconnections. ## Pipeline Integration Pipeline termination works through the same frame-based system as other pipeline operations: ```python pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ]) # EndFrame or CancelFrame flows through entire pipeline for shutdown ``` **Termination frames:** - **`EndFrame`**: A queued ControlFrame that triggers graceful shutdown after processing pending frames - **`CancelFrame`**: A SystemFrame that triggers immediate shutdown, discarding pending frames Both frames flow downstream through the pipeline, allowing each processor to clean up resources appropriately. ### Termination frames at a glance | Frame | Job | Push from / direction | | ------------------- | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | `EndFrame` | Graceful shutdown (drains pending frames) | From outside the pipeline via `worker.queue_frame(EndFrame())` | | `CancelFrame` | Immediate shutdown (discards pending frames) | Via `worker.cancel()` | | `EndWorkerFrame` | Graceful shutdown signal from inside the pipeline | `push_frame(EndWorkerFrame(), FrameDirection.DOWNSTREAM)`; the source converts it to an `EndFrame` | | `CancelWorkerFrame` | Immediate shutdown signal from inside the pipeline | `push_frame(CancelWorkerFrame(), FrameDirection.DOWNSTREAM)`; the source converts it to a `CancelFrame` | If you see `EndTaskFrame`, `CancelTaskFrame`, `PipelineTask`, or `task.cancel()` in older code or examples, those are **deprecated aliases** (since 1.3.0/1.4.0) of `EndWorkerFrame`, `CancelWorkerFrame`, `PipelineWorker`, and `worker.cancel()`. They still work but will be removed in 2.0.0. Use the `Worker` names in new code. ## Termination Methods Pipecat provides two primary approaches for pipeline termination, each designed for different scenarios: ### 1. Graceful Termination Graceful termination allows the bot to complete its current processing before shutting down. This is ideal when you want the bot to properly end a conversation. For example, after completing a specific task or reaching a natural conclusion. **When to use:** - Natural conversation endings - Task completion scenarios - When the bot should say goodbye **Implementation options:** Push an `EndFrame` from outside your pipeline: ```python # From outside the pipeline from pipecat.frames.frames import EndFrame await worker.queue_frame(EndFrame()) ``` Push an `EndWorkerFrame` downstream from inside your pipeline. Here `end_conversation` is a **direct function** — an `async` function whose first parameter is `params: FunctionCallParams`, with a Google-style docstring that becomes the tool description. Register it by passing the function itself in your tools list: ```python from pipecat.frames.frames import EndWorkerFrame, TTSSpeakFrame from pipecat.processors.aggregators.llm_context import LLMContext from pipecat.processors.frame_processor import FrameDirection from pipecat.services.llm_service import FunctionCallParams async def end_conversation(params: FunctionCallParams): """End the conversation and shut down the bot. Call this when the user says goodbye or the task is complete. """ await params.llm.push_frame(TTSSpeakFrame("Have a nice day!")) # Resolve the function call so the LLM call doesn't hang await params.result_callback({"status": "ended"}) # Signal that the worker should end after processing this frame await params.llm.push_frame(EndWorkerFrame(), FrameDirection.DOWNSTREAM) # Pass the function directly in the tools list; it's registered automatically context = LLMContext(tools=[end_conversation]) ``` Always call `params.result_callback(...)` in your handler before pushing the end frame. Skipping it can leave the LLM function call unresolved. If you don't want the LLM to respond, you can provide `None` as the result. **How graceful termination works:** 1. `EndFrame` is queued and processes after any pending frames (like goodbye messages) 2. All processors shutdown when they receive the `EndFrame` 3. Once the `EndFrame` reaches the end of the pipeline, shutdown is complete 4. Resources are cleaned up and the process terminates Graceful termination allows your bot to say goodbye and complete any final actions before terminating. ### 2. Immediate Termination Immediate termination cancels the pipeline without waiting for pending frames to complete. This is appropriate when the user is no longer active in the conversation. **When to use:** - User disconnections (browser closed, call ended) - Error conditions requiring immediate shutdown - When completing the conversation is no longer necessary **Implementation:** Use event handlers to detect disconnections and trigger cancellation: ```python @transport.event_handler("on_client_disconnected") async def on_client_disconnected(transport, client): logger.info("Client disconnected - terminating pipeline") await worker.cancel() ``` **How immediate termination works:** 1. An event triggers the cancellation (like client disconnection) 2. `worker.cancel()` pushes a `CancelFrame` downstream from the PipelineWorker 3. `CancelFrame`s are `SystemFrame`s, so they use the high-priority input queue and are processed before queued non-system frames 4. Processors handle the `CancelFrame` and shut down without waiting for pending non-system work to drain 5. Any pending frames are discarded during shutdown Immediate termination will discard any pending frames in the pipeline. Use this approach when completing the conversation is no longer necessary. ## Automatic Termination ### Pipeline Idle Detection Pipecat includes automatic idle detection to prevent hanging pipelines. This feature monitors activity and can automatically cancel tasks when no meaningful bot interactions occur for an extended period. **How it works:** - Monitors pipeline activity for meaningful bot interactions - Automatically triggers termination after configured idle timeout - Serves as a safety net for anomalous behavior or forgotten sessions **Configuration:** ```python worker = PipelineWorker( pipeline, # Configure idle detection timeout cancel_on_idle_timeout=True, # Default is True idle_timeout_secs=600, # Default is 300 seconds idle_timeout_frames=(BotSpeakingFrame,), # Default is (BotSpeakingFrame, UserSpeakingFrame) ) ``` You can further configure the idle detection behavior. To learn more, refer to the Pipeline Idle Detection documentation: Learn how to configure and customize idle detection for your use case Pipeline Idle Detection is enabled by default and helps prevent resources from being wasted on inactive conversations. ### Maximum Call Duration Idle detection ends a call that goes quiet, but it does not cap the total length of an active call. To enforce a maximum call duration, run an `asyncio` timer that speaks a goodbye and then queues an `EndFrame` so the bot can sign off gracefully before shutdown: ```python import asyncio from pipecat.frames.frames import EndFrame, TTSSpeakFrame async def end_after(worker, timeout_secs: float): await asyncio.sleep(timeout_secs) await worker.queue_frame(TTSSpeakFrame("We've reached our time limit. Goodbye!")) await worker.queue_frame(EndFrame()) # Graceful: plays the goodbye, then shuts down @transport.event_handler("on_client_connected") async def on_client_connected(transport, client): asyncio.create_task(end_after(worker, timeout_secs=300)) # 5-minute cap ``` On Pipecat Cloud, there is also a platform-level hard cap via [`maxSessionDuration`](/pipecat-cloud/fundamentals/active-sessions#session-duration-limits) (default 7200s). That cap cancels the bot and shuts the pipeline down with no goodbye, so use the bot-level timer above when you want the bot to speak before the call ends. ## Implementation Patterns ### Event-Driven Termination Connect termination to transport events for automatic cleanup: ```python @transport.event_handler("on_client_connected") async def on_client_connected(transport, client): logger.info("Client connected - starting conversation") await worker.queue_frames([LLMRunFrame()]) @transport.event_handler("on_client_disconnected") async def on_client_disconnected(transport, client): logger.info("Client disconnected - immediate termination") await worker.cancel() # Run the pipeline runner = WorkerRunner(handle_sigint=False) await runner.run(worker) ``` ### Conditional Termination Use function calling or other logic to determine when conversations should end: ```python async def check_conversation_complete(params: FunctionCallParams): # Your logic to determine if conversation should end conversation_complete = await evaluate_completion_criteria() if conversation_complete: await params.llm.push_frame(TTSSpeakFrame("Thank you for using our service!")) await params.llm.push_frame(EndWorkerFrame(), FrameDirection.DOWNSTREAM) await params.result_callback({"status": "complete" if conversation_complete else "continuing"}) ``` ### Error Handling Ensure pipelines can terminate properly even when exceptions occur: ```python try: runner = WorkerRunner(handle_sigint=False) await runner.run(worker) except Exception as e: logger.error(f"Pipeline error: {e}") # Ensure cleanup happens even on errors await worker.cancel() ``` ### Running Cleanup Code on Shutdown To run cleanup or persist data when a call ends, use the `on_pipeline_finished` event handler. It fires after the pipeline reaches any terminal state, so it runs for **both** graceful (`EndFrame`) and cancelled (`CancelFrame`) shutdowns. This makes it the single write point for end-of-call work like saving a transcript or recording: ```python @worker.event_handler("on_pipeline_finished") async def on_pipeline_finished(worker, frame): # Runs for both graceful and cancelled shutdown await save_transcript_to_db() ``` `on_client_disconnected`, by contrast, fires only when the client disconnects. Use it to _tag the reason_ for the shutdown (for example, "user hung up"), and do the actual persistence in `on_pipeline_finished` so you write data exactly once regardless of how the call ended. See the [`PipelineWorker` events](/api-reference/server/pipeline/pipeline-worker) reference for the full event signature. ## Troubleshooting If your pipeline isn't shutting down properly, check these common issues: ### Custom Processors Not Propagating Frames **Problem:** Custom processors that don't call `push_frame()` can block termination frames from reaching the end of the pipeline. **Solution:** Ensure your custom processors propagate all frames downstream, including `EndFrame` and `CancelFrame`: ```python async def process_frame(self, frame: Frame, direction: FrameDirection): await super().process_frame(frame, direction) # Your custom processing logic here # Always push frames downstream (including termination frames) await self.push_frame(frame, direction) ``` ### Incorrect Termination Frame Direction **Problem:** Pushing `EndFrame` or `CancelFrame` from the middle of the pipeline may not reach the pipeline source properly. **Solution:** Use the appropriate frame type and direction: ```python await self.push_frame(EndWorkerFrame(), FrameDirection.DOWNSTREAM) await self.push_frame(CancelWorkerFrame(), FrameDirection.DOWNSTREAM) # The pipeline source will then convert these to proper termination frames # and push them downstream through the entire pipeline ``` The pipeline source automatically converts `EndWorkerFrame` to `EndFrame` and `CancelWorkerFrame` to `CancelFrame` when pushing downstream, ensuring proper termination handling throughout the pipeline. ### "dangling tasks detected" Warning **Problem:** On shutdown you see a log warning like `PipelineWorker#0 dangling tasks detected: [...]`. **Solution:** This means `asyncio` tasks created during the session were never cancelled or awaited before the pipeline shut down. The usual causes are the two above: a custom processor that doesn't propagate termination frames, or a background task started inside a processor (for example, a timer or long-running coroutine) that isn't cleaned up. Create background tasks through the pipeline task manager so they are tracked and cancelled on shutdown, and make sure your processors push `EndFrame`/`CancelFrame` downstream. ## Key Takeaways - **Frame-based termination** - shutdown uses the same frame system as processing - **Choose the right method** - graceful for natural endings, immediate for disconnections - **Event handlers enable automatic termination** - respond to user disconnections cleanly - **Idle detection provides safety net** - prevents hanging processes and resource waste - **SystemFrames have priority** - CancelFrames are processed before queued non-system frames for fast shutdown - **Resource cleanup is automatic** - proper termination ensures clean resource disposal ## What's Next You now understand how to build, run, and properly terminate voice AI pipelines! With the single-agent basics covered, let's see how Pipecat coordinates multiple agents, starting with giving an agent its own LLM and tools. Give an agent its own LLM and register tools with the @tool decorator # Multiple LLM Agents Source: https://docs.pipecat.ai/pipecat/learn/multiple-llm-agents.md Run separate agents that each own their own LLM, tools, and conversation context. So far you've built a single agent: one LLM, one context, one set of tools. But many problems are better served by **several agents working together**, each owning its own LLM. A greeter hands off to a support agent. A researcher runs in the background while the main agent keeps talking. A screen-driving agent acts on the UI while a voice agent converses. Giving each agent its own LLM keeps every context small and focused. Instead of one model juggling every instruction and tool, each agent reasons over just its own job. That's cheaper, faster, and less prone to the model getting lost. ## LLMWorker overview Multi-agent systems often run several LLM-backed agents -- a greeter, a support agent, a researcher -- each with its own instructions, tools, and (optionally) its own conversation context. `LLMWorker` is the building block for each one. It extends `PipelineWorker` with everything you need to run an LLM-powered agent: - A pipeline with your LLM service, automatically built - Tool registration via the `@tool` decorator - Activation handling that injects messages and runs the LLM To create an LLM agent, subclass `LLMWorker` so you can host `@tool` methods, then instantiate it with its own LLM service. Pass `bridged=()` so the agent receives frames from the bus: ```python import os from pipecat.services.openai.llm import OpenAILLMService from pipecat.workers.llm import LLMWorker class MyAgent(LLMWorker): """An LLM agent. ``@tool`` methods go here.""" def build_agent() -> MyAgent: llm = OpenAILLMService( api_key=os.environ["OPENAI_API_KEY"], settings=OpenAILLMService.Settings( system_instruction="You are a helpful assistant.", ), ) return MyAgent("assistant", llm=llm, bridged=()) ``` You never pass a `bus=` argument to the constructor -- the worker gets its bus when you register it with `runner.add_workers(...)`. The default pipeline is `Pipeline([llm])`, with tools from `build_tools()` automatically registered. When `bridged=()` is set, the framework wraps this pipeline with edge processors that connect it to the bus. ## The @tool decorator The `@tool` decorator marks a method as an LLM-callable tool. The framework automatically collects all `@tool`-decorated methods and registers them with the LLM service. ```python from pipecat.services.llm_service import FunctionCallParams from pipecat.workers.llm import tool class MyAgent(LLMWorker): @tool async def get_weather(self, params: FunctionCallParams, city: str): """Get the current weather for a city. Args: city (str): The city name (e.g. 'San Francisco'). """ weather = await fetch_weather(city) await params.result_callback(weather) ``` The tool's name comes from the method name. The docstring becomes the tool description. Parameter types and descriptions are extracted from the type annotations and the `Args` section in the docstring. ### Tool options The `@tool` decorator accepts options: ```python @tool(cancel_on_interruption=False, timeout=60) async def long_running_tool(self, params: FunctionCallParams, query: str): """A tool that takes a while. Args: query (str): The search query. """ result = await expensive_search(query) await params.result_callback(result) ``` | Option | Default | Description | | ------------------------ | ------- | -------------------------------------- | | `cancel_on_interruption` | `True` | Cancel the tool if the user interrupts | | `timeout` | `None` | Maximum execution time in seconds | ### Tool parameters Every tool method receives `self` and `params: FunctionCallParams` as the first two arguments. Additional arguments are the tool's parameters that the LLM fills in. The `params` object gives you access to: - `params.result_callback(result)` -- return the result to the LLM - `params.llm` -- the LLM service instance, useful for queuing frames ### Returning results Always call `params.result_callback()` to return the tool result to the LLM: ```python @tool async def lookup(self, params: FunctionCallParams, item: str): """Look up an item. Args: item (str): The item to look up. """ data = await database.get(item) await params.result_callback({"found": True, "data": data}) ``` ## Activation with messages When an `LLMWorker` is activated, you can inject messages into its context. Pass an `LLMWorkerActivationArgs` via the `args` parameter: ```python from pipecat.workers.llm import LLMWorkerActivationArgs await self.activate_worker( "support", args=LLMWorkerActivationArgs( messages=[{"role": "developer", "content": "The user asked about pricing."}], run_llm=True, # Run the LLM immediately after injection ), ) ``` The default `on_activated()` implementation: 1. Sets the tools from `build_tools()` 2. Injects the provided messages into the LLM context 3. Runs the LLM if `run_llm` is `True` (the default when `messages` is set) ## Managing context with LLMContextWorker A plain `LLMWorker` runs an LLM but doesn't manage conversation context on its own -- it relies on context coming from elsewhere (for example, the main agent's aggregators bridged in). When an agent needs to keep its **own** history, use `LLMContextWorker`. It extends `LLMWorker` with a built-in `LLMContext` and the user/assistant aggregator pair, building the pipeline as `[user_aggregator, llm, assistant_aggregator]` for you. ```python from pipecat.workers.llm import LLMContextWorker class AssistantAgent(LLMContextWorker): """An LLM agent that keeps its own conversation context.""" agent = AssistantAgent("assistant", llm=llm) # gets its own context ``` Each `LLMContextWorker` gets its own context by default, so agents don't see each other's history. To give several agents a **shared** conversation, pass the same `context=` to each: ```python from pipecat.processors.aggregators.llm_context import LLMContext shared = LLMContext() agent_a = AssistantAgent("agent_a", llm=llm_a, context=shared) agent_b = AssistantAgent("agent_b", llm=llm_b, context=shared) ``` Access the managed aggregators via `self.user_aggregator` and `self.assistant_aggregator`. ## Custom pipelines If you need more control, you can pass a custom `pipeline=` to the `LLMWorker` constructor. For example, to add TTS to the agent's own pipeline: ```python from pipecat.pipeline.pipeline import Pipeline from pipecat.services.cartesia.tts import CartesiaTTSService class AgentWithTTS(LLMWorker): def __init__(self, name: str, *, llm, voice_id: str): tts = CartesiaTTSService( api_key=os.environ["CARTESIA_API_KEY"], settings=CartesiaTTSService.Settings(voice=voice_id), ) super().__init__(name, llm=llm, pipeline=Pipeline([llm, tts]), bridged=()) ``` This is how you give each agent its own voice -- each agent adds its own TTS after the LLM, so a handoff sounds like a real transfer between distinct speakers. ## What's next Now that your agents can run LLMs and call tools, here's a powerful one: an agent that sees and drives the user's screen. A UIWorker that reads the screen and acts on it over a two-way RTVI interface # Controlling the UI Source: https://docs.pipecat.ai/pipecat/learn/ui-worker.md Bridge a voice agent and a client GUI with a UIWorker over a two-way RTVI interface. ## What is a UIWorker? When you put a voice agent in front of an app, talking isn't enough — the agent needs to _see what the user sees_ and _act on the screen_: read the page, point at things, fill in fields, click buttons. A `UIWorker` is the server-side agent that makes this possible. It voice-enables a client UI by connecting an LLM to whatever the user is looking at. The connection is **two-way**, over the RTVI UI channel: - **Client → server.** The client streams the screen to the worker as accessibility snapshots, and forwards the user's UI interactions as events. - **Server → client.** The worker drives the page back — scrolling, highlighting, selecting text, filling inputs, clicking, or running app-defined commands — and can surface long-running work as progress cards. A `UIWorker` is the screen half of a voice/UI split: a **voice agent** owns the conversation, and the `UIWorker` owns the screen. Each is a separate LLM with its own focused context. The worker auto-injects the latest screen state into _its_ context before every turn, so the conversational voice LLM never has to carry a giant accessibility tree: - The **voice agent** converses and decides what's worth saying. - The **UIWorker** reasons over the current page and acts on it. The result is two small, fast contexts instead of one bloated one — cheaper, and less prone to the model getting lost. The two directions map to RTVI UI messages: the client sends `ui-snapshot` and `ui-event`; the worker sends `ui-command` and `ui-job-group`. You rarely touch these directly — `PipelineWorker` wires the channel up automatically when RTVI is enabled (the default). See [The RTVI Standard](/client/rtvi-standard#user-interface) for the wire protocol and the [UIWorker API reference](/api-reference/server/workers/ui-worker) for the class. ## The two-way interface A `UIWorker` gives an LLM a handful of capabilities, split across the two directions of the interface. Everything below works out of the box on any `UIWorker` subclass. ### What the worker sees (client → server) **The screen, as a snapshot.** The client sends an accessibility snapshot of the page whenever it changes. The worker renders the latest one as a `` block and — with `auto_inject_ui_state` on (the default) — injects it into the LLM context before every turn, so the model always reasons over what's currently on screen. Each element carries a stable `ref` the worker uses to act on it: ``` - heading "Shopping list" [level=1] [ref=e3] - list: - checkbox "milk" [checked] [ref=e5] - checkbox "eggs" [ref=e6] ``` When the user has text selected, the snapshot includes a `` block so the LLM can resolve deictic references like "this paragraph" or "what I selected". **User interactions, as events.** The client dispatches app-defined events (a button click, a custom gesture) with `sendUIEvent(name, payload)`. Route them to handlers with `@ui_event(name)`; each runs in its own task: ```python from pipecat.workers.ui import UIWorker, ui_event class MyUIWorker(UIWorker): @ui_event("note_click") async def on_note_click(self, message): ref = (message.payload or {}).get("ref") await self.scroll_to(ref) await self.select_text(ref) ``` ### What the worker does (server → client) **Drives the page.** The worker acts on the screen by sending UI commands. The built-in helpers cover the common actions, and `send_command(name, payload)` sends any app-defined command: | Helper | Effect | | ----------------------------- | -------------------------------------------- | | `scroll_to(ref)` | Bring an element into view | | `highlight(ref)` | Briefly flash an element | | `select_text(ref)` | Select an element's text (pointing / deixis) | | `click(ref)` | Click a checkbox, radio, or button | | `set_input_value(ref, value)` | Fill a text input or textarea | | `send_command(name, payload)` | Any app-defined command (e.g. `"add_note"`) | The standard client handlers ship in `@pipecat-ai/client-react`; apps can override them or define their own command names. **Answers back.** A `UIWorker` answers via a built-in single-flight `respond` job: a requester dispatches `job("ui", name="respond", payload={"query": ...})`, the worker runs one screen-grounded LLM turn, and a `@tool` ends it by calling `respond_to_job()`. That call chooses how the answer reaches the user: - `respond_to_job(text, tts_speak=True)` — speak `text` verbatim through the requester's TTS. - `respond_to_job(text)` — return `{"answer": text}` for the requester's voice LLM to phrase. - `respond_to_job()` — complete the turn silently (the worker acted, but said nothing). **Surfaces long work.** When a turn kicks off background work, `ui_job_group` / `start_ui_job_group` fan it out to peer workers _and_ surface it to the client as a cancellable progress card, streaming each worker's updates as they arrive: ```python await self.start_ui_job_group( "wikipedia", "news", "scholar", payload={"query": research_query}, label=f"Research: {research_query}", ) ``` By default a `UIWorker` is stateless: it clears its context at the start of each `respond` job, so every turn sees only the current `` and query. Set `keep_history=True` to accumulate history across turns — useful for multi-turn references like "can we add a note for that?" — at the cost of more tokens. ## Hello world The smallest `UIWorker` ties the interface together: a delegate that answers questions about the page. The voice agent forwards screen-relevant utterances to it; the worker reads the screen (client → server) and speaks the reply (server → client). The worker needs only an LLM and one `@tool` that ends the turn with `respond_to_job()`: ```python from pipecat.workers.ui import UIWorker from pipecat.workers.llm import tool class HelloWorker(UIWorker): @tool async def answer(self, params, text: str): """Speak `text` back to the user.""" await self.respond_to_job(text, tts_speak=True) await params.result_callback(None) ``` The voice agent exposes a tool that dispatches a `respond` job to the worker and speaks back whatever it returns: ```python async def answer_about_screen(params, query: str): """Ask the screen-aware UI layer to answer about the current page.""" async with params.pipeline_worker.job( "hello", name="respond", payload={"query": query}, timeout=30 ) as t: pass await params.result_callback(t.response) ``` Register both with the runner — the `UIWorker` comes online to receive snapshots and jobs as soon as its pipeline starts: ```python await runner.add_workers(HelloWorker(), worker) ``` Here's the full round trip for one utterance: The client streams the current screen as a `ui-snapshot`. `PipelineWorker` broadcasts it on the bus; the `UIWorker` stores the latest one. The user speaks. The voice LLM calls `answer_about_screen`, which dispatches a `respond` job to the `UIWorker`. The worker's `respond` job runs one LLM turn with the latest `` auto-injected, so its answer is grounded in what's on screen. The worker's `answer` tool calls `respond_to_job(text, tts_speak=True)`. The voice agent speaks the reply verbatim. ## Patterns How the worker gets triggered — and who speaks — falls into two patterns. ### Delegation In the hello-world example, the voice agent is the gatekeeper: it decides which turns involve the screen and routes those to the `UIWorker`, then voices the result. This is the **delegation** pattern, and it's the common one. The voice LLM stays small and screen-unaware; the worker owns all screen reasoning. Most apps don't need a custom tool per action. `ReplyToolMixin` provides a single bundled `reply` tool — a required spoken `answer` plus optional `scroll_to`, `highlight`, `select_text`, `fills`, and `click` — covering pointing, reading, and form apps: ```python from pipecat.workers.ui import ReplyToolMixin, UIWorker class FormWorker(ReplyToolMixin, UIWorker): def __init__(self): super().__init__("ui", llm=OpenAILLMService(api_key="...")) ``` The LLM uses whichever fields fit the turn — `select_text` to point at "this paragraph", `fills` + `click` to complete a form — and unused fields stay `null`. Delegation also scales to background work. A `@tool` can fan out to peer workers with `start_ui_job_group`, which surfaces a cancellable progress card on the client and returns immediately so the voice agent isn't blocked: ```python class ResearchWorker(UIWorker): @tool async def reply(self, params, answer: str, research_query: str | None = None): if research_query: await self.start_ui_job_group( "wikipedia", "news", "scholar", payload={"query": research_query}, label=f"Research: {research_query}", ) await self.respond_to_job(answer) await params.result_callback(None) ``` ### Parallel handling Sometimes the screen should react to _every_ user turn, not only the ones the voice agent chooses to delegate. In the **parallel handling** pattern, both agents receive each turn and act in parallel: the voice agent converses while the `UIWorker` updates the screen, independently. The key difference: there's no tool call routing work to the worker. Instead, the voice pipeline's user aggregator fires `on_user_turn_stopped` once per turn, and that handler dispatches the transcript to the worker as a `respond` job. Because it runs in its own task, the voice LLM (running from the same turn) and the worker act concurrently: ```python @user_aggregator.event_handler("on_user_turn_stopped") async def on_user_turn_stopped(aggregator, strategy, message): transcript = (message.content or "").strip() if not transcript: return async with worker.job("ui", name="respond", payload={"query": transcript}, timeout=15): pass # fire-and-forget; the worker acts on its own ``` The worker acts **silently** — its `@tool` completes the job with `respond_to_job()` and no answer, so nothing it does reaches TTS. The separate voice layer owns speech: ```python class ListWorker(UIWorker): @tool async def update_list(self, params, add=None, check=None, remove=None): for text in add or []: await self.send_command("add_item", {"text": text}) for ref in check or []: await self.send_command("set_checked", {"ref": ref, "checked": True}) for ref in remove or []: await self.send_command("remove_item", {"ref": ref}) await self.respond_to_job() # no answer — acts silently await params.result_callback(None) ``` Here the **snapshot is the shared source of truth**. The worker acts on it, and the voice agent reads it through a read-only tool — so the voice agent can answer "what's left on my list?" from what's actually on screen (including items the user checked off by hand), not from conversation memory: ```python async def check_list(params): """Look up what's currently on the list.""" await params.result_callback(list_worker.list_summary()) # reads the live snapshot ``` ### Choosing a pattern | | Delegation | Parallel handling | | ------------------------------- | --------------------------------------------------------------------------------- | ------------------------------------------------------------- | | **How the worker is triggered** | The voice LLM calls a tool | The voice pipeline's `on_user_turn_stopped` event, every turn | | **Who speaks** | Often the worker (`tts_speak=True`), or the voice LLM phrases the worker's result | The voice agent; the worker acts silently | | **Voice LLM's role** | Gatekeeper — decides what's screen-relevant | Converses; reads shared state but never mutates the UI | | **Best when** | The page matters only for some turns | Every turn should drive the UI and speech is incidental | Both keep the voice LLM's context small. Delegation gives the voice agent control over when the screen is involved; parallel handling makes the screen a first-class output of every turn. ## What's next You've built agents that converse, call tools, and drive the screen. Next, learn how to transfer control between them. Activation, deactivation, and seamless control transfer Full reference for the `UIWorker` class, UI commands, job groups, and `ReplyToolMixin`. # Agent Handoff Source: https://docs.pipecat.ai/pipecat/learn/agent-handoff.md Transfer control between agents with activation, deactivation, and handoff. ## The activation model In a multi-agent system, only one agent is **active** at a time (per bridge). The active agent receives frames from the bus. Inactive agents exist but don't process frames. Every worker has an `active` property. `LLMWorker` defaults to `active=False`, so in a handoff setup the LLM agents start inactive and the main agent activates the first one explicitly. You control which agent is active with two methods: | Method | What it does | | --------------------------------------------- | ------------------------------------------ | | `activate_worker(name, args=...)` | Activate another agent | | `deactivate_worker(name)` | Deactivate another agent | | `activate_worker(name, deactivate_self=True)` | Hand off: deactivate self, activate target | Passing `deactivate_self=True` to `activate_worker()` is the most common form -- it's a single call that transfers control from the current agent to another. ## Building a handoff system Let's walk through how the two-agent handoff works. You need three pieces. ### 1. A main agent with a bus bridge The main agent owns the transport (audio I/O) and places a `BusBridgeProcessor` in its pipeline instead of an LLM. The bridge routes frames to whichever LLM agent is active. The main agent is a `PipelineWorker` wrapping that pipeline: ```python from pipecat.bus import BusBridgeProcessor from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.worker import PipelineParams, PipelineWorker MAIN_NAME = "main" stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY")) tts = CartesiaTTSService(api_key=os.getenv("CARTESIA_API_KEY")) context = LLMContext() aggregators = LLMContextAggregatorPair( context, user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()), ) bridge = BusBridgeProcessor(bus=runner.bus, worker_name=MAIN_NAME) pipeline = Pipeline([ transport.input(), stt, aggregators.user(), bridge, # Where the LLM would go tts, transport.output(), aggregators.assistant(), ]) main = PipelineWorker(pipeline, name=MAIN_NAME, params=PipelineParams()) ``` The `BusBridgeProcessor` is what lets a transport-owning agent delegate its LLM turn to other agents over the bus. For how it routes frames and how to filter by bridge name, see [Understanding the Bus Bridge](/pipecat/fundamentals/understanding-the-bus-bridge). ### 2. LLM agents with bridged=() Each LLM agent is an `LLMWorker` created with `bridged=()` so it receives frames from the bus. Subclass `LLMWorker` to add tools, then instantiate it with its own LLM: ```python from pipecat.workers.llm import LLMWorker, tool class AcmeAgent(LLMWorker): # tools defined below def build_greeter() -> AcmeAgent: llm = OpenAILLMService( api_key=os.getenv("OPENAI_API_KEY"), settings=OpenAILLMService.Settings( system_instruction="You are a friendly greeter. Route product questions to support.", ), ) return AcmeAgent("greeter", llm=llm, bridged=()) ``` `bridged=()` means the agent receives frames from all bridges. You can filter by bridge name with `bridged=("voice",)` if you have multiple bridges. ### 3. Handoff via tools The LLM decides when to transfer by calling a tool. The tool calls `activate_worker()` with `deactivate_self=True`: ```python from pipecat.workers.llm import LLMWorker, LLMWorkerActivationArgs, tool class AcmeAgent(LLMWorker): @tool(cancel_on_interruption=False) async def transfer_to_agent(self, params: FunctionCallParams, agent: str, reason: str): """Transfer the user to another agent. Args: agent (str): The agent to transfer to (e.g. 'support'). reason (str): Why the user is being transferred. """ await self.activate_worker( agent, args=LLMWorkerActivationArgs( messages=[{"role": "developer", "content": reason}], ), deactivate_self=True, result_callback=params.result_callback, ) ``` For `LLMWorker`, `activate_worker()` also accepts a `messages` parameter. These messages are injected and spoken by the current agent _before_ the transfer happens -- useful for announcing the handoff: ```python await self.activate_worker( agent, messages=[{"role": "developer", "content": f"Tell the user about the transfer ({reason})."}], args=LLMWorkerActivationArgs( messages=[{"role": "developer", "content": reason}], ), deactivate_self=True, result_callback=params.result_callback, ) ``` When the greeter calls `activate_worker("support", deactivate_self=True, ...)`: 1. The greeter is deactivated -- it stops receiving frames from the bus 2. The support agent is activated with the provided arguments 3. The support agent's `on_activated()` fires, injecting the reason message into its LLM context 4. The support agent starts responding to the user The transition is seamless -- the user experiences it as a natural conversation flow. ## Activation arguments When activating an `LLMWorker`, pass `LLMWorkerActivationArgs` via `args=` to give the target agent context about why it was activated: ```python from pipecat.workers.llm import LLMWorkerActivationArgs await self.activate_worker( "support", args=LLMWorkerActivationArgs( messages=[{"role": "developer", "content": "The user asked about Rocket Boots."}], ), ) ``` The target agent receives these messages in its LLM context and immediately runs the LLM to respond. ## Activating the first agent Before you can activate an agent, it needs to be ready (its pipeline must be started). The simplest place to kick off the conversation is the transport's `on_client_connected` handler, where you activate the first agent: ```python @transport.event_handler("on_client_connected") async def on_client_connected(transport, client): await main.activate_worker( "greeter", args=LLMWorkerActivationArgs( messages=[{"role": "developer", "content": "Welcome the user."}], ), ) ``` If you need to react to a specific agent registering (for example in a distributed setup), use the `@worker_ready` decorator instead. See [Agent Registry and Discovery](/pipecat/fundamentals/agent-registry-and-discovery). ## Putting it all together Here's the full flow: `WorkerRunner` creates the bus. You add every agent (the main agent and the LLM agents) with `runner.add_workers(...)`. The transport's `on_client_connected` handler activates the greeter. Audio flows: transport -> STT -> BusBridge -> bus -> greeter's LLM -> bus -> BusBridge -> TTS -> transport. The greeter's LLM calls `transfer_to_agent`. The tool calls `activate_worker("support", deactivate_self=True, ...)`. Greeter deactivates, support activates with context. Audio now flows through the support agent. ## What's next Handoff transfers a conversation between agents. But sometimes you need agents to do work in parallel. Next, let's look at job coordination. Dispatch work to multiple agents in parallel # Job Coordination Source: https://docs.pipecat.ai/pipecat/learn/job-coordination.md Dispatch work from one Pipecat agent to others and collect results with job coordination in the multi-agent framework. ## What are jobs? Jobs let one agent dispatch work to other agents. While handoff transfers the conversation, jobs are for background work -- an agent asks other agents to do something and collects their results. Use cases for jobs: - Researching a topic from multiple perspectives in parallel - Running a code analysis while talking to the user - Fetching data from multiple sources simultaneously ## Single job The simplest form is `self.job()`, which sends a job to one agent and waits for the response: ```python async with self.job("worker", payload={"question": "What is Pipecat?"}, timeout=30) as j: pass print(j.response) ``` `self.job()` is a context manager that handles the full lifecycle: it waits for the agent to be ready, sends the request, and collects the response. Under the hood, this is equivalent to calling `request_job()` (fire-and-forget) and then handling `on_job_response()` and `on_job_completed()` yourself. ## Job group When you need to dispatch work to multiple agents in parallel, use `self.job_group()`: ```python async with self.job_group("worker1", "worker2", "worker3", payload={"topic": "AI safety"}, timeout=30) as jg: pass for worker_name, response in jg.responses.items(): print(f"{worker_name}: {response}") ``` `self.job_group()` works the same way but for multiple agents: it waits for all of them to be ready, sends a request to each, and collects all responses. Under the hood, this is equivalent to calling `request_job_group()` and then handling `on_job_response()` for each agent and a final `on_job_completed()` yourself. The same payload is sent to every agent. If you need different arguments per agent, you can structure the payload so each one reads its own key: ```python async with self.job_group("researcher", "fact_checker", payload={ "researcher": {"topic": "AI safety", "depth": "detailed"}, "fact_checker": {"claims": ["AI can self-improve", "AGI is near"]}, }, timeout=30) as jg: pass ``` ## Handling job requests Agents handle incoming jobs in two ways. ### The @job decorator The `@job` decorator marks a method as a job handler. The framework automatically dispatches matching requests to it. The decorator requires a `name` argument. The handler responds with `message.job_id`: ```python from pipecat.pipeline.job_decorator import job class MyWorker(BaseWorker): @job(name="process") async def on_process(self, message: BusJobRequestMessage): result = await self._do_work(message.payload) await self.send_job_response(message.job_id, result) ``` You can use different job names to route work to different handlers: ```python class MyWorker(BaseWorker): @job(name="research") async def on_research(self, message: BusJobRequestMessage): await self.send_job_response(message.job_id, {"answer": "..."}) @job(name="summarize") async def on_summarize(self, message: BusJobRequestMessage): await self.send_job_response(message.job_id, {"summary": "..."}) ``` The requester specifies the job name when dispatching: ```python async with self.job("worker", name="research", payload={"topic": "AI"}) as j: pass ``` Each request runs in its own asyncio task, so multiple requests to the same handler are executed concurrently without blocking the bus message loop. ### Overriding on_job_request Alternatively, you can override `on_job_request()` directly without the `@job` decorator: ```python class MyWorker(BaseWorker): async def on_job_request(self, message: BusJobRequestMessage) -> None: await super().on_job_request(message) result = await self._do_work(message.payload) await self.send_job_response(message.job_id, result) ``` This is useful when you need custom routing logic or want to integrate with an existing pipeline, as shown in the example below. `send_job_response()`, `send_job_update()`, and `send_job_stream_*()` all require an explicit `job_id`. This lets an agent handle multiple concurrent jobs and respond to each one correctly. For simple handlers, pass `message.job_id` from the request. For asynchronous responses (see the example below), track the `job_id` yourself until you're ready to respond. ## Building a job system Let's build a debate system where a moderator dispatches a topic to three agents, each arguing from a different perspective. ### Worker agents Each debate agent runs its own LLM pipeline. The LLM response arrives asynchronously through an event handler, so the agent tracks the current job ID until it has something to respond with: ```python from pipecat.workers.llm import LLMContextWorker from pipecat.bus.messages import BusJobRequestMessage from pipecat.frames.frames import LLMMessagesAppendFrame class DebateWorker(LLMContextWorker): def __init__(self, role: str): llm = OpenAILLMService( api_key=os.getenv("OPENAI_API_KEY"), settings=OpenAILLMService.Settings(system_instruction=f"You argue as a {role}."), ) super().__init__(role, llm=llm) self._role = role self._current_job_id: str | None = None @self.assistant_aggregator.event_handler("on_assistant_turn_stopped") async def on_assistant_turn_stopped(aggregator, message): if self._current_job_id: job_id = self._current_job_id self._current_job_id = None await self.send_job_response(job_id, {"role": self._role, "text": message.content}) async def on_job_request(self, message: BusJobRequestMessage) -> None: await super().on_job_request(message) self._current_job_id = message.job_id await self.queue_frame( LLMMessagesAppendFrame( messages=[{"role": "developer", "content": f"Topic: {message.payload['topic']}"}], run_llm=True, ) ) ``` `LLMContextWorker` extends `LLMWorker` with a built-in `LLMContext` and aggregator pair. It builds the pipeline as `[user_aggregator, llm, assistant_aggregator]` automatically, so you don't need to wire the context plumbing yourself. Access the aggregators via `self.user_aggregator` and `self.assistant_aggregator`. The agent: 1. Receives a job request and stores the `job_id` 2. Injects the topic into its LLM context and runs the LLM 3. When the LLM finishes its turn, the event handler sends the response with the stored `job_id` ### Coordinator agent The moderator triggers jobs via a tool and synthesizes the results: ```python from pipecat.workers.llm import LLMWorker, tool class ModeratorAgent(LLMWorker): @tool(cancel_on_interruption=False) async def debate(self, params: FunctionCallParams, topic: str): """Analyze a topic from multiple perspectives. Args: topic (str): The topic to debate. """ async with self.job_group("advocate", "critic", "analyst", payload={"topic": topic}, timeout=30) as jg: pass result = "\n\n".join( f"{r['role'].upper()}: {r['text']}" for r in jg.responses.values() ) await params.result_callback(result) ``` Create the debate agents and the moderator, then register them all with `runner.add_workers(ModeratorAgent("moderator", llm=...), DebateWorker("advocate"), DebateWorker("critic"), DebateWorker("analyst"))`. When the user says "debate whether AI should be regulated," the moderator: 1. The LLM calls the `debate` tool 2. `job_group()` sends the topic to all three agents in parallel 3. Each agent runs its own LLM and responds with its perspective 4. The moderator collects all responses and returns them to the LLM 5. The LLM synthesizes a balanced summary and speaks it to the user ## Job lifecycle The requester calls `job()` or `job_group()`. The framework waits for agents to be ready, then sends a job request to each. Agents receive the request in `on_job_request()` and do their work. Agents call `send_job_response(job_id, ...)` with their results. When all agents respond (or timeout occurs), the context manager exits and results are available. The framework also supports fire-and-forget jobs, progress updates, and streaming. ## Job cancellation Jobs can be cancelled in several ways. The framework handles cleanup automatically and notifies agents so they can stop in-progress work. ### Automatic cancellation with context managers When using `job()` or `job_group()`, cancellation happens automatically if the context block raises an exception. This includes tool interruptions (`CancelledError`) when using `cancel_on_interruption=True`: ```python # If an exception or CancelledError is raised inside the block, # all agents are cancelled automatically async with self.job_group("w1", "w2", payload=data) as jg: # ... if this raises, the agents get cancelled pass ``` ### Worker errors By default (`cancel_on_error=True`), if any agent responds with an error status, the remaining agents in the group are cancelled and `JobGroupError` is raised: ```python try: async with self.job_group("w1", "w2", payload=data) as jg: pass except JobGroupError as e: # An agent errored -- remaining agents were cancelled pass ``` ### Manual cancellation For fire-and-forget jobs (using `request_job()`), you manage cancellation yourself by tracking job IDs: ```python job_ids = [] try: job_ids.append(await self.request_job("w1", payload={"job": 1})) job_ids.append(await self.request_job("w2", payload={"job": 2})) # ... except asyncio.CancelledError: for jid in job_ids: await self.cancel_job_group(jid, reason="tool cancelled") ``` ### Handling cancellation on the agent side When a job is cancelled, the agent's `on_job_cancelled` hook fires. The framework automatically sends a `CANCELLED` response back to the requester, so you only need to override this hook if you have resources to clean up: ```python class MyWorker(BaseWorker): async def on_job_cancelled(self, message: BusJobCancelMessage) -> None: # Optional: clean up resources, stop in-progress work logger.info(f"Job {message.job_id} cancelled: {message.reason}") ``` ### Agent shutdown When an agent stops with jobs still in flight, it automatically sends a `CANCELLED` response for each active job so requesters aren't left waiting on a timeout. ## What's next You can now build agents, hand off between them, and dispatch jobs. So far everything has run in a single process. Next, let's scale across processes and machines. Run agents across processes and machines on a shared bus # Distributed Agents Source: https://docs.pipecat.ai/pipecat/learn/distributed-agents.md Run Pipecat agents across processes and machines connected to the same bus for distributed multi-agent deployments. ## Overview Distributed agents are agents connected to the **same bus** but running in different processes or on different machines. By default, all agents run in a single process using a local bus. For distributed setups, swap to a network bus -- all agents share the same channel and discover each other automatically. Your agent code stays the same. Distributed agents share a bus. If you need to connect agents on **different buses** (separate networks, third-party services), see [Proxy Agents](/pipecat/learn/proxy-agents) instead. ## Setting up a distributed bus Pipecat provides two distributed bus implementations: RedisBus and PgmqBus. Choose based on your infrastructure. ### RedisBus Each process creates its own `WorkerRunner` with a `RedisBus` connected to the same Redis channel: ```python from redis.asyncio import Redis from pipecat.bus.network.redis import RedisBus from pipecat.workers.runner import WorkerRunner redis = Redis.from_url("redis://localhost:6379") bus = RedisBus(redis=redis, channel="pipecat:my-app") runner = WorkerRunner(bus=bus, handle_sigint=True) ``` All runners sharing the same `channel` can exchange messages. Agents discover each other automatically through registry snapshots. Install the Redis extra: `uv add "pipecat-ai[redis]"` ### PgmqBus Alternatively, use `PgmqBus` backed by PostgreSQL Message Queue: ```python from pgmq.async_queue import PGMQueue from pipecat.bus.network.pgmq import PgmqBus from pipecat.workers.runner import WorkerRunner pgmq = PGMQueue( host="localhost", port="5432", database="postgres", username="postgres", password="...", pool_size=4, ) await pgmq.init() bus = PgmqBus(pgmq=pgmq, channel="pipecat:my-app") runner = WorkerRunner(bus=bus, handle_sigint=True) ``` Install the PGMQ extra: `uv add "pipecat-ai[pgmq]"` ## Example: distributed handoff This example splits the two-agent handoff across separate processes. The main agent handles transport on one machine, and LLM agents run independently on other machines. ### Process 1: Main transport agent The main agent owns the transport and bridges frames to the bus. It has no LLM -- it waits for the remote greeter agent to register, then activates it. ```python # main.py import os from redis.asyncio import Redis from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.bus import BusBridgeProcessor from pipecat.bus.network.redis import RedisBus from pipecat.pipeline.pipeline import Pipeline from pipecat.workers.runner import WorkerRunner from pipecat.pipeline.worker import PipelineParams, PipelineWorker from pipecat.processors.aggregators.llm_context import LLMContext from pipecat.processors.aggregators.llm_response_universal import ( LLMContextAggregatorPair, LLMUserAggregatorParams, ) from pipecat.registry.types import WorkerReadyData from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.services.deepgram.stt import DeepgramSTTService from pipecat.workers.llm import LLMWorkerActivationArgs MAIN_NAME = "acme" async def run_bot(transport, runner_args): redis = Redis.from_url(runner_args.cli_args.redis_url) bus = RedisBus(redis=redis, channel=runner_args.cli_args.channel) runner = WorkerRunner(bus=bus, handle_sigint=runner_args.handle_sigint) stt = DeepgramSTTService(api_key=os.environ["DEEPGRAM_API_KEY"]) tts = CartesiaTTSService( api_key=os.environ["CARTESIA_API_KEY"], settings=CartesiaTTSService.Settings( voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc", ), ) context = LLMContext() aggregators = LLMContextAggregatorPair( context, user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()), ) bridge = BusBridgeProcessor( bus=runner.bus, worker_name=MAIN_NAME, name=f"{MAIN_NAME}::BusBridge", ) pipeline = Pipeline( [ transport.input(), stt, aggregators.user(), bridge, tts, transport.output(), aggregators.assistant(), ] ) main = PipelineWorker( pipeline, name=MAIN_NAME, params=PipelineParams(enable_metrics=True, enable_usage_metrics=True), ) # The remote greeter may take a moment to register on the bus, so only # activate it once both the client is connected and the greeter is ready. state = {"client_connected": False, "greeter_ready": False} async def maybe_activate(): if not (state["client_connected"] and state["greeter_ready"]): return await main.activate_worker( "greeter", args=LLMWorkerActivationArgs( messages=[{"role": "developer", "content": "Welcome the user."}], ), ) async def on_greeter_ready(_data: WorkerReadyData) -> None: state["greeter_ready"] = True await maybe_activate() await runner.registry.watch("greeter", on_greeter_ready) @transport.event_handler("on_client_connected") async def on_client_connected(transport, client): state["client_connected"] = True await maybe_activate() @transport.event_handler("on_client_disconnected") async def on_client_disconnected(transport, client): await runner.cancel() await runner.add_workers(main) await runner.run() ``` ```bash python main.py --redis-url redis://localhost:6379 ``` ### Process 2 & 3: LLM agents Each LLM agent runs as a standalone process. It connects to the same Redis channel and waits for activation. Subclass `LLMWorker` to host `@tool` methods and pass the LLM service in the constructor: ```python # llm.py import argparse import asyncio import os from redis.asyncio import Redis from pipecat.bus.network.redis import RedisBus from pipecat.workers.runner import WorkerRunner from pipecat.services.llm_service import FunctionCallParams from pipecat.services.openai.llm import OpenAILLMService from pipecat.workers.llm import LLMWorker, LLMWorkerActivationArgs, tool class AcmeLLMAgent(LLMWorker): def __init__(self, name: str, *, system_instruction: str, watch: list[str]): llm = OpenAILLMService( api_key=os.environ["OPENAI_API_KEY"], settings=OpenAILLMService.Settings(system_instruction=system_instruction), ) super().__init__(name, llm=llm, bridged=()) self._watch = watch async def start(self) -> None: """Watch sibling agents so handoff knows when they are available.""" await super().start() await self.watch_workers(*self._watch) @tool(cancel_on_interruption=False) async def transfer_to_agent(self, params: FunctionCallParams, agent: str, reason: str): """Transfer the user to another agent. Args: agent (str): The target agent. reason (str): Transfer reason. """ await self.activate_worker( agent, args=LLMWorkerActivationArgs( messages=[{"role": "developer", "content": reason}] ), deactivate_self=True, result_callback=params.result_callback, ) async def main_async(): parser = argparse.ArgumentParser() parser.add_argument("worker", choices=["greeter", "support"]) parser.add_argument("--redis-url", default="redis://localhost:6379") parser.add_argument("--channel", default="pipecat:my-app") args = parser.parse_args() redis = Redis.from_url(args.redis_url) bus = RedisBus(redis=redis, channel=args.channel) agent = AcmeLLMAgent( args.worker, system_instruction="You are a greeter..." if args.worker == "greeter" else "You are support...", watch=["support"] if args.worker == "greeter" else ["greeter"], ) runner = WorkerRunner(bus=bus, handle_sigint=True) await runner.add_workers(agent) await runner.run() asyncio.run(main_async()) ``` ```bash # Run on Machine B python llm.py greeter --redis-url redis://your-redis-host:6379 # Run on Machine C python llm.py support --redis-url redis://your-redis-host:6379 ``` Each LLM agent watches its sibling so that, when the user is transferred, the target is already known to be available. You can do the same with the `@worker_ready` decorator on a worker subclass to react automatically when a specific agent registers. ## How discovery works Runners exchange registry information automatically over the shared bus. To get notified when an agent is ready, watch it with `runner.registry.watch(...)`, call `watch_workers()` from inside a worker, or use the `@worker_ready` decorator -- they all work the same way locally and distributed. ## Considerations - **Latency**: Network buses add overhead. For latency-sensitive voice applications, keep the main transport agent and its active LLM agent geographically close to each other and the bus server (Redis or PostgreSQL). - **Serialization**: Both `RedisBus` and `PgmqBus` serialize messages to JSON. Custom frame types need to be registered with the serializer. - **Single channel**: All agents on the same channel see all messages. Use different channels for different sessions or applications. ## What's next Distributed agents share one bus. When you need to connect agents across separate buses or networks, bridge them with a proxy. Bridge agents on different buses point-to-point over WebSocket # Proxy Agents Source: https://docs.pipecat.ai/pipecat/learn/proxy-agents.md Connect Pipecat agents running on different buses with proxy agents, extending multi-agent pipelines across processes. ## Overview Proxy agents connect agents running on **different buses**. Unlike [distributed agents](/pipecat/learn/distributed-agents) (which all share the same bus), proxy agents bridge two isolated bus instances point-to-point. This is useful when: - You want to run an LLM agent on a separate server without shared infrastructure - You need fine-grained control over which messages cross the network - You're connecting to a third-party service that hosts agents ## Architecture ![Proxy agents architecture](/images/proxy-agents-architecture.png) Each side has its own `WorkerRunner` and bus. The proxy agents relay messages between the two buses over a WebSocket connection. Like any worker, a proxy gets its bus from the runner when you register it -- you never pass `bus=` to its constructor. ## Client side: WebSocketProxyClient The client connects to a remote server and forwards specific message types: ```python from pipecat.bus import BusFrameMessage from pipecat.workers.proxy.websocket import WebSocketProxyClient proxy = WebSocketProxyClient( "proxy", url="ws://remote-server:8765/ws", local_worker_name="acme", # Agent on this bus remote_worker_name="assistant", # Agent on remote bus forward_messages=(BusFrameMessage,), ) await runner.add_workers(proxy) ``` | Parameter | Description | | -------------------- | ------------------------------------------------------------- | | `url` | WebSocket URL of the remote server | | `local_worker_name` | Name of the local agent that exchanges frames with the remote | | `remote_worker_name` | Name of the remote agent to communicate with | | `forward_messages` | Tuple of message types to forward across the connection | | `headers` | Optional HTTP headers (e.g. authentication tokens) | The proxy connects when activated. Activate it when the client connects, by calling `activate_worker` on the main agent: ```python @transport.event_handler("on_client_connected") async def on_client_connected(transport, client): await main.activate_worker("proxy") ``` ## Server side: WebSocketProxyServer The server accepts WebSocket connections and creates a proxy for each session: ```python from fastapi import FastAPI, WebSocket from pipecat.bus import BusFrameMessage from pipecat.workers.runner import WorkerRunner from pipecat.workers.proxy.websocket import WebSocketProxyServer app = FastAPI() @app.websocket("/ws") async def websocket_endpoint(websocket: WebSocket): await websocket.accept() runner = WorkerRunner(handle_sigint=False) proxy = WebSocketProxyServer( "gateway", websocket=websocket, worker_name="assistant", # Agent on this bus remote_worker_name="acme", # Agent on remote bus forward_messages=(BusFrameMessage,), ) assistant = build_assistant() # an LLMWorker named "assistant" await runner.add_workers(proxy, assistant) await runner.run() ``` Each WebSocket connection gets its own `WorkerRunner`, bus, and set of agents. This isolates sessions from each other. ## Message filtering Proxy agents provide security through message filtering: - Only messages targeted at the configured agent names cross the connection - Broadcast messages (no target) are **not** forwarded - Local-only messages (`BusLocalMessage`) never cross - `forward_messages` controls which message types are allowed This means internal bus traffic stays local. Only the specific message types you opt into are relayed. ## Full example ### Client (main.py) ```python async def run_bot(transport, runner_args): runner = WorkerRunner(handle_sigint=runner_args.handle_sigint) # ... build the main transport pipeline with a BusBridgeProcessor ... main = PipelineWorker(pipeline, name="acme", params=PipelineParams(...)) proxy = WebSocketProxyClient( "proxy", url=runner_args.cli_args.remote_url, local_worker_name="acme", remote_worker_name="assistant", forward_messages=(BusFrameMessage,), ) async def on_assistant_ready(_data): await main.activate_worker( "assistant", args=LLMWorkerActivationArgs( messages=[{"role": "developer", "content": "Welcome the user."}], ), ) await runner.registry.watch("assistant", on_assistant_ready) @transport.event_handler("on_client_connected") async def on_client_connected(transport, client): await main.activate_worker("proxy") @transport.event_handler("on_client_disconnected") async def on_client_disconnected(transport, client): await runner.cancel() await runner.add_workers(proxy, main) await runner.run() ``` ### Server (assistant.py) ```python @app.websocket("/ws") async def websocket_endpoint(websocket: WebSocket): await websocket.accept() runner = WorkerRunner(handle_sigint=False) proxy = WebSocketProxyServer( "gateway", websocket=websocket, worker_name="assistant", remote_worker_name="acme", forward_messages=(BusFrameMessage,), ) @proxy.event_handler("on_client_connected") async def on_client_connected(proxy, client): logger.info("WebSocket client connected") @proxy.event_handler("on_client_disconnected") async def on_client_disconnected(proxy, client): await runner.cancel() assistant = AcmeAssistant() # an LLMWorker named "assistant" await runner.add_workers(proxy, assistant) await runner.run() ``` ### Running ```bash # Terminal 1: Start the LLM server python assistant.py --port 8765 # Terminal 2: Start the transport client python main.py --remote-url ws://localhost:8765/ws ``` Install the WebSocket extra: `uv add "pipecat-ai[websocket]"` ## Proxy agents vs distributed agents | | Proxy Agents (different buses) | Distributed (same bus) | | --------------------- | ------------------------------------- | ------------------------------ | | **Topology** | Point-to-point | Many-to-many | | **Session isolation** | Each connection is isolated | All agents share a channel | | **Use case** | Separate networks, third-party agents | Scaling agents across machines | ## What's next You've built a multi-agent system and scaled it across processes, machines, and networks. Here's where to go from here. Explore Fundamentals and advanced patterns # What's Next Source: https://docs.pipecat.ai/pipecat/learn/whats-next.md Continue your Pipecat journey with advanced features, examples, and production deployment ## You've Mastered Voice AI Pipelines! 🎉 Congratulations! You've learned how to build complete voice AI applications with Pipecat. You're now well on your way to understanding pipelines, processors, transports, and all the components needed to create sophisticated conversational AI. ## Choose Your Path **Production-ready applications** - Dive into 30+ complete examples including multimodal bots, creative applications, and enterprise integrations. **Master advanced features** - Explore specialized topics like telephony, deployment, custom processors, and production optimization. ## 💬 Join the Community Get involved with the Pipecat community to share your projects, get support, and contribute to the ecosystem: - **[Discord Community](https://discord.gg/pipecat)**: Get support, share projects, and connect with other developers - **[GitHub](https://github.com/pipecat-ai/pipecat)**: Contribute to the project and report issues - **[Examples Repository](https://github.com/pipecat-ai/pipecat-examples)**: Community-contributed applications # Service Settings Source: https://docs.pipecat.ai/pipecat/fundamentals/service-settings.md Learn how to configure and update AI service settings at initialization and runtime. Settings are the runtime-configurable properties of AI services, including things like the model, voice, language, temperature, and other provider-specific options. Every service exposes a `Settings` class that you can use in two ways: 1. **Configure at initialization**: pass a `settings=` argument when constructing a service. 2. **Update at runtime**: push an `*UpdateSettingsFrame` through the pipeline to change settings mid-conversation. ## Configuring settings at initialization Pass a `settings=` argument to any service constructor using that service's `Settings` class: ```python from pipecat.services.openai.llm import OpenAILLMService llm = OpenAILLMService( api_key=os.getenv("OPENAI_API_KEY"), settings=OpenAILLMService.Settings( model="gpt-4o", temperature=0.7, system_instruction="You are a helpful assistant.", ), ) ``` ```python from pipecat.services.cartesia.tts import CartesiaTTSService tts = CartesiaTTSService( api_key=os.getenv("CARTESIA_API_KEY"), settings=CartesiaTTSService.Settings( voice="71a7ad14-091c-4e8e-a314-022ece01c121", ), ) ``` ```python from pipecat.services.deepgram.stt import DeepgramSTTService stt = DeepgramSTTService( api_key=os.getenv("DEEPGRAM_API_KEY"), settings=DeepgramSTTService.Settings( model="nova-3", language="en", smart_format=True, ), ) ``` You only need to specify the settings you want to override. Everything else uses the service's defaults. ## Common settings by service type Each service type has a base set of settings. Individual services may extend these with provider-specific fields. ### LLM settings | Setting | Type | Description | | -------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `model` | `str` | Model identifier (e.g. `"gpt-4o"`, `"claude-sonnet-4-5-20250929"`) | | `system_instruction` | `str` | System prompt for the model. See [Context Management](/pipecat/learn/context-management#system-instruction-developer-messages-and-system-messages) for how this interacts with developer messages and context system messages. | | `temperature` | `float` | Sampling temperature | | `max_tokens` | `int` | Maximum tokens to generate | | `top_p` | `float` | Nucleus sampling probability | | `top_k` | `int` | Top-k sampling parameter | | `frequency_penalty` | `float` | Frequency penalty | | `presence_penalty` | `float` | Presence penalty | | `seed` | `int` | Random seed for reproducibility | ### TTS settings | Setting | Type | Description | | ---------- | ----------------- | ----------------------------- | | `model` | `str` | TTS model identifier | | `voice` | `str` | Voice identifier or name | | `language` | `Language \| str` | Language for speech synthesis | ### STT settings | Setting | Type | Description | | ---------- | ----------------- | ------------------------------- | | `model` | `str` | STT model identifier | | `language` | `Language \| str` | Language for speech recognition | Individual services extend these base settings with provider-specific fields. For example, Deepgram STT adds `endpointing`, `smart_format`, `diarize`, and more. See the individual [service documentation](/api-reference/server/services/supported-services) for the full list of available settings. ## Updating settings at runtime You can change service settings while the pipeline is running by pushing an update settings frame. This is useful for scenarios like switching languages, changing voices, or adjusting LLM parameters mid-conversation. Use these frame types: | Frame | Target | | ------------------------ | ------------ | | `LLMUpdateSettingsFrame` | LLM services | | `TTSUpdateSettingsFrame` | TTS services | | `STTUpdateSettingsFrame` | STT services | Only include the fields you want to change. Unspecified fields are left as-is. ```python from pipecat.frames.frames import TTSUpdateSettingsFrame # Change TTS voice mid-conversation await worker.queue_frame( TTSUpdateSettingsFrame( delta=CartesiaTTSService.Settings(voice="new-voice-id") ) ) ``` ```python from pipecat.frames.frames import LLMUpdateSettingsFrame # Lower the temperature for more deterministic responses await worker.queue_frame( LLMUpdateSettingsFrame( delta=OpenAILLMService.Settings(temperature=0.2) ) ) ``` ```python from pipecat.frames.frames import STTUpdateSettingsFrame from pipecat.transcriptions.language import Language # Switch STT language to Spanish await worker.queue_frame( STTUpdateSettingsFrame( delta=DeepgramSTTService.Settings(language=Language.ES) ) ) ``` Update settings frames are uninterruptible. They will always be processed even if a user interruption occurs. ## Service-specific settings Services can extend the base settings with provider-specific fields. For example: - **Cartesia TTS** adds `generation_config` (volume, speed, emotion) and `pronunciation_dict_id` - **OpenAI TTS** adds `instructions` and `speed` - **Deepgram STT** adds `endpointing`, `smart_format`, `diarize`, `punctuate`, and many more - **OpenAI LLM** adds `max_completion_tokens` These service-specific fields work the same way — set them at initialization or update them at runtime: ```python from pipecat.services.cartesia.tts import CartesiaTTSService, GenerationConfig tts = CartesiaTTSService( api_key=os.getenv("CARTESIA_API_KEY"), settings=CartesiaTTSService.Settings( voice="71a7ad14-091c-4e8e-a314-022ece01c121", generation_config=GenerationConfig(speed=1.2, emotion="excited"), ), ) ``` See the individual service documentation for a complete list of available settings: ## Passing extra parameters Every Settings class includes an `extra` dict for passing provider-specific parameters that Pipecat doesn't have a dedicated field for. This is useful when a provider supports options that haven't been explicitly added to the Settings dataclass yet: ```python llm = OpenAILLMService( api_key=os.getenv("OPENAI_API_KEY"), settings=OpenAILLMService.Settings( model="gpt-4o", extra={"logprobs": True, "top_logprobs": 5}, ), ) ``` Values in `extra` are passed through to the underlying API call. You can also update `extra` at runtime: ```python await worker.queue_frame( LLMUpdateSettingsFrame( delta=OpenAILLMService.Settings( extra={"logprobs": False}, ) ) ) ``` The `extra` dict is merged on updates — new keys are added and existing keys are overwritten, but keys not present in the delta are left unchanged. ## Migration from InputParams The `InputParams` / `params=` pattern is deprecated as of v0.0.105. Use `Settings` / `settings=` instead. **Before:** ```python llm = OpenAILLMService( model="gpt-4o", params=OpenAILLMService.InputParams( temperature=0.7, ), ) ``` **After:** ```python llm = OpenAILLMService( settings=OpenAILLMService.Settings( model="gpt-4o", temperature=0.7, ), ) ``` Note that `model` has moved from a top-level constructor argument into `Settings`. Both the old and new patterns still work during the deprecation period, but `settings` values take precedence when both are provided. # Interruptions Source: https://docs.pipecat.ai/pipecat/fundamentals/interruptions.md How Pipecat stops the bot when the user speaks, what happens to in-flight LLM and TTS output, and how to control it. Interruptions (also called barge-in) let the user talk over the bot. When the user starts speaking while the bot is talking, the bot stops immediately, in-flight work is cancelled, and the pipeline is ready for the new user input. Interruptions are **enabled by default**. This page explains how they work under the hood, what ends up in the conversation context, and how to configure or trigger them yourself. Looking for how Pipecat decides *when* a user turn starts and ends? See [Speech Input & Turn Detection](/pipecat/learn/speech-input) and [User Turn Strategies](/api-reference/server/utilities/turn-management/user-turn-strategies). This page covers what happens *after* an interruption is triggered. ## What happens when the user interrupts When a [user turn start strategy](/api-reference/server/utilities/turn-management/user-turn-strategies#start-strategies) triggers with `enable_interruptions=True` (the default), the user aggregator broadcasts an `InterruptionFrame` both upstream and downstream through the pipeline. From there: `InterruptionFrame` is a [SystemFrame](/api-reference/server/frames/system-frames), so each processor handles it immediately instead of waiting behind queued frames. Each processor cancels its processing task and discards queued `DataFrame`s and `ControlFrame`s. Frames marked uninterruptible (like `FunctionCallResultFrame` and `EndFrame`) are preserved and still processed. The in-flight LLM completion is cancelled mid-stream. Any registered function calls with `cancel_on_interruption=True` are cancelled and emit a `FunctionCallCancelFrame`. See [Function Calling](/pipecat/learn/function-calling) for details. The TTS service stops synthesizing, clears its text aggregation and word timestamps, and drops pending output. The output transport drains its audio queue, discarding audio that was generated but not yet played. If a background audio mixer is active, the transport drains only the bot's speech so the background audio keeps playing without a gap. The result: the bot goes silent within roughly one audio write, and the pipeline is clean and ready for the user's new turn. ## What ends up in the context A common question: if the bot is cut off mid-sentence, what does the LLM context contain? **Only the words that were actually spoken.** As the bot speaks, the output transport pushes `TTSTextFrame`s downstream in sync with audio playback. Text that never played never reaches the assistant context aggregator. On interruption, the aggregator commits the partial, spoken-so-far text to the context as the assistant message. This means the LLM's next completion sees an accurate transcript of the conversation: the bot's message ends where the user cut it off, not where the LLM's generation ended. You can observe this with the `on_assistant_turn_stopped` event, which reports the committed text and whether the turn was interrupted: ```python @assistant_aggregator.event_handler("on_assistant_turn_stopped") async def on_assistant_turn_stopped(aggregator, message): if message.interrupted: print(f"Bot was cut off after saying: {message.content}") ``` ## Controlling interruptions **Let the bot finish speaking** with [user input muting](/pipecat/fundamentals/user-input-muting). Mute strategies block user audio, transcriptions, and interruption signals while active, so speech over the bot is discarded rather than answered later. Use `AlwaysUserMuteStrategy` to mute whenever the bot is speaking, or `FirstSpeechUserMuteStrategy` to protect just the introduction: ```python from pipecat.processors.aggregators.llm_response_universal import ( LLMContextAggregatorPair, LLMUserAggregatorParams, ) from pipecat.turns.user_mute import AlwaysUserMuteStrategy user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, user_params=LLMUserAggregatorParams( user_mute_strategies=[AlwaysUserMuteStrategy()], ), ) ``` **Require a minimum number of words** so short utterances like "okay" or "yeah" don't interrupt the bot. This is configured on the [user turn start strategy](/api-reference/server/utilities/turn-management/user-turn-strategies#start-strategies): ```python from pipecat.turns.user_start import MinWordsUserTurnStartStrategy start_strategy = MinWordsUserTurnStartStrategy(min_words=3) ``` **Filter out backchannels with a model.** The [Krisp VIVA Interruption Prediction strategy](/api-reference/server/utilities/turn-management/user-turn-strategies#krispvivaipuserturnstartstrategy) distinguishes genuine interruptions from acknowledgments like "uh-huh". **Disable interruptions** so in-flight work is never cancelled: ```python from pipecat.turns.user_start import VADUserTurnStartStrategy start_strategy = VADUserTurnStartStrategy(enable_interruptions=False) ``` Disabling interruptions does not ignore the user. Speech over the bot is still transcribed and processed as a normal user turn — the bot's reply is queued and plays as soon as the current speech finishes. If you want the bot to finish speaking *and* discard what the user said over it, use mute strategies instead. ## Triggering an interruption yourself Sometimes the bot should stop itself: a timeout fires, an external event arrives, or your own logic decides the current response is no longer relevant. From inside a custom `FrameProcessor`, broadcast the interruption directly: ```python await self.broadcast_interruption() ``` From code that has a reference to a processor or the worker, you can also push an `InterruptionWorkerFrame`. The pipeline worker converts it into an `InterruptionFrame` and sends it through the whole pipeline: ```python from pipecat.frames.frames import InterruptionWorkerFrame await worker.queue_frame(InterruptionWorkerFrame()) ``` Both approaches run the same interruption flow described above: the bot stops speaking, in-flight work is cancelled, and the spoken-so-far text is committed to context. ## Speech-to-speech services Realtime speech-to-speech services (like Gemini Live and OpenAI Realtime) handle interruption detection on the provider side. In these pipelines, the service emits the speaking events and Pipecat's aggregators follow along using [external turn strategies](/api-reference/server/utilities/turn-management/user-turn-strategies#externaluserturnstartstrategy). The provider decides when the user barged in; Pipecat still flushes local audio output so the bot goes silent right away. ## Related - [Speech Input & Turn Detection](/pipecat/learn/speech-input) - how user turns are detected - [User Turn Strategies](/api-reference/server/utilities/turn-management/user-turn-strategies) - full strategy reference - [User Input Muting](/pipecat/fundamentals/user-input-muting) - suppress user input while the bot speaks - [System Frames](/api-reference/server/frames/system-frames) - `InterruptionFrame` reference # User Input Muting Source: https://docs.pipecat.ai/pipecat/fundamentals/user-input-muting.md Learn how to control when user speech is processed in your conversational bot ## Overview In conversational applications, there are moments when you don't want to process user speech, such as during bot introductions or while executing function calls. Pipecat's user mute strategies let you selectively "mute" user input based on different conversation states. ## When to Use Mute Strategies Common scenarios for muting user input include: - **During introductions**: Prevent the bot from being interrupted during its initial greeting - **While processing functions**: Block input while the bot is retrieving external data - **During bot speech**: Reduce false transcriptions while the bot is speaking - **For guided conversations**: Create more structured interactions with clear turn-taking ## How It Works User mute strategies work by blocking specific user-related frames from flowing through your pipeline. When muted, the following frames are filtered: - Voice activity detection (VAD) events - Interruption signals - Raw audio input frames - Transcription frames (both interim and final) This prevents user speech from being processed during muted periods. Mute strategies are configured on the `LLMUserAggregator` via the `user_mute_strategies` parameter. ## Mute Strategies Pipecat provides several built-in strategies for determining when to mute user input: Mute only during the bot's first speech utterance. Useful for introductions when you want the bot to complete its greeting before the user can speak. Start muted and remain muted until the first bot utterance completes. Ensures the bot's initial instructions are fully delivered. Mute during function calls. Prevents users from speaking while the bot is processing external data requests. Mute whenever the bot is speaking. Creates a strict turn-taking conversation pattern. The `FirstSpeechUserMuteStrategy` and `MuteUntilFirstBotCompleteUserMuteStrategy` strategies should not be used together as they handle the first bot speech differently. ## Basic Implementation Import and configure the mute strategies you need: ```python from pipecat.processors.aggregators.llm_response_universal import ( LLMContextAggregatorPair, LLMUserAggregatorParams, ) from pipecat.turns.user_mute import AlwaysUserMuteStrategy # Configure with one or more strategies user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, user_params=LLMUserAggregatorParams( user_mute_strategies=[AlwaysUserMuteStrategy()], ), ) ``` ## Combining Multiple Strategies Multiple strategies can be combined. They use OR logic—if **any** strategy indicates the user should be muted, input is suppressed: ```python from pipecat.processors.aggregators.llm_response_universal import ( LLMContextAggregatorPair, LLMUserAggregatorParams, ) from pipecat.turns.user_mute import ( MuteUntilFirstBotCompleteUserMuteStrategy, FunctionCallUserMuteStrategy, ) user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, user_params=LLMUserAggregatorParams( user_mute_strategies=[ MuteUntilFirstBotCompleteUserMuteStrategy(), # Mute until first response FunctionCallUserMuteStrategy(), # Mute during function calls ], ), ) ``` ## Building Custom Strategies Subclass `BaseUserMuteStrategy` (in `pipecat.turns.user_mute`) when none of the built-in strategies fit. A strategy only needs to answer one question per frame: should the user be muted right now? Override `process_frame(self, frame: Frame) -> bool` to update internal state and return the current mute decision. ### Which frames reach a strategy Each strategy's `process_frame` is called for every frame that passes through the user aggregator, **except** `StartFrame`, `EndFrame`, and `CancelFrame`. This includes: - User-direction frames from the input transport and STT: `TranscriptionFrame`, `InterimTranscriptionFrame`, `UserStartedSpeakingFrame`, `UserStoppedSpeakingFrame`, `VADUserStartedSpeakingFrame`, `VADUserStoppedSpeakingFrame`, `InputAudioRawFrame`, `InterruptionFrame` - Bot and function-calling lifecycle frames from elsewhere in the pipeline: `BotStartedSpeakingFrame`, `BotStoppedSpeakingFrame`, `FunctionCallsStartedFrame`, `FunctionCallResultFrame`, `FunctionCallCancelFrame` Frames that don't naturally reach the user aggregator (for example `LLMTextFrame` or `TTSTextFrame`, which flow downstream from the LLM or TTS) won't be seen by a strategy directly. To react to those signals, place a companion `FrameProcessor` where the frames do flow and have it toggle state on your strategy. See [Toggling a strategy at runtime](#toggling-a-strategy-at-runtime) below. ### Which frames get suppressed when muted Returning `True` from your strategy sets the aggregator's mute state. While muted, only these frame types are actually dropped: - `InterruptionFrame` - `VADUserStartedSpeakingFrame`, `VADUserStoppedSpeakingFrame` - `UserStartedSpeakingFrame`, `UserStoppedSpeakingFrame` - `InputAudioRawFrame` - `InterimTranscriptionFrame`, `TranscriptionFrame` All other frames continue to flow so the rest of the pipeline keeps functioning. ### Toggling a strategy at runtime Strategies are plain Python objects. Anything that holds a reference to one can flip its state between frames, which means a companion processor placed elsewhere in the pipeline can drive the mute decision based on signals the strategy can't observe directly (LLM text, tool results, external events). This example strategy adds its own `enable`/`disable` methods (not part of the base contract) and returns their state from `process_frame`: ```python from pipecat.frames.frames import Frame from pipecat.turns.user_mute import BaseUserMuteStrategy class ToggleableUserMuteStrategy(BaseUserMuteStrategy): def __init__(self): super().__init__() self._muted = False def enable(self): self._muted = True def disable(self): self._muted = False async def process_frame(self, frame: Frame) -> bool: await super().process_frame(frame) return self._muted ``` A companion processor watches for the trigger and toggles the strategy: ```python from pipecat.frames.frames import ( BotStartedSpeakingFrame, BotStoppedSpeakingFrame, Frame, LLMTextFrame, ) from pipecat.processors.frame_processor import FrameDirection, FrameProcessor class DisclaimerGuardProcessor(FrameProcessor): def __init__(self, strategy: ToggleableUserMuteStrategy, trigger_phrase: str, **kwargs): super().__init__(**kwargs) self._strategy = strategy self._trigger = trigger_phrase # Keep a small sliding window so cross-frame matches work without # the buffer growing unbounded if the trigger never appears. self._max_buffer = max(len(trigger_phrase) * 4, 512) self._buffer = "" self._active = False async def process_frame(self, frame: Frame, direction: FrameDirection): await super().process_frame(frame, direction) if isinstance(frame, BotStartedSpeakingFrame): # Start each bot turn with a fresh buffer. self._buffer = "" elif isinstance(frame, LLMTextFrame) and direction == FrameDirection.DOWNSTREAM: self._buffer = (self._buffer + frame.text)[-self._max_buffer :] if not self._active and self._trigger in self._buffer: self._active = True self._strategy.enable() elif isinstance(frame, BotStoppedSpeakingFrame) and self._active: self._active = False self._buffer = "" self._strategy.disable() await self.push_frame(frame, direction) ``` Wire them together by passing the same strategy instance to both the aggregator and the processor: ```python mute_strategy = ToggleableUserMuteStrategy() user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, user_params=LLMUserAggregatorParams(user_mute_strategies=[mute_strategy]), ) disclaimer_guard = DisclaimerGuardProcessor( strategy=mute_strategy, trigger_phrase="Please read the following disclosure", ) pipeline = Pipeline([ transport.input(), stt, user_aggregator, llm, disclaimer_guard, # positioned where LLMTextFrame flows downstream tts, transport.output(), assistant_aggregator, ]) ``` ## Responding to Mute Events You can register event handlers to be notified when muting starts or stops. This is particularly useful for providing visual feedback to users: ```python @user_aggregator.event_handler("on_user_mute_started") async def on_user_mute_started(aggregator): logger.info("User mute started") # Send a visual indicator to your client # e.g., show a "Bot is speaking" indicator @user_aggregator.event_handler("on_user_mute_stopped") async def on_user_mute_stopped(aggregator): logger.info("User mute stopped") # Update your client UI # e.g., show a "You can speak now" indicator ``` These events fire whenever the mute state changes, allowing you to keep your UI synchronized with the bot's state. ### RTVI Events When mute strategies activate or deactivate, the server automatically sends RTVI messages (`user-mute-started` and `user-mute-stopped`) to the client. You can listen for these in the JavaScript client to update your UI: The client should continue sending audio normally during mute. These events are purely informational — muting happens server-side. ```typescript JavaScript import { PipecatClient, RTVIEvent } from "@pipecat-ai/client-js"; const pcClient = new PipecatClient({ callbacks: { onUserMuteStarted: () => { // Show a visual indicator that the bot is not listening // e.g., disable a microphone button or show "Bot is speaking..." }, onUserMuteStopped: () => { // Remove the indicator, show the user they can speak }, }, }); // Or using event listeners pcClient.on(RTVIEvent.UserMuteStarted, () => { console.log("Server is ignoring user audio"); }); pcClient.on(RTVIEvent.UserMuteStopped, () => { console.log("Server is listening to user audio again"); }); ``` ## Best Practices - **Choose strategies wisely**: Select the minimal set of strategies needed for your use case - **Test user experience**: Excessive muting can frustrate users; balance control with usability - **Provide feedback**: Use the mute event handlers to show visual cues when the user is muted to improve the experience ## Next Steps Read the complete API reference documentation for all available mute strategies and their behavior. Learn how to configure turn detection behavior for more control over conversation flow. Experiment with different muting strategies to find the right balance for your application. # Detecting Idle Users Source: https://docs.pipecat.ai/pipecat/fundamentals/detecting-user-idle.md Learn how to detect and respond when users are inactive in conversations ## Overview In conversational applications, it's important to handle situations where users go silent or inactive. Pipecat provides built-in idle detection through `LLMUserAggregator` and `UserTurnProcessor`, allowing your bot to respond appropriately when users haven't spoken for a defined period. ## How It Works Idle detection monitors user activity and: 1. Starts a timer when the bot finishes speaking (`BotStoppedSpeakingFrame`) 2. Cancels the timer when the user or bot starts speaking 3. Suppresses the timer during function calls and active user turns (to avoid false triggers during interruptions) 4. Emits an `on_user_turn_idle` event when the timer expires 5. Allows you to implement escalating responses or gracefully end the conversation in your application code ## Basic Implementation ### Step 1: Enable Idle Detection Enable idle detection by setting the `user_idle_timeout` parameter when creating your aggregator: ```python from pipecat.processors.aggregators.llm_response_universal import ( LLMContextAggregatorPair, LLMUserAggregatorParams, ) user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, user_params=LLMUserAggregatorParams( user_idle_timeout=5.0, # Detect idle after 5 seconds ), ) ``` ### Step 2: Handle Idle Events Create an event handler to respond when the user becomes idle: ```python @user_aggregator.event_handler("on_user_turn_idle") async def on_user_turn_idle(aggregator): # Send a reminder to the user message = { "role": "developer", "content": "The user has been quiet. Politely ask if they're still there.", } await aggregator.push_frame(LLMMessagesAppendFrame([message], run_llm=True)) ``` ### Step 3: Implement Retry Logic (Optional) For escalating responses, track retry count in your application: ```python class IdleHandler: def __init__(self): self._retry_count = 0 def reset(self): self._retry_count = 0 async def handle_idle(self, aggregator): self._retry_count += 1 if self._retry_count == 1: # First attempt - gentle reminder message = { "role": "developer", "content": "The user has been quiet. Politely ask if they're still there.", } await aggregator.push_frame(LLMMessagesAppendFrame([message], run_llm=True)) elif self._retry_count == 2: # Second attempt - more direct message = { "role": "developer", "content": "The user is still inactive. Ask if they'd like to continue.", } await aggregator.push_frame(LLMMessagesAppendFrame([message], run_llm=True)) else: # Third attempt - end conversation await aggregator.push_frame( TTSSpeakFrame("It seems like you're busy. Have a nice day!") ) await aggregator.push_frame(EndWorkerFrame(), FrameDirection.UPSTREAM) # Use the handler idle_handler = IdleHandler() @user_aggregator.event_handler("on_user_turn_idle") async def on_user_turn_idle(aggregator): await idle_handler.handle_idle(aggregator) @user_aggregator.event_handler("on_user_turn_started") async def on_user_turn_started(aggregator, strategy): idle_handler.reset() # Reset retry count when user speaks ``` ## Updating Timeout at Runtime You can enable, disable, or change the idle timeout at runtime by pushing a `UserIdleTimeoutUpdateFrame`: ```python from pipecat.frames.frames import UserIdleTimeoutUpdateFrame # Enable idle detection (or change timeout) await worker.queue_frame(UserIdleTimeoutUpdateFrame(timeout=10.0)) # Disable idle detection await worker.queue_frame(UserIdleTimeoutUpdateFrame(timeout=0)) ``` **Immediate application**: Timeout updates take effect immediately. If an idle timer is currently running, it restarts with the new duration. If the bot is waiting for the user to speak and you enable a positive timeout (e.g., upgrading from `0` to `10.0`), the timer arms right away without waiting for the next bot turn. This is useful when you want to enable idle detection only at certain points in the conversation, or adjust the timeout based on context. ## Best Practices - **Set appropriate timeouts**: Shorter timeouts (5-10 seconds) work well for voice conversations - **Use escalating responses**: Start with gentle reminders and gradually become more direct - **Limit retry attempts**: After 2-3 unsuccessful attempts, consider [ending the conversation](/pipecat/learn/pipeline-termination) gracefully by pushing an `EndWorkerFrame` - **Reset on user activity**: Use the `on_user_turn_started` event to reset your retry counter when the user speaks - **Let the LLM respond naturally**: Use developer messages to prompt the LLM rather than hardcoded TTS responses for more natural interactions ## Next Steps Explore a complete working example that demonstrates how to detect and respond to user inactivity in Pipecat. Learn about all available turn events and their parameters. Implementing idle user detection improves the conversational experience by ensuring your bot can handle periods of user inactivity gracefully, either by prompting for re-engagement or politely ending the conversation when appropriate. # STT Latency Tuning Source: https://docs.pipecat.ai/pipecat/fundamentals/stt-latency-tuning.md Measure and tune STT latency in Pipecat to improve turn detection timing and end-of-turn responsiveness. ## What is TTFS? **Time To Final Segment (TTFS)** measures how long it takes from the moment a user stops speaking until the STT service delivers the final transcript. This latency directly affects how long your bot waits before it starts responding. ``` User stops → [TTFS latency] → Final transcript arrives → Bot starts ``` Every STT service has a different TTFS profile based on its architecture, model complexity, and infrastructure. Pipecat ships with measured P99 latency values for each supported service so that turn detection can account for this delay automatically. Measured values were benchmarked using the [stt-benchmark](https://github.com/pipecat-ai/stt-benchmark) tool. ## Why TTFS matters TTFS feeds directly into [turn stop strategies](/api-reference/server/utilities/turn-management/user-turn-strategies), which decide when the user has finished speaking and the bot should respond. - **Value too low**: The turn stop strategy gives up waiting before the final transcript arrives. The bot responds based on incomplete text, or misses the user's input entirely. - **Value too high**: The bot waits longer than necessary after the user stops speaking, creating awkward pauses in the conversation. - **Value just right**: The bot waits long enough for the transcript to arrive, then responds immediately. Getting TTFS right is one of the most impactful tuning knobs for perceived conversation responsiveness. TTFS is a configuration value the turn stop strategy uses, not a metric that is logged at runtime. To observe live latency in your running bot (time to first byte, user-to-bot latency), see the [Metrics guide](/pipecat/fundamentals/metrics). ## Default P99 latency values Pipecat includes measured P99 TTFS values for every supported STT service. These are used automatically when you create a service — no configuration required. For the current P99 TTFS value for each service, see [`stt_latency.py`](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/services/stt_latency.py). These values are refined and added to often, so the source file is the source of truth. These built-in values were all measured with `VADParams.stop_secs=0.2`, the recommended default. If you change `stop_secs`, the built-in value no longer matches your setup and Pipecat logs a warning. Re-run the benchmark with your VAD settings and pass the measured value to your STT service constructor. See the [Stop Strategies section](/api-reference/server/utilities/turn-management/user-turn-strategies) for the full explanation of how this interacts with turn detection. Local services (NVIDIA, Whisper) default to 1.0s since actual latency depends entirely on your hardware. Always measure and override for local deployments. Turn-based STT services (for example `CartesiaTurnsSTTService` and `DeepgramFluxSTTService`) have no meaningful TTFS value. The server defines the turn boundary directly, so there is no separate "speech end to final transcript" interval to measure. ## Measuring latency for your deployment The default values are measured under standard conditions, but your actual latency depends on: - **Network distance** to the STT provider - **Region** where the service is hosted - **Service configuration** (model size, language, features enabled) - **Audio quality** and encoding settings Use the [stt-benchmark](https://github.com/pipecat-ai/stt-benchmark) tool to measure TTFS for your specific setup. The tool sends standardized audio samples to your STT service and reports P50, P90, and P99 latency values. To run it, clone the repo and install with [`uv`](https://docs.astral.sh/uv/): ```bash git clone https://github.com/pipecat-ai/stt-benchmark cd stt-benchmark uv sync # Add your provider API keys cp env.example .env # Download standardized audio samples uv run stt-benchmark download --num-samples 100 # Run the benchmark for one or more services uv run stt-benchmark run --services deepgram,openai # View the P50/P90/P99 report uv run stt-benchmark report --service deepgram ``` If you run your bot with a non-default VAD setting, match the benchmark to it with `--vad-stop-secs` so the measured value reflects your configuration: ```bash uv run stt-benchmark run --services deepgram --vad-stop-secs 0.3 ``` See the [stt-benchmark README](https://github.com/pipecat-ai/stt-benchmark) for the full command reference. ## Overriding the default value Pass the `ttfs_p99_latency` parameter to any STT service constructor to override the built-in default: ```python from pipecat.services.deepgram.stt import DeepgramSTTService # Use a measured value from your deployment stt = DeepgramSTTService( api_key=os.getenv("DEEPGRAM_API_KEY"), ttfs_p99_latency=0.45, # Override with your measured P99 ) ``` This value is broadcast to the pipeline via an `STTMetadataFrame` at startup, so turn stop strategies automatically adjust their timing. If you're deploying to a specific region or using a self-hosted STT service, always measure and override the default TTFS value. Even small differences (e.g., 0.35s vs 0.55s) can noticeably affect conversation responsiveness. # Context Summarization Source: https://docs.pipecat.ai/pipecat/fundamentals/context-summarization.md Automatically compress older conversation history in long-running Pipecat conversations to manage LLM token usage and cost. ## Overview In long-running voice AI conversations, context grows with every exchange. This increases token usage, raises costs, and can eventually hit context window limits. Pipecat includes built-in context summarization that automatically compresses older conversation history while preserving recent messages and important context. ## How It Works Context summarization automatically triggers when **either** condition is met: - **Token limit reached**: Context size exceeds `max_context_tokens` (estimated using ~4 characters per token) - **Message count reached**: Number of new messages exceeds `max_unsummarized_messages` You can disable either threshold by setting it to `None`, but at least one must remain active. Summarization always generates a summary and cannot be reduced to pure truncation. When triggered, the system: 1. Sends a `LLMContextSummaryRequestFrame` to the LLM service 2. The LLM generates a concise summary of older messages 3. Context is reconstructed as: `[system_message (if present)] + [summary] + [recent_messages]` 4. Incomplete function call sequences and recent messages are preserved Context summarization is asynchronous and happens in the background without blocking the pipeline. The system uses request IDs to match summary requests with results and handles interruptions gracefully. ## Enabling Context Summarization Enable summarization by setting `enable_auto_context_summarization=True` in `LLMAssistantAggregatorParams`: ```python from pipecat.processors.aggregators.llm_response_universal import ( LLMAssistantAggregatorParams, LLMContextAggregatorPair, ) # Create aggregators with summarization enabled user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, assistant_params=LLMAssistantAggregatorParams( enable_auto_context_summarization=True, ), ) ``` Automatic summarization is **disabled by default** (`enable_auto_context_summarization=False`). When enabled with the default configuration, summarization triggers at 8000 estimated tokens or after 20 new messages, whichever comes first. ## Customizing Behavior Use `LLMAutoContextSummarizationConfig` and `LLMContextSummaryConfig` to tune the summarization triggers and output: ```python from pipecat.utils.context.llm_context_summarization import ( LLMAutoContextSummarizationConfig, LLMContextSummaryConfig, ) user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, assistant_params=LLMAssistantAggregatorParams( enable_auto_context_summarization=True, auto_context_summarization_config=LLMAutoContextSummarizationConfig( max_context_tokens=4000, # Trigger at 4000 tokens max_unsummarized_messages=10, # Or trigger after 10 new messages summary_config=LLMContextSummaryConfig( target_context_tokens=3000, # Target summary size min_messages_after_summary=2, # Keep last 2 messages uncompressed ), ), ), ) ``` See the [reference page](/api-reference/server/utilities/context-summarization) for all available configuration parameters. ## What Gets Preserved Context summarization intelligently preserves: - **System messages**: If the first message (`messages[0]`) is a system message, it is preserved as the initial system prompt. Mid-conversation system messages (e.g., idle notifications or context injections) are treated as regular messages and included in the summarization range. When using [`system_instruction`](/pipecat/learn/context-management#using-system_instruction-recommended) in LLM Settings instead, the system prompt is not part of the context messages and is automatically prepended by the service on each request, so there is nothing to preserve in the context. - **Recent messages**: The last N messages stay uncompressed (configured by `min_messages_after_summary`) - **Function call sequences**: Incomplete function call/result pairs are not split during summarization - **Developer messages are NOT preserved**: Developer messages (`"role": "developer"`) are included in the summarization range like any other message and may be compressed or dropped. If instructions need to survive summarization, use [`system_instruction`](/pipecat/learn/context-management#using-system_instruction-recommended) instead. ## Custom Summarization Prompts You can override the default summarization prompt to control how the LLM generates summaries: ```python custom_prompt = """Summarize this conversation concisely. Focus on: key decisions, user preferences, and action items. Keep the summary under {target_tokens} tokens.""" config = LLMAutoContextSummarizationConfig( summary_config=LLMContextSummaryConfig( summarization_prompt=custom_prompt, ), ) ``` When no custom prompt is provided, Pipecat uses a built-in prompt that instructs the LLM to create a concise summary preserving key information, user preferences, and conversation flow. ## Dedicated Summarization LLM By default, summarization uses the same LLM service that handles conversation. You can route summarization to a separate, cheaper model by setting the `llm` field: ```python from pipecat.services.google import GoogleLLMService # Use a fast/cheap model for summarization summarization_llm = GoogleLLMService( api_key=os.getenv("GOOGLE_API_KEY"), model="gemini-2.5-flash", ) config = LLMAutoContextSummarizationConfig( summary_config=LLMContextSummaryConfig( llm=summarization_llm, ), ) ``` When a dedicated LLM is configured, summarization requests bypass the pipeline entirely and call the dedicated service directly, so the primary conversation LLM is never interrupted. ## On-Demand Summarization In addition to automatic summarization, you can trigger context summarization on demand by pushing an `LLMSummarizeContextFrame` into the pipeline. This is useful when you want to give users explicit control over when summarization happens — for example, via a function call tool. ```python from pipecat.frames.frames import LLMSummarizeContextFrame from pipecat.services.llm_service import FunctionCallParams async def summarize_conversation(params: FunctionCallParams): """Summarize and compress the conversation history. Call this when the user asks you to summarize the conversation or when you want to free up context space.""" await params.result_callback({"status": "summarization_requested"}) await params.llm.queue_frame(LLMSummarizeContextFrame()) ``` Above, `summarize_conversation` is a [direct function](/pipecat/learn/function-calling#1-define-a-tool). List it in the context's `tools` so the LLM can invoke it when the user asks to summarize: ```python context = LLMContext(messages, tools=[summarize_conversation]) ``` On-demand summarization works even when `enable_auto_context_summarization` is `False` — the summarizer is always created internally to handle manually pushed frames. You can also pass a per-request `LLMContextSummaryConfig` to override the default settings: ```python from pipecat.utils.context.llm_context_summarization import LLMContextSummaryConfig await llm.queue_frame( LLMSummarizeContextFrame( config=LLMContextSummaryConfig( target_context_tokens=2000, min_messages_after_summary=2, ) ) ) ``` See the [complete example](https://github.com/pipecat-ai/pipecat/blob/main/examples/context-summarization/context-summarization-manual-openai.py) for a full working implementation. ## Observability The summarizer emits an `on_summary_applied` event after each successful summarization, providing message count metrics: ```python from pipecat.processors.aggregators.llm_context_summarizer import SummaryAppliedEvent summarizer = assistant_aggregator._summarizer if summarizer: @summarizer.event_handler("on_summary_applied") async def on_summary_applied(summarizer, event: SummaryAppliedEvent): logger.info( f"Context summarized: {event.original_message_count} messages -> " f"{event.new_message_count} messages " f"({event.summarized_message_count} summarized, " f"{event.preserved_message_count} preserved)" ) ``` ## Next Steps Full reference for configuration parameters, events, and classes. Learn how Pipecat manages conversation context in pipelines. # Saving Conversation Transcripts Source: https://docs.pipecat.ai/pipecat/fundamentals/saving-transcripts.md Learn how to collect and save conversation transcripts between users and your bot ## Overview Recording transcripts of conversations between users and your bot is useful for debugging, analysis, and creating a record of interactions. Pipecat's turn events make it easy to collect both user and assistant messages as they occur. ## How It Works Transcripts are collected using turn events on the context aggregators: 1. Capturing what the user says via `on_user_turn_stopped` 2. Capturing what the assistant says via `on_assistant_turn_stopped` 3. Each event provides the complete transcript for that turn 4. Allowing you to handle these events with custom logic Turn events are emitted by the context aggregators (`LLMUserAggregator` and `LLMAssistantAggregator`), which are created as part of the `LLMContextAggregatorPair`. ## Basic Implementation ### Step 1: Create Context Aggregators First, create the context aggregator pair and get references to both aggregators: ```python from pipecat.processors.aggregators.llm_response_universal import ( LLMContextAggregatorPair, UserTurnStoppedMessage, AssistantTurnStoppedMessage, ) # Create context aggregator pair user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context) ``` ### Step 2: Add to Your Pipeline Include the aggregators in your pipeline: ```python pipeline = Pipeline( [ transport.input(), stt, # Speech-to-text user_aggregator, llm, tts, # Text-to-speech transport.output(), assistant_aggregator, ] ) ``` ### Step 3: Handle Turn Events Register event handlers to capture transcripts when turns complete: ```python @user_aggregator.event_handler("on_user_turn_stopped") async def on_user_turn_stopped(aggregator, strategy, message: UserTurnStoppedMessage): print(f"[{message.timestamp}] user: {message.content}") @assistant_aggregator.event_handler("on_assistant_turn_stopped") async def on_assistant_turn_stopped(aggregator, message: AssistantTurnStoppedMessage): if message.content: print(f"[{message.timestamp}] assistant: {message.content}") if message.interrupted: print(f"[{message.timestamp}] (assistant was interrupted)") ``` In addition to console logging, you can save transcripts to a database or file for later analysis. With a realtime (speech-to-speech) service and `LLMContextAggregatorPair(context, realtime_service_mode=True)`, capture the user transcript from [`on_user_turn_message_added`](/api-reference/server/utilities/turn-management/turn-events#on_user_turn_message_added) instead of `on_user_turn_stopped`, whose `message.content` is `None` in that mode. ## Next Steps Learn about all available turn events and their parameters. See more examples for collecting and processing transcriptions. Consider implementing transcript recording in your application for debugging during development and preserving important conversations in production. The transcript data can also be useful for analyzing conversation patterns and improving your bot's responses over time. # Recording Conversation Audio Source: https://docs.pipecat.ai/pipecat/fundamentals/recording-audio.md Learn how to record and save audio from conversations between users and your bot ## Overview Recording audio from conversations provides valuable data for analysis, debugging, and quality control. You have two options for how to record with Pipecat: ### Option 1: Record using your transport service provider Record without writing custom code by using your [transport](/api-reference/server/services/supported-services#transports) provider's recording capabilities. In addition to saving you development time, some providers offer unique recording capabilities. Refer to your service provider's documentation to learn more. For example, see [Recording](/api-reference/server/services/transport/daily#recording) in the `DailyTransport` reference. ### Option 2: Create your own recording pipeline Pipecat's `AudioBufferProcessor` makes it easy to capture high-quality audio recordings of both the user and bot during interactions. Opt for this approach if you want more control over your recording. The `AudioBufferProcessor.start_recording()` / `stop_recording()` methods below control this in-pipeline buffer. They are separate from a transport provider's own recording. This guide focuses on how to recording using the `AudioBufferProcessor`, including high-level guidance for how to set up post-processing jobs for longer recordings. ## How the AudioBufferProcessor Works The `AudioBufferProcessor` captures audio by: 1. Collecting audio frames from both the user (input) and bot (output) 2. Emitting events with recorded audio data 3. Providing options for composite or separate track recordings Add the processor to your pipeline after the `transport.output()` to capture both the user audio and the bot audio as it's spoken. ## Audio Recording Options The `AudioBufferProcessor` offers several configuration options: - **Composite recording**: Combined audio from both user and bot - `on_audio_data` event handler - **Track-level recording**: Separate audio files for user and bot - `on_track_audio_data` event handler - **Turn-based recording**: Individual audio clips for each speaking turn - `on_user_turn_audio_data` and `on_bot_turn_audio_data` event handlers - **Mono or stereo output**: Single channel mixing or two-channel separation - `num_channels=1` for mono; `num_channels=2` for stereo ## Basic Implementation ### Step 1: Create an Audio Buffer Processor Initialize the audio buffer processor with your desired configuration: ```python from pipecat.processors.audio.audio_buffer_processor import AudioBufferProcessor # Create audio buffer processor with default settings audiobuffer = AudioBufferProcessor( num_channels=1, # 1 for mono, 2 for stereo (user left, bot right) enable_turn_audio=False, # Enable per-turn audio recording auto_start_recording=False, # Start recording automatically when pipeline starts ) ``` ### Step 2: Add to Your Pipeline Place the processor in your pipeline after all audio-producing components: ```python pipeline = Pipeline( [ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), audiobuffer, # Add after all audio components context_aggregator.assistant(), ] ) ``` ### Step 3: Start Recording You have two options for starting recording: **Option A: Automatic recording** (recommended for most use cases) Set `auto_start_recording=True` when initializing the processor to begin recording as soon as the pipeline starts: ```python audiobuffer = AudioBufferProcessor( num_channels=1, auto_start_recording=True, # Recording starts automatically ) ``` **Option B: Manual control** Explicitly start recording when needed, typically when a session begins: ```python @transport.event_handler("on_client_connected") async def on_client_connected(transport, client): logger.info(f"Client connected") # Start recording explicitly await audiobuffer.start_recording() # Continue with session initialization... ``` Without `auto_start_recording=True`, you must call `start_recording()` explicitly to begin capturing audio. ### Step 4: Handle Audio Data Register an event handler to process audio data: ```python @audiobuffer.event_handler("on_audio_data") async def on_audio_data(buffer, audio, sample_rate, num_channels): # Save or process the composite audio timestamp = datetime.datetime.now().strftime("%Y%m%d_%H%M%S") filename = f"recordings/conversation_{timestamp}.wav" # Create the WAV file with wave.open(filename, "wb") as wf: wf.setnchannels(num_channels) wf.setsampwidth(2) # 16-bit audio wf.setframerate(sample_rate) wf.writeframes(audio) logger.info(f"Saved recording to {filename}") ``` If recording separate tracks, you can use the `on_track_audio_data` event handler to save user and bot audio separately. ## Recording Longer Conversations For conversations that last a few minutes, it may be sufficient to just buffer the audio in memory. However, for longer sessions, storing audio in memory poses two challenges: 1. **Memory Usage**: Long recordings can consume significant memory, leading to potential crashes or performance issues. 2. **Conversation Loss**: If the application crashes or the connection drops, you may lose all recorded audio. Instead, use the `buffer_size` parameter to record audio in manageable segments. This allows you to periodically save audio data to disk or upload it to cloud storage, reducing memory usage and ensuring data persistence. See an example of how to upload chunked audio to AWS cloud storage [here](). ### Chunked Recording Set a reasonable `buffer_size` to trigger periodic uploads: ```python # 30-second chunks (recommended for most use cases) SAMPLE_RATE = 24000 CHUNK_DURATION = 30 # seconds audiobuffer = AudioBufferProcessor( sample_rate=SAMPLE_RATE, buffer_size=SAMPLE_RATE * 2 * CHUNK_DURATION # 2 bytes per sample (16-bit) ) chunk_counter = 0 @audiobuffer.event_handler("on_track_audio_data") async def on_chunk_ready(buffer, user_audio, bot_audio, sample_rate, num_channels): global chunk_counter # Upload or save individual chunks await upload_audio_chunk(f"user_chunk_{chunk_counter:03d}.wav", user_audio, sample_rate, 1) await upload_audio_chunk(f"bot_chunk_{chunk_counter:03d}.wav", bot_audio, sample_rate, 1) chunk_counter += 1 ``` ### Multipart Upload Strategy For cloud storage, use multipart uploads to stream audio chunks. For example AWS cloud storage, use the [s3 multipart upload API](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html). If you are rolling your own multipart upload code, consider the following: **Conceptual Approach:** 1. **Initialize multipart upload** when recording starts 2. **Upload chunks as parts** when buffers fill (every ~30 seconds) 3. **Complete multipart upload** when recording ends 4. **Post-process** to create final WAV file(s), concatenate audio chunks **Benefits:** - Memory efficient for long sessions - Fault tolerant (no data loss if connection drops) - Enables real-time processing and analysis - Parallel upload of multiple tracks ### [Optional] Post-Processing Pipeline If not using a managed multipart upload framework like AWS s3 multipart upload, concatenate audio chunks together to create final audio files. This can be done with tools like FFmpeg: **Concatenating Audio Files:** ```bash # Method 1: Simple concatenation (same format) ffmpeg -i "concat:chunk_001.wav|chunk_002.wav|chunk_003.wav" -acodec copy final.wav # Method 2: Using file list (recommended for many chunks) # Create filelist.txt with format: # file 'chunk_001.wav' # file 'chunk_002.wav' # ... ffmpeg -f concat -safe 0 -i filelist.txt -c copy final_recording.wav ``` **Automation Considerations:** - Use sequence numbers in chunk filenames for proper ordering - Include metadata (sample rate, channels, duration) with each chunk - Implement retry logic for failed uploads - Consider using cloud functions/lambdas for automatic post-processing ## Next Steps Explore a complete working example that demonstrates how to record and save both composite and track-level audio with Pipecat. Read the complete API reference documentation for advanced configuration options and event handlers. Consider implementing audio recording in your application for quality assurance, training data collection, or creating conversation archives. The recorded audio can be stored locally, uploaded to cloud storage, or processed in real-time for further analysis. # Metrics Source: https://docs.pipecat.ai/pipecat/fundamentals/metrics.md Monitor Pipecat performance metrics: TTFB, processing time, and LLM and TTS usage across pipeline services. When developing real-time, multimodal AI applications, monitoring two key factors is crucial: performance (latency) and LLM/TTS usage. Performance impacts user experience, while usage can affect operational costs. Pipecat offers built-in metrics for both, which can be enabled with straightforward configuration options. ## Enabling performance metrics Set `enable_metrics=True` in `PipelineParams` when creating a worker: ```python Example config worker = PipelineWorker( pipeline, params=PipelineParams( ... enable_metrics=True, ... ), ) ``` Once enabled, Pipecat logs the following metrics: | Metric | Description | | ---------------- | -------------------------------------------------------------------------------- | | TTFB | Time To First Byte in seconds | | TTFA | Time To First Audio in seconds (TTS services only) | | Processing Time | Time taken by the service to respond in seconds | | Text Aggregation | Time from the first LLM token to the first complete sentence (TTS services only) | ```console Sample output AnthropicLLMService#0 TTFB: 0.8378312587738037 CartesiaTTSService#0 text aggregation time: 0.2134 CartesiaTTSService#0 TTFB: 0.17177796363830566 AnthropicLLMService#0 processing time: 2.4927797317504883 ``` ### Limiting TTFB responses If you only want the **first** TTFB measurement for each service, you can optionally pass `report_only_initial_ttfb=True` in `PipelineParams`: ```python Example config worker = PipelineWorker( pipeline, params=PipelineParams( ... enable_metrics=True, report_only_initial_ttfb=True, ... ), ) ``` > **Note:** `enable_metrics=True` is required for this setting to have an > effect. ### Disabling initial empty metrics By default, Pipecat sends an initial `MetricsFrame` with zero values for all services when the pipeline starts. To disable this behavior: ```python Example config worker = PipelineWorker( pipeline, params=PipelineParams( ... enable_metrics=True, send_initial_empty_metrics=False, ... ), ) ``` ## Enabling LLM/TTS Usage Metrics Set `enable_usage_metrics=True` in PipelineParams when creating a worker: ```python Example config worker = PipelineWorker( pipeline, params=PipelineParams( ... enable_usage_metrics=True, ... ), ) ``` Pipecat will log the following as applicable: | Metric | Description | | --------- | ------------------------------------------- | | LLM Usage | Number of prompt and completion tokens used | | TTS Usage | Number of characters processed | ```console Sample output CartesiaTTSService#0 usage characters: 65 AnthropicLLMService#0 prompt tokens: 104, completion tokens: 53 ``` > **Note:** Usage metrics are recorded per interaction and do not represent > running totals. ## Capturing Metrics Data When metrics are enabled, Pipecat emits a `MetricsFrame` for each interaction. The `MetricsFrame` contains a list of metrics data objects, which can include: - `TTFBMetricsData` — Time To First Byte - `TTFAMetricsData` — Time To First Audio (TTS) - `ProcessingMetricsData` — Processing time - `LLMUsageMetricsData` — LLM token usage - `TTSUsageMetricsData` — TTS character usage - `TextAggregationMetricsData` — Sentence aggregation latency (TTS) - `TurnMetricsData` — Turn completion predictions You can access the metrics data by either adding a custom [FrameProcessor](/pipecat/fundamentals/custom-frame-processor) to your pipeline or adding an [observer](/api-reference/server/utilities/observers/observer-pattern) to monitor `MetricsFrame`s. ### Example: Using MetricsLogObserver The simplest way to log metrics is with the built-in `MetricsLogObserver`. Pass it as an observer when creating your `PipelineWorker`: ```python from pipecat.observers.loggers.metrics_log_observer import MetricsLogObserver worker = PipelineWorker( pipeline, params=PipelineParams(enable_metrics=True, enable_usage_metrics=True), observers=[MetricsLogObserver()], ) ``` You can filter which metrics types are logged by passing `include_metrics`: ```python from pipecat.metrics.metrics import LLMUsageMetricsData, TTSUsageMetricsData from pipecat.observers.loggers.metrics_log_observer import MetricsLogObserver observers = [ MetricsLogObserver( include_metrics={LLMUsageMetricsData, TTSUsageMetricsData} ) ] ``` ### Example: Using a Custom FrameProcessor Create a custom FrameProcessor to handle metrics data. Here's an example Metrics Processor that can be added to your pipeline after the TTS processor. ```python from pipecat.frames.frames import MetricsFrame from pipecat.metrics.metrics import ( LLMUsageMetricsData, ProcessingMetricsData, TextAggregationMetricsData, TTFAMetricsData, TTFBMetricsData, TTSUsageMetricsData, ) from pipecat.processors.frame_processor import FrameDirection, FrameProcessor class MetricsLogger(FrameProcessor): async def process_frame(self, frame: Frame, direction: FrameDirection): await super().process_frame(frame, direction) if isinstance(frame, MetricsFrame): for d in frame.data: if isinstance(d, TTFBMetricsData): print(f"!!! MetricsFrame: {frame}, ttfb: {d.value}") elif isinstance(d, TTFAMetricsData): print(f"!!! MetricsFrame: {frame}, ttfa: {d.ttfa}, ttfb: {d.ttfb}, leading_silence: {d.leading_silence}") elif isinstance(d, ProcessingMetricsData): print(f"!!! MetricsFrame: {frame}, processing: {d.value}") elif isinstance(d, LLMUsageMetricsData): tokens = d.value print( f"!!! MetricsFrame: {frame}, prompt_tokens: {tokens.prompt_tokens}, completion_tokens: {tokens.completion_tokens}" ) elif isinstance(d, TextAggregationMetricsData): print(f"!!! MetricsFrame: {frame}, text aggregation: {d.value}") elif isinstance(d, TTSUsageMetricsData): print(f"!!! MetricsFrame: {frame}, characters: {d.value}") await self.push_frame(frame, direction) ``` ## Metrics Data Reference All metrics data classes inherit from `MetricsData`, which includes `processor` (the name of the processor that generated the metric) and an optional `model` field. ### TTFBMetricsData Time To First Byte — measures how long until the first byte of a response is received from a service. | Field | Type | Description | | ------- | ------- | --------------------------- | | `value` | `float` | TTFB measurement in seconds | ### TTFAMetricsData Time To First Audio — measures the time from a TTS request to the first audible audio sample. This includes the time to first byte plus any leading silence padding added by the service. `TTFAMetricsData` reports the latency breakdown directly, showing how much of the perceived latency is silence padding. | Field | Type | Description | | ----------------- | ------- | ------------------------------------------------------------------------------------------------------------------- | | `ttfa` | `float` | TTFA measurement in seconds (`ttfb` plus `leading_silence`) | | `ttfb` | `float` | Time-to-first-byte in seconds. Mirrors the standalone `TTFBMetricsData` for convenience, not a separate measurement | | `leading_silence` | `float` | Silence padding before the first audible sample, in seconds (`ttfa` minus `ttfb`) | ### ProcessingMetricsData Measures the total time taken by a service to process a request. | Field | Type | Description | | ------- | ------- | -------------------------------------- | | `value` | `float` | Processing time measurement in seconds | WebSocket-based TTS services (Cartesia, Deepgram, ElevenLabs WebSocket variants) and `DeepgramSageMakerTTSService` do not report processing metrics. TTFB and TTFA capture the meaningful latency for these services. ### TextAggregationMetricsData Measures the time from the first LLM token to the first complete sentence, representing the latency cost of sentence aggregation in the TTS pipeline. | Field | Type | Description | | ------- | ------- | --------------------------- | | `value` | `float` | Aggregation time in seconds | ### LLMUsageMetricsData Token usage for an LLM interaction. The `value` field is an `LLMTokenUsage` object with: | Field | Type | Description | | ------------------------------- | --------------- | --------------------------------------------------- | | `prompt_tokens` | `int` | Number of tokens in the input prompt | | `completion_tokens` | `int` | Number of tokens in the generated completion | | `total_tokens` | `int` | Total tokens used (prompt + completion) | | `cache_read_input_tokens` | `Optional[int]` | Tokens read from cache, if applicable | | `cache_creation_input_tokens` | `Optional[int]` | Tokens used to create cache entries | | `reasoning_tokens` | `Optional[int]` | Reasoning tokens (for reasoning models) | | `input_audio_tokens` | `Optional[int]` | Prompt tokens that were audio (realtime models) | | `output_audio_tokens` | `Optional[int]` | Completion tokens that were audio (realtime models) | | `cache_read_input_audio_tokens` | `Optional[int]` | Cache-read tokens that were audio (realtime models) | ### TTSUsageMetricsData Character usage for a TTS interaction. | Field | Type | Description | | ------- | ----- | ------------------------------------- | | `value` | `int` | Number of characters processed by TTS | ### TurnMetricsData Metrics from turn completion prediction, emitted by turn analyzers like Krisp Viva Turn and Smart Turn. | Field | Type | Description | | ------------------------ | ------- | ------------------------------------------------------------------------------------------ | | `is_complete` | `bool` | Whether the turn is predicted to be complete | | `probability` | `float` | Confidence probability of the prediction | | `e2e_processing_time_ms` | `float` | End-to-end processing time in ms, from VAD speech-to-silence transition to turn completion | ## Related Observers In addition to `MetricsLogObserver`, Pipecat provides observers that track higher-level conversational metrics. ### StartupTimingObserver Measures the time taken by each processor to start up. ```python from pipecat.observers.startup_timing_observer import StartupTimingObserver startup_observer = StartupTimingObserver() @observer.event_handler("on_startup_timing_report") async def on_startup_timing_report(observer, report): print(f"Total startup: {report.total_duration_secs:.3f}s") for timing in report.processor_timings: print(f" {timing.processor_name}: {timing.duration_secs:.3f}s") ``` Additionally, it tracks the time taken to connect to the transport and the time taken to connect to the client. ```python @observer.event_handler("on_transport_timing_report") async def on_transport_timing_report(observer, report): if report.bot_connected_secs is not None: print(f"Bot connected: {report.bot_connected_secs:.3f}s") print(f"Client connected: {report.client_connected_secs:.3f}s") ``` ### UserBotLatencyObserver Measures the time between when a user stops speaking and when the bot starts speaking. ```python from pipecat.observers.user_bot_latency_observer import UserBotLatencyObserver latency_observer = UserBotLatencyObserver() @latency_observer.event_handler("on_latency_measured") async def on_latency_measured(observer, latency_seconds): print(f"User-to-bot latency: {latency_seconds:.3f}s") worker = PipelineWorker(pipeline, observers=[latency_observer]) ``` ### TurnTrackingObserver Tracks conversation turns, emitting events when turns start and end. Handles interruptions and configurable timeouts. ```python from pipecat.observers.turn_tracking_observer import TurnTrackingObserver turn_observer = TurnTrackingObserver(turn_end_timeout_secs=2.5) @turn_observer.event_handler("on_turn_started") async def on_turn_started(observer, turn_count): print(f"Turn {turn_count} started") @turn_observer.event_handler("on_turn_ended") async def on_turn_ended(observer, turn_count, duration, was_interrupted): status = "interrupted" if was_interrupted else "completed" print(f"Turn {turn_count} {status} after {duration:.2f}s") worker = PipelineWorker(pipeline, observers=[turn_observer]) ``` # Voicemail Detection Source: https://docs.pipecat.ai/pipecat/fundamentals/voicemail.md Automatically classify outbound calls as conversation or voicemail and respond appropriately ## Overview The VoicemailDetector classifies incoming communication as either live conversation or voicemail systems. This module is built primarily for voice AI bots that perform outbound calling, enabling them to respond appropriately based on whether a human answered or the call went to voicemail. The detector is optimized for fast conversation response times, where TTS output is generated immediately but held in a gate until the classification decision is made. This ensures minimal latency for live conversations while preventing inappropriate responses to voicemail systems. ## How It Works The VoicemailDetector uses a parallel pipeline architecture to perform real-time classification without interrupting conversation flow. It analyzes the initial response from the called party and determines whether it's a human greeting or an automated voicemail system. Key features: - **Real-time classification** - Determines conversation vs voicemail as soon as audio is received - **TTS gating** - Holds generated audio until classification is complete - **Event-driven** - Triggers custom handlers when voicemail is detected - **Configurable timing** - Adjustable delay for voicemail response timing ## Basic Setup ### 1. Initialize the Detector ```python from pipecat.extensions.voicemail.voicemail_detector import VoicemailDetector from pipecat.services.openai.llm import OpenAILLMService # Create an LLM for classification classifier_llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY")) # Initialize the detector voicemail_detector = VoicemailDetector( llm=classifier_llm, voicemail_response_delay=2.0 # Default: 2 seconds ) ``` The VoicemailDetector works with two LLMs: your main conversation LLM (which must be text-based) and the classifier LLM (which can be either text-based or realtime with a `text` output modality). Realtime LLMs are not compatible for the main conversation LLM due to output control requirements. ### 2. Configure the Pipeline The VoicemailDetector requires two components in your pipeline: - `detector()`: between STT and user context aggregator - `gate()`: immediately after TTS service ```python pipeline = Pipeline([ transport.input(), stt, voicemail_detector.detector(), # Between STT and user context aggregator context_aggregator.user(), llm, tts, voicemail_detector.gate(), # Immediately after TTS service transport.output(), context_aggregator.assistant(), ]) ``` ### 3. Handle Voicemail Events When a voicemail is detected, the `on_voicemail_detected` event is triggered. In your event handler, you have access to `processor`, which is a FrameProcessor instance, allowing you to push frames to the pipeline. For example, you may want to have the bot output a pre-canned message and then end the call. ```python @voicemail_detector.event_handler("on_voicemail_detected") async def handle_voicemail(processor): logger.info("Voicemail detected! Leaving a message...") # Leave a voicemail message await processor.push_frame( TTSSpeakFrame("Hello, this is Jamie calling about your appointment. Please call me back at 555-0123.") ) # Optionally end the call after leaving the message await processor.push_frame(EndWorkerFrame(), FrameDirection.UPSTREAM) ``` ## Detecting a Conversation When a conversation is detected, no additional processing is required by the VoicemailDetector. Once the VoicemailDetector classifies the call as a conversation, the TTSGate will be flushed, allowing the conversation to continue. ## Configuration Options ### Custom System Prompt For specialized use cases, you can provide a custom classification prompt. The prompt must instruct the LLM to respond with exactly "CONVERSATION" or "VOICEMAIL": ```python custom_prompt = """ Your custom classification logic here. Consider factors like business hours, call patterns, etc. """ + VoicemailDetector.CLASSIFIER_RESPONSE_INSTRUCTION voicemail_detector = VoicemailDetector( llm=classifier_llm, custom_system_prompt=custom_prompt ) ``` The `CLASSIFIER_RESPONSE_INSTRUCTION` constant contains the required response format: `'Respond with ONLY "CONVERSATION" if a person answered, or "VOICEMAIL" if it\'s voicemail/recording.'` For most use cases, the default classifier prompt will work effectively. If you need to customize the behavior, reference the built-in prompt as a starting point and modify from there. ### Response Timing The `voicemail_response_delay` parameter controls how long to wait after the user stops speaking before triggering the voicemail event: ```python voicemail_detector = VoicemailDetector( llm=classifier_llm, voicemail_response_delay=3.0 # Wait 3 seconds instead of default 2 ) ``` This delay ensures: - Voicemail greetings have finished playing - The response occurs during the recording period - Proper timing for different voicemail system behaviors # IVR Navigation Source: https://docs.pipecat.ai/pipecat/fundamentals/ivr.md Automatically navigate phone system menus using AI-powered decision making ## Overview The `IVRNavigator` enables your bot to automatically navigate Interactive Voice Response (IVR) phone systems to reach specific goals. Instead of manually programming navigation paths, you provide an end goal and the bot handles the complex decision-making required to traverse phone menus using DTMF tones and conversational responses. ## How IVR Navigation Works The IVRNavigator combines several intelligent capabilities: 1. **Goal-oriented navigation**: You specify what you want to accomplish (e.g., "reach billing support") 2. **Automatic classification**: Detects whether incoming audio is an IVR system or human conversation 3. **Smart decision making**: Analyzes menu options and selects the best path toward your goal 4. **Multi-modal responses**: Uses both DTMF tones for menu selection, natural language for prompts, and waits when no input is appropriate 5. **Status tracking**: Monitors progress and reports completion, waiting, or stuck states ## Navigation Outcomes The `IVRNavigator` can reach several outcomes during navigation: ### Completed ✅ The navigator successfully reaches its goal and is ready for the next step. Common actions: - [Terminate the pipeline](/pipecat/learn/pipeline-termination) if the goal is complete - Transfer the call to a human agent - Allow your bot to start a conversation with the reached department ```python @ivr_navigator.event_handler("on_ivr_status_changed") async def handle_ivr_status(processor, status): if status == IVRStatus.COMPLETED: logger.info("Successfully navigated to target department") # Start conversation # Replace the context with new messages for the upcoming conversation messages = [{"role": "developer", "content": "You are a helpful customer service assistant."}] await worker.queue_frame(LLMMessagesUpdateFrame(messages)) # Adjust VAD stop_secs for conversation vad_params = VADParams(stop_secs=0.8) await worker.queue_frame(VADParamsUpdateFrame(vad_params)) ``` ### Stuck ⚠️ The navigator cannot find a path forward in the IVR system. This might happen when: - Required information (account numbers, PINs) isn't available - The menu options don't align with the stated goal - The system encounters errors or invalid selections ```python @ivr_navigator.event_handler("on_ivr_status_changed") async def handle_ivr_status(processor, status): if status == IVRStatus.STUCK: logger.warning("IVR navigation stuck - terminating call") # Log the issue and clean up await log_navigation_failure() await worker.queue_frame(EndFrame()) ``` ## Flexible Entry Points One of the `IVRNavigator`'s key features is flexible entry point handling. When you dial a phone number, you might encounter either: - An IVR system with menu options - A direct connection to a human The navigator automatically detects which scenario occurs and emits appropriate events: ### Human Conversation Detected When a human answers instead of an IVR system, an `on_conversation_detected` event is emitted. You can handle that event to transition to a conversation. ```python @ivr_navigator.event_handler("on_conversation_detected") async def on_conversation_detected(processor, conversation_history): # Set up conversation prompt and preserve conversation history messages = [ { "role": "developer", "content": "You are an assistant calling to check on the prescription for John Smith, date of birth 01/01/1990.", } ] # Add preserved conversation history if available if conversation_history: messages.extend(conversation_history) await worker.queue_frame(LLMMessagesUpdateFrame(messages=messages, run_llm=True)) ``` Note that the `on_conversation_detected` event also emits a `conversation_history` parameter that contains the previous conversation history. This allows you to build a prompt that includes your conversation system prompt plus any conversation history up to that point in time. ### IVR System Detected When an IVR system is detected, the IVR Navigator automatically transitions into navigation mode: - **System prompt**: Updates to use your specified navigation goal - **VAD timing**: Adjusts to `stop_secs=2.0` (or your custom `ivr_vad_params`) to allow time for complete menu announcements - **Navigation logic**: Begins analyzing menu options and making decisions toward your goal No additional code is required. The navigator handles this transition automatically. **Optional Event Handling** If you need to log the detection or perform custom actions, you can handle the `on_ivr_status_changed` event: ```python @ivr_navigator.event_handler("on_ivr_status_changed") async def on_ivr_status_changed(processor, status): if status == IVRStatus.DETECTED: logger.info("IVR system detected - beginning navigation") # Optional: Add analytics tracking, custom setup, etc. ``` ## Basic Implementation ### Step 1: Create the IVR Navigator ```python from pipecat.extensions.ivr.ivr_navigator import IVRNavigator from pipecat.audio.vad.vad_analyzer import VADParams # Define your navigation goal ivr_goal = "Navigate to the billing department to discuss my account balance" # Create navigator with extended response time for IVR systems ivr_vad_params = VADParams(stop_secs=2.0) # Longer wait for IVR menus ivr_navigator = IVRNavigator( llm=your_llm_service, ivr_prompt=ivr_goal, ivr_vad_params=ivr_vad_params ) ``` ### Step 2: Set Up Event Handlers ```python from pipecat.frames.frames import LLMMessagesUpdateFrame from pipecat.extensions.ivr.ivr_navigator import IVRStatus @ivr_navigator.event_handler("on_conversation_detected") async def on_conversation_detected(processor, conversation_history): """Handle when a human conversation is detected instead of IVR""" logger.info("Human conversation detected") # Set up conversation context messages = [ {"role": "developer", "content": "You are a customer service representative."} ] if conversation_history: messages.extend(conversation_history) await worker.queue_frame(LLMMessagesUpdateFrame(messages=messages, run_llm=True)) @ivr_navigator.event_handler("on_ivr_status_changed") async def on_ivr_status_changed(processor, status): """Handle IVR navigation status changes""" if status == IVRStatus.COMPLETED: logger.info("IVR navigation completed successfully") # Your success handling logic here elif status == IVRStatus.STUCK: logger.warning("IVR navigation got stuck") # Your error handling logic here await handle_navigation_failure() ``` ### Step 3: Add to Pipeline Add the IVR Navigator to your pipeline in the place where you would normally add the LLM. The IVR Navigator contains your LLM and will perform the same functions as an LLM would, but in addition it will navigate the IVR system. ```python from pipecat.pipeline.pipeline import Pipeline pipeline = Pipeline([ transport.input(), stt_service, ivr_navigator, # Add the navigator to your pipeline tts_service, transport.output() ]) ``` ## VAD Parameter Optimization The IVRNavigator automatically optimizes Voice Activity Detection (VAD) parameters for different scenarios: ### IVR Navigation Mode - **Default**: `stop_secs=2.0` - **Purpose**: Allows time to hear complete menu options before responding - **Result**: Higher navigation success rates ### Conversation Mode - **Recommended**: `stop_secs=0.8` - **Purpose**: Enables natural conversation flow with quick responses - **Implementation**: Push `VADParamsUpdateFrame` when transitioning to conversation ## Next Steps Learn how to properly terminate pipelines when IVR navigation completes Integrate structured conversation flows after successful IVR navigation The IVRNavigator provides a powerful foundation for automating phone system interactions, allowing your bots to handle the complex task of menu navigation while you focus on the core conversation logic. # Custom FrameProcessor Source: https://docs.pipecat.ai/pipecat/fundamentals/custom-frame-processor.md Write a custom Pipecat FrameProcessor: handle frames, push new ones downstream, and slot into a pipeline. Pipecat's architecture is made up of a Pipeline, FrameProcessors, and Frames. See the [Core Concepts](/pipecat/learn/pipeline) for a full review. From that architecture, recall that FrameProcessors are the workers in the pipeline that receive frames and complete actions based on the frames received. Pipecat comes with many FrameProcessors built in. These consist of services, like `OpenAILLMService` or `CartesiaTTSService`, utilities, like `LLMTextProcessor`, and other things. Largely, you can build most of your application with these built-in FrameProcessors, but commonly, your application code may require custom frame processing logic. For example, you may want to perform an action as a result of a frame that's pushed in the pipeline. ## Example: MetricsFrame logger This custom FrameProcessor format and logs MetricsFrames: ```python class MetricsFrameLogger(FrameProcessor): """MetricsFrameLogger formats and logs all MetericsFrames""" def __init__(self): super().__init__() async def process_frame(self, frame: Frame, direction: FrameDirection): await super().process_frame(frame, direction) if isinstance(frame, MetricsFrame): logger.info(f"{frame.name}\n {format_metrics(frame.data)}") await self.push_frame(frame, direction) # ALWAYS push all frames else: # SUPER IMPORTANT: always push every frame! await self.push_frame(frame, direction) ``` This frame processor looks for `MetricsFrames`. When it sees one, it formats the data and logs it. It uses this `format_metrics` function: ```python def format_metrics(metrics, indent=0): lines = [] tab = "\t" * indent for metric in metrics: lines.append(tab + type(metric).__name__) for field, value in vars(metric).items(): if hasattr(value, "__dict__") and not isinstance( value, (str, int, float, bool, type(None)) ): lines.append(f"{tab}\t{field}={type(value).__name__}") for k, v in vars(value).items(): lines.append(f"{tab}\t\t{k}={repr(v)}") else: lines.append(f"{tab}\t{field}={repr(value)}") return "\n".join(lines) ``` See this [working example](https://github.com/pipecat-ai/pipecat/blob/main/examples/features/features-custom-frame-processor.py) using the `MetricsFrameLogger` FrameProcessor ## Add to a Pipeline ```python # Create and initialize the custom FrameProcessor metrics_frame_processor = MetricsFrameLogger() pipeline = Pipeline( [ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), metrics_frame_processor, # Our custom FrameProcessor that pretty prints metrics frames ] ) ``` With this positioning, the `MetricsFrameLogger` FrameProcessor will receive every MetericsFrame in the pipeline. ## Key Requirements FrameProcessors must inherit from the base `FrameProcessor` class. This ensures that your custom FrameProcessor will correctly handle frames like `StartFrame`, `EndFrame`, `InterruptionFrame` without having to write custom logic for those frames. This inheritance also provides it with the ability to `process_frame()` and `push_frame()`: - **`process_frame()`** is what allows the FrameProcessor to receive frames and add custom conditional logic based on the frames that are received. - **`push_frame()`** allows the FrameProcessor to push frames to the pipeline. Normally, frames are pushed DOWNSTREAM, but based on which processors need the output, you can also push UPSTREAM or in both directions. ### Essential Implementation Details To ensure proper base class inheritance, it's critical to include: 1. **`super().__init__()`** in your `__init__` method 2. **`await super().process_frame(frame, direction)`** in your `process_frame()` method ```python class MyCustomProcessor(FrameProcessor): def __init__(self, **kwargs): super().__init__(**kwargs) # ✅ Required # Your initialization code here async def process_frame(self, frame: Frame, direction: FrameDirection): await super().process_frame(frame, direction) # ✅ Required # Your custom frame processing logic here if isinstance(frame, SomeSpecificFrame): # Handle the frame pass await self.push_frame(frame, direction) # ✅ Required - pass frame through ``` ## Critical Responsibility: Frame Forwarding FrameProcessors receive **all** frames that are pushed through the pipeline. This gives them a lot of power, but also a great responsibility. Critically, they must push all frames through the pipeline; if they don't, they block frames from moving through the Pipeline, which will cause issues in how your application functions. As well as formatting and logging MetricsFrames, `MetricsFrameLogger` also has an `await self.push_frame(frame, direction)` which pushes the frame through to the next processor in the pipeline. ## Frame Direction When pushing frames, you can specify the direction: ```python # Push downstream (default) await self.push_frame(frame, FrameDirection.DOWNSTREAM) # Push upstream await self.push_frame(frame, FrameDirection.UPSTREAM) ``` Most custom FrameProcessors will push frames downstream, but upstream can be useful for sending control frames or error notifications back up the pipeline. ## Best Practices 1. **Always call the parent methods**: Use `super().__init__()` and `await super().process_frame()` 2. **Forward all frames**: Make sure every frame is pushed through with `await self.push_frame(frame, direction)` 3. **Handle frames conditionally**: Use `isinstance()` checks to handle specific frame types 4. **Use proper error handling**: Wrap risky operations in try/catch blocks 5. **Position carefully in pipeline**: Consider where in the pipeline your processor needs to be to receive the right frames With these patterns, you can create powerful custom FrameProcessors that extend Pipecat's capabilities for your specific use case. # The Worker Bus Source: https://docs.pipecat.ai/pipecat/fundamentals/agent-bus.md How Pipecat workers communicate through the shared message bus: publish, subscribe, and message flow between agents. ## What is the bus? The worker bus is the communication backbone of every multi-worker system. All workers connect to the same bus and exchange typed messages for frame routing, lifecycle events, and job coordination. Think of it as an internal event bus -- workers publish messages and other workers receive them through their subscriptions. The bus handles priority queuing so that urgent messages (like cancellations) are delivered before queued data. ## Bus implementations Pipecat provides three bus implementations: ### AsyncQueueBus (local) The default bus, created automatically by `WorkerRunner` when you don't provide one. It uses asyncio queues for in-process communication with no serialization overhead. ```python runner = WorkerRunner() # Creates AsyncQueueBus automatically ``` This is all you need for single-process applications where all workers run together. ### RedisBus (distributed) For distributed setups where workers run in separate processes or on different machines, use `RedisBus`. It uses Redis pub/sub to relay messages across process boundaries. ```python from redis.asyncio import Redis from pipecat.bus.network.redis import RedisBus redis = Redis.from_url("redis://localhost:6379") bus = RedisBus(redis=redis, channel="pipecat:my-app") runner = WorkerRunner(bus=bus) ``` All processes that share the same Redis channel can exchange messages. The programming model stays the same -- your agent code doesn't change between local and distributed setups. `RedisBus` requires the `redis` extra: `uv add "pipecat-ai[redis]"` ### PgmqBus (distributed) An alternative distributed bus backed by PGMQ (PostgreSQL Message Queue). Each instance creates its own queue and broadcasts to peer queues. ```python from pgmq.async_queue import PGMQueue from pipecat.bus.network.pgmq import PgmqBus pgmq = PGMQueue( host="localhost", port="5432", database="postgres", username="postgres", password="...", pool_size=4, ) await pgmq.init() bus = PgmqBus(pgmq=pgmq, channel="pipecat:my-app") runner = WorkerRunner(bus=bus) ``` `PgmqBus` requires the `pgmq` extra: `uv add "pipecat-ai[pgmq]"` ## Message types Messages on the bus fall into four categories: ### Data messages Normal-priority messages that carry data between agents. The most important one is `BusFrameMessage`, which wraps a Pipecat frame (audio, text, etc.) for transport across the bus. ### System messages High-priority messages for lifecycle events: activation, deactivation, shutdown, and worker readiness. These are delivered before data messages in the queue. ### Job messages Messages for coordinating work between agents: job requests, responses, progress updates, streaming, and cancellation. ### Local messages Some messages are local-only and never cross process boundaries. For example, child agent errors stay local to the parent. This keeps internal state from leaking across distributed runners. ## Message routing Messages have `source` and `target` fields: - **Targeted messages** (with a specific `target`) are delivered only to the named agent - **Broadcast messages** (with no `target`) are delivered to all subscribers The bus handles this routing automatically. When you call `activate_worker("greeter")`, it sends a `BusActivateWorkerMessage` targeted at `"greeter"` -- only that agent receives it. ## The agent registry The runner maintains a `WorkerRegistry` that tracks which agents are available. To get notified when an agent is ready, use the `@worker_ready` decorator (or call `watch_workers()` explicitly): ```python from pipecat.pipeline.base_worker import BaseWorker from pipecat.pipeline.worker_ready_decorator import worker_ready from pipecat.registry.types import WorkerReadyData class MainAgent(BaseWorker): @worker_ready(name="greeter") async def on_greeter_ready(self, data: WorkerReadyData) -> None: await self.activate_worker("greeter") ``` The framework automatically calls `watch_workers()` for each `@worker_ready` handler when the agent starts. If the watched agent is already registered, the handler fires immediately, so you don't need to worry about race conditions. # Agent Registry and Discovery Source: https://docs.pipecat.ai/pipecat/fundamentals/agent-registry-and-discovery.md How agents discover each other and get notified when other agents are ready. ## What is the agent registry? The `WorkerRegistry` tracks all known agents across local and remote runners. It is owned by the runner and shared with all its agents. When an agent becomes ready, it registers itself, and other agents that are watching for it get notified. You don't interact with the registry directly in most cases. Instead, you use the `@worker_ready` decorator or `watch_workers()` to express interest in a specific agent, and the registry handles the rest. ## Watching for agents ### The @worker_ready decorator The most common way to watch for an agent. Decorate a method with the agent name, and it fires when that agent registers: ```python from pipecat.pipeline.base_worker import BaseWorker from pipecat.pipeline.worker_ready_decorator import worker_ready from pipecat.registry.types import WorkerReadyData class MainAgent(BaseWorker): @worker_ready(name="greeter") async def on_greeter_ready(self, data: WorkerReadyData) -> None: await self.activate_worker("greeter") ``` The framework automatically calls `watch_workers()` for each `@worker_ready` handler when the agent starts. If the watched agent is already registered, the handler fires immediately. ### watch_workers() For dynamic cases where you don't know the agent name at class definition time, call `watch_workers()` directly. The best place to do this is in `start()`, after calling `super().start()`: ```python class MainAgent(BaseWorker): async def start(self) -> None: await super().start() for name in self._dynamic_worker_names: await self.watch_workers(name) async def on_worker_ready(self, data: WorkerReadyData) -> None: await super().on_worker_ready(data) await self.activate_worker(data.worker_name) ``` This is useful when agent names come from configuration or are created at runtime. Watched agents that are not handled by a `@worker_ready` decorator dispatch to the `on_worker_ready` hook. ## How discovery works ### Local agents When all agents run in the same process, discovery is straightforward. An agent registers in the shared registry when its pipeline is ready, and watchers are notified immediately. ### Distributed agents In distributed setups (agents across different processes connected to the same bus), runners exchange registry snapshots automatically. When a remote agent becomes ready, its runner broadcasts a registry message over the bus. Other runners update their local registry, and any matching watchers fire. Only root agents (added via `runner.add_workers()`) are discoverable across runners. Child agents (added via `parent.add_workers()`) are not broadcast and remain invisible to other runners. ## Agent readiness data When a watcher fires, it receives a `WorkerReadyData` object: ```python @worker_ready(name="greeter") async def on_greeter_ready(self, data: WorkerReadyData) -> None: print(data.worker_name) # "greeter" print(data.runner) # Name of the runner managing the agent ``` ## Uniqueness Agent names must be unique within a registry. If the same name is registered from two different runners, the registry logs a warning. In distributed setups, choose unique names across all runners to avoid conflicts. # Understanding the Bus Bridge Source: https://docs.pipecat.ai/pipecat/fundamentals/understanding-the-bus-bridge.md How BusBridgeProcessor routes frames between the transport pipeline and the agent bus. ## What is the bus bridge? The `BusBridgeProcessor` is a pipeline processor that connects a transport pipeline to the agent bus. It sits in the main agent's pipeline where an LLM would normally go, routing frames to whichever agent is currently active. Without the bridge, your main agent's pipeline would look like: ``` transport.input → STT → context_agg → LLM → TTS → transport.output ``` With the bridge, the LLM is replaced: ``` transport.input → STT → context_agg → BusBridge → TTS → transport.output ``` The bridge sends outgoing frames (like transcribed text) to the bus, and receives incoming frames (like LLM-generated text) from the active agent. The examples above show a voice pipeline, but `BusBridgeProcessor` can be used in any type of pipeline. It works with any frames, not just audio. ## How it works The bridge operates in two directions: **Outgoing** (pipeline to bus): Frames flowing downstream through the pipeline are captured by the bridge and published as `BusFrameMessage` on the bus. The active agent receives these frames. **Incoming** (bus to pipeline): When an active agent sends frames back through the bus, the bridge pushes them into the pipeline. These frames continue downstream to TTS and the transport output. Certain frames are never sent across the bus: - Lifecycle frames (`StartFrame`, `EndFrame`, `CancelFrame`, `StopFrame`) - Transport-urgent frames (these pass through the bridge locally) ## Basic usage The main agent is a `PipelineWorker` wrapping a pipeline that contains a `BusBridgeProcessor`. The bridge gets its bus from `runner.bus`: ```python from pipecat.bus import BusBridgeProcessor from pipecat.pipeline.pipeline import Pipeline from pipecat.workers.runner import WorkerRunner from pipecat.pipeline.worker import PipelineParams, PipelineWorker MAIN_NAME = "acme" runner = WorkerRunner() bridge = BusBridgeProcessor( bus=runner.bus, worker_name=MAIN_NAME, name=f"{MAIN_NAME}::BusBridge", ) pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), bridge, tts, transport.output(), context_aggregator.assistant(), ]) main = PipelineWorker(pipeline, name=MAIN_NAME, params=PipelineParams()) await runner.add_workers(main) ``` The worker gets its bus from the runner when added with `runner.add_workers()`. You build the `BusBridgeProcessor` ahead of time with `runner.bus` and place it in the pipeline where an LLM would normally go. ## Named bridges When you have multiple bridges in a system, you can name them to control which agents receive frames from which bridge. This is useful with [parallel pipelines](/server/pipeline/parallel-pipeline) where each branch has its own bridge (for example, separate audio and video branches): ```python voice_bridge = BusBridgeProcessor( bus=runner.bus, worker_name=MAIN_NAME, name=f"{MAIN_NAME}::VoiceBridge", bridge="voice", ) ``` Agents control which bridges they listen to through the `bridged` argument when they are created: ```python # Receives frames from all bridges all_bridges_agent = MyLLMWorker("all", llm=llm, bridged=()) # Receives frames only from the "voice" bridge voice_only_agent = MyLLMWorker("voice-only", llm=llm, bridged=("voice",)) ``` ## The bridged agent side When an agent sets `bridged=()` (or a tuple of bridge names), the framework automatically wraps its pipeline with edge processors that handle bus frame conversion. An LLM agent subclasses `LLMWorker` and passes its own LLM service: ```python from pipecat.services.openai.llm import OpenAILLMService from pipecat.workers.llm import LLMWorker, tool class MyLLMWorker(LLMWorker): @tool async def my_function(self, params, arg: str): ... llm = OpenAILLMService(api_key="...") agent = MyLLMWorker("greeter", llm=llm, bridged=()) ``` A worker never takes a `bus=` argument. It receives its bus from the runner when you register it with `runner.add_workers()`. The resulting pipeline looks like: ``` BusEdgeProcessor (upstream) → [your pipeline] → BusEdgeProcessor (downstream) ``` The upstream edge processor receives `BusFrameMessage` from the bus and converts them back to regular Pipecat frames. The downstream edge processor captures output frames and sends them back through the bus. You don't need to manage this -- it happens automatically when `bridged` is set. # Pipecat Evals Source: https://docs.pipecat.ai/pipecat/evals/overview.md Behavioral testing for your agents: scripted conversations, semantic assertions, and an LLM judge. Pipecat Evals is the framework's built-in system for testing agent behavior. You describe a conversation and the behavior you expect, and Pipecat runs it against your real agent (the same pipeline, the same services, the same code) and tells you whether the expectation still holds. ```yaml capital_question.yaml name: capital_question turns: - user: "What is the capital of Germany?" expect: - event: response eval: "the response says the capital of Germany is Berlin" ``` ```bash pipecat eval run capital_question.yaml ``` ## Why evals matter Voice agents are probabilistic systems. The same agent can answer differently run to run, and a prompt tweak, a model upgrade, or a service swap can quietly break behavior that used to work: a function that no longer gets called, context that stops carrying across turns, an interruption that derails the conversation. Manual testing catches some of this, but it's slow, unrepeatable, and impractical to run on every change. Evals make agent behavior testable the way unit tests make code testable: - **Regression safety**: run your scenarios after every prompt, model, or pipeline change and catch breakage before users do. - **Fast iteration**: text-mode evals skip STT and TTS entirely, so a full conversation test runs in seconds with no audio service cost. - **Semantic assertions**: an LLM judge checks meaning ("the response says the capital is Berlin"), not exact strings, so tests don't break when wording changes. - **A feedback signal for AI coding assistants**: evals give a coding assistant a command it can run and a pass/fail result it can read, closing the loop between writing agent code and verifying it. See [The Eval Loop](/pipecat/evals/the-eval-loop). Pipecat itself relies on this framework: before every release, an eval suite drives 100+ example agents end to end. ## How it works Pipecat Evals has two halves: 1. **The eval transport.** Your agent runs unchanged with the eval transport. If your agent uses `create_transport()` and the development runner, this is already built in: start it with `-t eval` and it hosts a local WebSocket server speaking RTVI, instead of connecting to Daily, WebRTC, or telephony. 2. **The eval harness.** The harness connects to that transport as an RTVI client, plays the scenario's user turns (as text, or as synthesized speech in audio mode), collects the events your agent emits, and asserts on them in order: transcriptions, LLM responses, spoken output, function calls, and timing. When a scenario asserts on meaning rather than exact text, a **judge LLM** evaluates the agent's response against a natural-language criterion. The judge runs locally with [Ollama](https://ollama.com) by default, or against OpenAI or any OpenAI-compatible endpoint. ### Text and audio modes Every scenario runs in one of two modes: | Mode | User input | Agent output | Best for | | ------------------ | -------------------------------------------------------- | --------------------------------------------------------------- | --------------------------------------------------------------- | | **Text** (default) | Sent as text, bypassing the STT | LLM text; TTS is skipped automatically | Fast, cheap iteration on prompts, logic, and function calling | | **Audio** | Synthesized by a TTS the harness runs (local by default) | Real synthesized speech, transcribed by an STT the harness runs | True end-to-end coverage of the full STT, LLM, and TTS pipeline | Text mode exercises your agent's actual pipeline and context handling while skipping the audio services, so it costs nothing in TTS or STT usage and runs fast. Audio mode synthesizes the user's voice, streams it through your agent's real STT, and transcribes the agent's actual spoken audio for judging, catching issues that only surface with real speech (turn detection, homophones, barge-in). ## What you can test - **Response content**: substring checks (`text_contains`) or semantic judging (`eval`) of the agent's replies. - **Multi-turn context**: verify the agent remembers earlier turns. - **Function calling**: assert that specific tools were called, with specific arguments. - **Interruptions**: barge in mid-response and verify the agent recovers (`send_after`). - **Latency**: per-event budgets with `within_ms`. - **Vision**: serve an image when the agent requests one and judge its description. ## YAML or Python Scenarios are YAML files, so they're easy to write, review, and share. Everything is also available as a library: load and run scenarios programmatically, build them in code, inject a custom judge, or orchestrate whole suites from your own tooling. See [Using the Library](/pipecat/evals/library). ## Requirements - **Pipecat CLI**: the `pipecat eval` commands ship with the CLI extra: `uv tool install "pipecat-ai[cli]"`. If you've added `pipecat-ai[cli]` to your project instead, run them with `uv run pipecat eval` (just like `uv run bot.py`). The same commands are also available as `python -m pipecat.evals`. - **A judge LLM** (for `eval:` assertions): Ollama by default (`ollama pull gemma2:9b`), or point the scenario's `judge:` block at OpenAI or any OpenAI-compatible endpoint. - **Audio services** (audio mode only): the harness needs a TTS to synthesize the user's voice and an STT to transcribe the agent's speech. Both can be local models or HTTP-based services; the defaults are local (Kokoro and Moonshine or Whisper, installed with `uv add "pipecat-ai[kokoro,moonshine]"` or `uv add "pipecat-ai[kokoro,whisper]"`), which download once on first use and run with no keys and no per-run cost. WebSocket-streaming services aren't supported here, which keeps the harness simple. - **Your agent's own credentials**: the agent under test is your real agent, so it needs the same service API keys it normally would. ## Production evaluation Pipecat Evals is built for development: fast, local, repeatable, and run on every change. Once your agent is deployed, third-party evaluation platforms complement it with testing and monitoring at production scale: - **Simulations**: scripted or AI-driven test calls over API, WebSocket, or telephony, exercising multi-turn flows, edge cases, and real phone-network conditions before they reach users. - **Observability**: continuous evaluation of live traffic, with automated quality scoring of calls and transcripts, and metrics tracked over time to catch quality drift. Simulation, observability, and evaluation platform with native Pipecat Cloud integration. Supports no-code API, WebSocket, and telephony testing. Automated testing and monitoring platform with native Pipecat Integration for WebRTC/Text based testing and support for Mock Tools, Custom Dynamic Variables and more! AI-native simulation and evaluation platform for voice agents, trusted by QA, Engineering, Operations, AI, and Executive teams. Simulation, observability, tracing, and metrics with native Pipecat Cloud integration over the Daily transport — personas, flows, built-in metrics, and a drop-in observer for capturing production calls. Observability and online evaluation for voice agents. Auto-instrument Pipecat with OpenInference (OpenTelemetry) to trace every turn, then run LLM-as-judge evals on live traffic in Arize AX or open-source Phoenix. Building an evaluation integration for Pipecat? We welcome contributions to this page. Open a PR on the [docs repository](https://github.com/pipecat-ai/docs). Pipecat's other building blocks feed into any evaluation workflow: [Metrics](/pipecat/fundamentals/metrics) for TTFB, processing time, and usage; [Saving Transcripts](/pipecat/fundamentals/saving-transcripts) for offline analysis; [OpenTelemetry](/api-reference/server/utilities/opentelemetry) for latency traces; and [Observers](/api-reference/server/utilities/observers/observer-pattern) for custom instrumentation. ## Next steps The full scenario format: turns, expectations, modalities, and the judge. Spawn multiple agents and run many scenarios concurrently from a manifest. Let a coding assistant write agent code, run evals, and iterate automatically until the agent is better. # Evals Lifecycle Source: https://docs.pipecat.ai/pipecat/evals/lifecycle.md How local Pipecat Evals fit with platform testing, monitoring, and team-scale evaluation. Pipecat Evals gives you a durable first layer for agent quality: fast, local, repeatable checks for the behavior your agent should preserve as code, prompts, models, and tools change. As the agent moves toward production, the evaluation surface expands: local scenarios stay active, and realistic simulations, audio-signal metrics, trace-backed checks, submitted production calls, and review workflows give product, QA, and operations teams shared quality signals. Strong evaluation programs keep both layers active: - **Pipecat Evals** for executable specifications close to the codebase. - **An [evaluation platform](/pipecat/evals/overview#production-evaluation)** for production-like conversations, richer voice analysis, dashboards, monitoring, human review, and longitudinal quality trends. ## Lifecycle at a glance ## Start local Use Pipecat Evals as soon as the agent has behavior worth preserving. This usually starts before deployment, while the agent still runs on a laptop or in a pull request. Good local evals look like small executable specs: - "The agent greets on connect." - "The agent remembers the user's name two turns later." - "The agent calls `lookup_order` before answering an order-status question." - "The agent recovers when the user interrupts a long answer." - "The first response starts within the expected latency budget." This is where Pipecat Evals is strongest. Text mode keeps the loop fast and cheap while you iterate on prompts, logic, and function calling. Audio mode adds an end-to-end check for VAD, STT, TTS, turn-taking, and speech transcription before you merge or release. The [Eval Suites](/pipecat/evals/suites) page describes Pipecat's own release evals as a manifest with 100+ example agents. The same pass/fail result is also useful for coding assistants because it gives them a concrete command to run and a clear failure artifact to fix against. ## Put local evals in CI Once a few scenarios exist, run them on every meaningful change. A small set of behavior-critical scenarios gives engineers and coding assistants a clear pass/fail signal. Use Pipecat Evals in CI when: - The agent is still changing quickly. - The question is "did this code or prompt change preserve an expected behavior?" - The expected behavior can be expressed as a scripted conversation. - The failure should block a pull request. - The debug artifact should live next to the code as a log or trace file. As a rule of thumb, local evals should cover the sharp edges that are easy to state and expensive to rediscover manually. ## When to add a simulation and evaluation platform Add a platform when the risk you need to test extends beyond a small local scenario. Keep the Pipecat suite in place, then layer on broader coverage, shared workflows, and production feedback. | Signal | Outside Pipecat Evals' scope | What to add | | --------------------------------------- | --------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | Realistic caller behavior matters | Scripted turns prove one path, not how varied users phrase, interrupt, or recover | Simulated callers with varied personas, edge cases, and abuse attempts | | Voice behavior is part of the product | Transcripts hide TTS loops, clipping, dropouts, phoneme stretch, timbre drift, and odd pauses | Audio-signal metrics, a Speech Artifact Score, and an audio LLM judge that scores the audio itself | | The deployed transport matters | The local eval transport isn't a real WebSocket, Pipecat Cloud, SIP, or telephony path | Tests over the deployed path users actually hit | | Volume and concurrency matter | Local suites skip sustained load, burst traffic, queueing, and service limits | Load tests for concurrency, latency, error rates, and saturation | | Hidden execution state matters | Transcripts don't show tool success, arguments, downstream errors, or timing | Trace-based checks on tool calls, span attributes, errors, timing, and custom metrics | | Product, QA, or Ops need to participate | YAML and CI logs aren't a shared review workspace | Review queues, per-reviewer assignments, annotations, agreement scores, and dashboards | | The agent is live with users | Pre-merge checks miss production drift and new real-world failures | Production monitoring that scores live calls and alerts on regressions | | Deterministic audio regression | Synthesized audio can't replay the exact call or wording that broke | Exact transcripts and pre-recorded audio as fixed regression cases | | Comparing releases, vendors, or configs | Local pass/fail can't do trend analysis or bake-offs | Persisted runs, metrics, recordings, traces, and score distributions | ## Choose the right layer Think of local evals as **behavioral unit tests for the agent**. Think of an evaluation platform as **system testing and quality operations for the agent in the world**. Local development, pull-request gates, coding-assistant loops, scripted behavioral specs, function-call assertions, latency budgets, and fast text mode iteration. Multi-turn simulations, caller variation, realistic voice and telephony paths, audio-signal metrics, trace metrics, submitted production calls, dashboards, human review, scheduled runs, and agent-native CLI, MCP, or skill-based workflows. ## Example: appointment booking Suppose your Pipecat agent books appointments. Start with a local scenario that protects the core behavior: ```yaml appointment_booking.yaml name: appointment_booking turns: - user: "Can you book me for Tuesday at 3 PM Pacific?" expect: - event: function_call calls: - name: book_appointment args: day: "Tuesday" time: "3 PM" timezone: "America/Los_Angeles" - event: response eval: "confirms the appointment request with the user" ``` This is exactly the kind of check you want in your repository. If a prompt edit stops the tool call, CI should fail. Before this agent handles real users, the quality question gets broader: - What happens when the caller changes the time three turns later? - Can the agent handle an impatient caller who interrupts while it is checking availability? - Does the agent still work when the caller is in a noisy room? - Did the booking API actually succeed, or did the agent only say that it did? - Are users getting frustrated when the available slots are limited? - Is the booking-success rate drifting after a model or voice-provider change? Those are platform-level questions. The same booking flow can become a simulation suite that varies caller goals, phrasing, interruptions, voice conditions, transport paths, load, and tool outcomes before launch. After launch, production conversations can feed monitoring, review, and future regression coverage. ## Simulate before launch Local scenarios are intentionally crisp. Platform simulations broaden the same flow across realistic user variation, voice conditions, transport paths, load, abuse attempts, and tool behavior. For regressions that need exact replay, use fixed transcripts, scripted turns, or pre-recorded audio. For tool-heavy flows, include traces so evaluation can check what happened under the transcript: tool calls, arguments, span attributes, errors, timing, and custom numerical metrics from [OpenTelemetry](/api-reference/server/utilities/opentelemetry). For the appointment agent, a transcript might show: > "You're all set for Tuesday at 3 PM." A simulation can also check that `book_appointment` was called with the requested day, time, and timezone, that the tool returned success, and that later spans stayed clear of errors, retry failures, and cancellations. When judgment matters, route selected conversations to human review. Review queues, labels, assignments, and agreement scores turn ambiguous calls into better metrics, tighter ground truth, and better future cases. ## Monitor production conversations Once users are live, send completed production conversations to the platform for monitoring. Score the same quality signals over time, route ambiguous calls into review, and track trends by agent version, transport, scenario, and customer segment. ## Feed findings back into tests Production monitoring and human review should create the next evaluation inputs: new local Pipecat scenarios for crisp regressions, new simulation cases for broader user behavior, new trace metrics for hidden tool failures, and new monitoring metrics for recurring production patterns. ## Practical adoption path Start small and grow in layers: 1. **Create 5-10 Pipecat scenarios** for the agent's most important scripted behaviors. 2. **Run the suite in CI** and require it before merging prompt, model, tool, or pipeline changes. 3. **Add audio-mode checks** before release for the flows most sensitive to STT, TTS, VAD, and turn-taking. 4. **Add platform simulations** when you need realistic multi-turn behavior, varied callers, telephony or WebSocket coverage, load testing, trace checks, and audio-signal metrics. 5. **Instrument traces** for tool-heavy workflows so evaluations can verify what happened under the hood. 6. **Expose platform controls to your coding assistant** through CLI, MCP, or agent-skill surfaces so the same assistant that fixes local evals can launch simulations, inspect failures, and triage monitoring results. 7. **Send production calls to monitoring** once users are live, then turn recurring failures into new simulations, metrics, human-review projects, or local scenarios. ## Next steps Write and run the first local scenario against an existing agent. Review the available platform integrations for simulation and monitoring. Learn the YAML format for turns, expectations, function calls, interruptions, latency budgets, and audio mode. See one concrete setup path for simulations, monitoring, traces, and team review workflows. # Evals Quickstart Source: https://docs.pipecat.ai/pipecat/evals/quickstart.md Run your first Pipecat behavioral eval against an existing agent with pipecat eval and a simple scenario. This guide takes an existing agent, starts it with the eval transport, and runs a two-turn scenario against it. Total time: a few minutes. ## Prerequisites - A working Pipecat agent that uses `create_transport()` and the development runner (the standard pattern from the [quickstart](/pipecat/get-started/quickstart) and all Pipecat examples), with its usual service API keys in `.env`. - The Pipecat CLI: `uv tool install "pipecat-ai[cli]"` (or add `pipecat-ai[cli]` to your project and run the commands below with `uv run pipecat eval`). - A judge LLM. Either: - **Ollama** (local, the default): install [Ollama](https://ollama.com) and run `ollama pull gemma2:9b`, or - **OpenAI**: set `OPENAI_API_KEY` and point the scenario's `judge:` block at it (shown below). If your agent uses `create_transport()`, it supports the eval transport with a one-line addition to its `transport_params`: ```python from pipecat.transports.websocket.server import WebsocketServerParams transport_params = { "eval": lambda: WebsocketServerParams( audio_in_enabled=True, audio_out_enabled=True, ), # ... your other transports (daily, webrtc, twilio, ...) } ``` Then start the agent with `-t eval`: ```bash uv run bot.py -t eval ``` ``` 🚀 Bot ready! (eval transport on ws://localhost:7860) ``` Instead of connecting to Daily or WebRTC, the agent now hosts a local WebSocket server and waits for the eval harness to connect. Nothing else in the agent changes: same pipeline, same services, same event handlers. The harness talks to your agent over RTVI. `PipelineWorker` adds an `RTVIProcessor` and `RTVIObserver` automatically, so the standard agent setup needs no extra wiring. All Pipecat example agents already include the `"eval"` transport entry. A scenario is a YAML file describing a scripted conversation and the behavior you expect. Save this as `scenarios/capital_question.yaml`: ```yaml name: capital_question turns: # The agent greets on connect; wait for the greeting before speaking. - expect: - event: response eval: "the bot opens the conversation with a greeting or an offer to help" - user: "What is the capital of Germany?" expect: - event: response eval: "the response says the capital of Germany is Berlin" ``` ```yaml name: capital_question judge: eval: service: openai model: gpt-4o-mini turns: # The agent greets on connect; wait for the greeting before speaking. - expect: - event: response eval: "the bot opens the conversation with a greeting or an offer to help" - user: "What is the capital of Germany?" expect: - event: response eval: "the response says the capital of Germany is Berlin" ``` Each turn optionally sends a user utterance and lists the events expected in response. The `eval:` field is a natural-language criterion checked by the judge LLM, so the test passes whether the agent says "Berlin is the capital of Germany" or "That would be Berlin!". This scenario runs in **text mode** (the default): the user turn is sent as text and the agent's TTS is skipped automatically, so the whole conversation costs nothing in audio services and finishes in seconds. Ollama with `gemma2:9b` is the default judge, which is why the first tab has no `judge:` block. To use a different judge LLM, add a `judge.eval:` block as in the OpenAI tab. With the agent still running, run the scenario from another terminal: ```bash pipecat eval run scenarios/capital_question.yaml ``` The harness connects to `ws://localhost:7860` (override with `--bot-url`), drives the conversation, and reports the result. Pass `-v` to watch each turn resolve: ``` turn 0 → (observe) ✓ llm_response — "Hello! How can I help you today?" turn 1 → "What is the capital of Germany?" ✓ llm_response — "The capital of Germany is Berlin." ✓ ws://localhost:7860 capital_question (3402ms) 1/1 passed · 3.4s ``` The command exits `0` when everything passes and `1` otherwise, so it slots directly into scripts and CI. Each scenario also writes a decision trace to `.eval.log`, which shows every event the harness saw and why each assertion passed or failed. Change the criterion to something false, for example `"the response says the capital of Germany is Madrid"`, and run again: ``` ✗ ws://localhost:7860 capital_question Failed (1): ✗ ws://localhost:7860 capital_question • turn 1 expectation 0 (llm_response): judge said no: the reply says the capital is Berlin, not Madrid 0/1 passed, 1 failed · 4.1s ``` A failing eval tells you which turn, which expectation, and why. That message (plus the `.eval.log` trace) is what you, or your AI coding assistant, iterate against. ## Where to go next - Learn the full scenario format, including multi-turn conversations, function call assertions, interruptions, latency budgets, and text vs audio modes, in [Writing Scenarios](/pipecat/evals/scenarios). - Have many scenarios or agents? Let Pipecat spawn the agents for you with [Eval Suites](/pipecat/evals/suites). - Want your coding assistant to run these for you? See [The Eval Loop](/pipecat/evals/the-eval-loop). # Writing Scenarios Source: https://docs.pipecat.ai/pipecat/evals/scenarios.md The scenario file format: configuration, turns, events, and assertions. A scenario is a YAML file describing a scripted conversation and the events you expect your agent to emit. This page covers the full format. If you haven't run a scenario yet, start with the [quickstart](/pipecat/evals/quickstart). ## Anatomy of a scenario ```yaml name: multi_turn # required: the eval's name judge: # optional: judge modality and LLM (defaults shown below) eval: service: ollama model: gemma2:9b turns: # required: the conversation, in order - user: "My name is Alex, and I'm planning a trip to Italy." expect: - event: response eval: "acknowledges the user's message (the name Alex and/or the trip to Italy)" - user: "Remind me, what's my name and where am I going?" expect: - event: response eval: "recalls that the user's name is Alex and the destination is Italy" ``` Each turn optionally sends a user utterance (`user:`) and lists the events expected in response (`expect:`). Expected events must arrive in the order listed, but the agent may emit other events in between, so you don't have to enumerate everything it does. The rest of this page is in four parts: Scenario-wide setup: modalities, the judge, context, and lifecycle. Drive each turn with an utterance, keypresses, an image, or timing. The semantic events the agent emits, and what each one means. Check an event's content or timing with `eval:`, `text_contains:`, and more. ## Configuration Everything in this section is optional. A scenario with no configuration blocks runs entirely in [text mode](#text-and-audio-modes) with the default judge, which is the fastest way to start. To write the turns themselves, skip ahead to [User turns](#user-turns) and [Events](#events). ### Text and audio modes Two top-level blocks control a scenario's modalities, and each has its own `modality:` field: - `user:` sets how each turn's utterance is delivered to the agent: sent as text, bypassing its STT (`modality: text`), or synthesized into real speech (`modality: audio`). - `judge:` sets what the judge evaluates: the agent's LLM text, with its TTS skipped (`modality: text`), or a transcription of its actual spoken audio (`modality: audio`). When `modality:` isn't specified, or a block is omitted entirely, it defaults to `text`. The two sides are also independent: you can drive the agent with text while judging its real speech, or speak to it and judge the LLM text. A scenario with neither block runs entirely in text mode. No audio flows on either side, so this is the fastest and cheapest way to test prompts, conversational logic, and function calling: no audio service cost, and a multi-turn scenario finishes in seconds. The judge LLM is the only service the harness itself needs (Ollama with `gemma2:9b` by default). The top-level `user:` block here only configures delivery. Each turn's `user:` field (see [User turns](#user-turns)) is the utterance itself, and is written the same way in both modes. ### User delivery with `user:` **Text (the default).** Each turn's utterance is sent to the agent as text, bypassing its STT. This needs no configuration; it's equivalent to: ```yaml user: modality: text ``` **Audio.** Each turn's utterance is synthesized by a TTS the harness runs and streamed into your agent's pipeline at real-time cadence, exercising its VAD, turn detection, and STT exactly as a live microphone would. Synthesized audio is cached across runs, so repeated turns don't re-synthesize. The `speech:` block (the TTS service and voice) is required: ```yaml user: modality: audio speech: service: kokoro # local TTS, no API key, no per-run cost voice: af_heart sample_rate: 16000 ``` The built-in speech services are `kokoro`, a local model and the recommended default, and `cartesia` (HTTP) when you want a cloud voice. ### Judging with `judge:` **Text (the default).** The agent's TTS is skipped automatically, including any on-connect greeting, and the judge evaluates the LLM's text output. Fast and silent; equivalent to: ```yaml judge: modality: text ``` **Audio.** The agent speaks for real. The harness captures its synthesized audio, transcribes it with the configured STT, and the `response` event becomes that transcription, so the judge evaluates what a user would actually have heard. This is the true end-to-end check: STT in, LLM in the middle, TTS out. The `transcription:` block is required: ```yaml judge: modality: audio transcription: service: moonshine # STT for the agent's audio (or: whisper) model: small-streaming eval: service: ollama # the judge LLM model: gemma2:9b ``` The built-in transcribers are `moonshine` and `whisper`, both local models. When `transcription.service:` is omitted, it defaults to `moonshine`. In either modality, the `judge.eval:` block selects the judge LLM: `ollama` (the default, `gemma2:9b`), `openai`, or any OpenAI-compatible endpoint via `endpoint:`. This is the LLM that decides [`eval:`](#semantic-judging-with-eval) assertions. ### Custom services with `factory:` To use a TTS or STT beyond the built-ins, both blocks accept a `factory:` escape hatch: a dotted path to a callable that receives the block's mapping and the resolved sample rate, and returns the service. Any extra keys you put in the block are passed through to your factory: ```yaml user: modality: audio speech: factory: "my_evals.services.make_tts" voice: luna # available to your factory as speech_cfg["voice"] judge: modality: audio transcription: factory: "my_evals.services.make_stt" ``` ```python my_evals/services.py import os from pipecat.services.fal.stt import FalSTTService from pipecat.services.rime.tts import RimeHttpTTSService def make_tts(speech_cfg, sample_rate): return RimeHttpTTSService( api_key=os.environ["RIME_API_KEY"], settings=RimeHttpTTSService.Settings(voice=speech_cfg["voice"]), sample_rate=sample_rate, ) def make_stt(transcription_cfg, sample_rate): return FalSTTService(api_key=os.environ["FAL_KEY"]) ``` The service your factory returns must be a local model or an HTTP-based service. WebSocket-streaming services aren't supported: they need a running pipeline to manage their connection lifecycle, and keeping them out keeps the evals code simple. For a fully custom setup (your own caching, a pre-built service instance), construct `EvalSpeech` or `EvalTranscriber` directly and inject them through the [library](/pipecat/evals/library). ### Seeding the context with `context:` By default the harness leaves the bot's LLM context alone: whatever the bot sets up for itself (for example, a system prompt added in its connect handler) is what the scenario runs against. Provide `context:` to replace that with messages of your own, which lets a scenario start mid-conversation: ```yaml context: - role: developer content: "The user has already introduced themselves as Alex." - role: assistant content: "Nice to meet you, Alex! How can I help?" ``` The harness sends these right after the bot-ready handshake as an `LLMMessagesUpdateFrame` that replaces the bot's context wholesale. Omit `context:` and the harness sends nothing, leaving the bot's own context in place. ### Sharing config with `!include` Any value can be pulled from another file with `!include`, resolved relative to the scenario file. This keeps per-scenario noise down when a whole directory of scenarios shares the same audio setup: ```yaml name: capital_question user: !include user_audio.yaml judge: !include judge_audio.yaml turns: - user: "What is the capital of Germany?" expect: - event: response eval: "the response says the capital of Germany is Berlin" ``` ### Running scenarios back to back By default the bot keeps running between scenarios. When a scenario ends its eval connection closes, but the eval transport suppresses the bot's `on_client_disconnected` handler, so the pipeline stays up to serve the next scenario. This is what lets `pipecat eval run a.yaml b.yaml c.yaml` drive a whole list against one bot instance with no reboot between them, which keeps a run fast. The trade-off is that anything the bot accumulated in one scenario is still there for the next. For results to be independent, each scenario has to start from a clean slate, and clearing that state is split between the harness and your bot: - **Conversation context**: seed or clear it per scenario with [`context:`](#seeding-the-context-with-context). The harness replaces the bot's LLM context with the messages you provide (via an `LLMMessagesUpdateFrame`); without it, the previous conversation carries forward, which is rarely what you want across independent scenarios. - **Application state**: counters, flags, cached data, anything your bot holds outside the LLM context. The harness can't see this, so resetting it is your bot's job. A common place is the bot's connect handler, which runs again for each scenario's connection. ### Exercising the disconnect path Some bots do meaningful work in `on_client_disconnected`, like a goodbye message, session teardown, or resource cleanup. Because the eval transport suppresses that handler by default, set `trigger_disconnect: true` on a scenario to fire it when that scenario ends: ```yaml name: test_goodbye_on_disconnect trigger_disconnect: true turns: - user: "Thanks for your help!" expect: - event: response eval: "the agent acknowledges the thanks" # on_client_disconnected fires after this turn, so the agent can # send a goodbye message or clean up resources. ``` Bots often cancel their pipeline in `on_client_disconnected`, so a scenario with `trigger_disconnect: true` usually ends the bot process. Treat it as a terminal run, last in a list. Enable it for every scenario in a run with `pipecat eval run --trigger-disconnect`; a scenario's own `trigger_disconnect` field still takes precedence. This is independent of `--stop-bot`, which tears the bot down via an `eval-cancel` message regardless of the disconnect handler. ## User turns Each turn drives the agent by speaking (a `user:` utterance) or pressing keys (a `dtmf:` sequence); the two are mutually exclusive. A turn can also register an `image:`, or be observation-only with no input. `send_after:` controls when the input is sent. ### Utterances with `user:` Each turn's `user:` field is the user's utterance for that turn, a plain string. You write it the same way in both modes; whether it's delivered as text or synthesized into real speech is set once by the [`user:` block](#user-delivery-with-user), not per turn. A turn without a `user:` field is observation-only: the harness just waits for the expected events. This is how you test agent-first behavior like an on-connect greeting: ```yaml turns: # No user input: just wait for the agent to speak first. - expect: - event: response eval: "the bot opens the conversation with a greeting or an offer to help" ``` ### DTMF keypresses with `dtmf:` Instead of a `user:` utterance, a turn can press phone keypad keys with `dtmf:`. The two are mutually exclusive: a turn either speaks or presses keys. This drives keypad menus (IVR) and any agent that reacts to telephony tones: ```yaml turns: - dtmf: "123#" expect: - event: user_transcription text_contains: "DTMF: 123#" - event: response eval: "confirms the entered digits" ``` Each character is sent as one `InputDTMFFrame`, the same path a telephony transport's keypress takes, regardless of the scenario's `user:`/`judge:` modality. Valid characters are the keypad entries `0`-`9`, `*`, and `#`; any other character is a parse error. Quote the value in YAML (`dtmf: "123#"`). An unquoted `#` starts a YAML comment, so `dtmf: 123#` would silently drop the `#`. An unquoted all-digit sequence (`dtmf: 123`) is coerced to a string for you, but quoting is the safe habit. A bot running a [`DTMFAggregator`](/api-reference/server/utilities/dtmf-aggregator) accumulates the keys and flushes them into a `DTMF: ...` transcription, which (with the default transcription-based turn-start strategy) drives a full user turn: `user_started_speaking`, `user_transcription`, `user_stopped_speaking`, and the agent's response. So a `dtmf:` turn can assert on `user_transcription` and `response` just like a spoken turn. The aggregator flushes either on the `#` terminator or on its idle timeout. To exercise the idle-timeout path, omit the `#` and pace the keys with a time-based [`send_after:`](#scheduling-with-send_after): ```yaml turns: # Flushes immediately on the '#' terminator. - dtmf: "1#" expect: - event: response eval: "states the business hours" # No terminator: the aggregator flushes on its idle timeout instead. - dtmf: "2" expect: - event: response eval: "gives the office location" ``` Like any input turn, `expect:` is optional on a `dtmf:` turn: omit it for a turn that only presses keys, with the assertion living on a later turn. ### Vision with `image:` A turn may register an image with `image:` (a path relative to the scenario file). When a vision agent requests a user image during the turn, the eval transport serves it: ```yaml turns: - user: "What do you see in this image?" image: assets/cat.jpg expect: - event: response eval: "the response describes a cat" ``` ### Scheduling with `send_after:` `send_after:` controls when a turn's input (its `user:` utterance or [`dtmf:`](#dtmf-keypresses-with-dtmf) keypresses) is sent, either relative to a prior event or after a plain delay. Anchoring it to an event is how you script barge-in tests: ```yaml turns: - user: "Tell me a long, detailed story about the history of Paris." expect: - event: llm_started # Interrupt 2 seconds after the agent starts its long answer. - user: "Actually, never mind that. What's the capital of Japan?" send_after: event: llm_started delay_ms: 2000 expect: - event: response eval: "the response says the capital of Japan is Tokyo, instead of continuing the Paris story" ``` The `event:` anchor is optional. A bare `send_after: { delay_ms: 500 }` is a pure time delay measured from the previous turn's send, with no event to wait on. This is handy for pacing turns by time rather than off a bot event (for example, spacing out [DTMF keypresses](#dtmf-keypresses-with-dtmf) to exercise an aggregator's idle-timeout flush): ```yaml turns: - dtmf: "1" # Wait 1.5s after the first keypress before sending the next. - dtmf: "2" send_after: delay_ms: 1500 ``` A `send_after:` with no `event:` and a zero `delay_ms` is rejected as a no-op: give it an `event:`, a positive `delay_ms`, or both. ## Events Scenarios assert on a small set of semantic events, mapped from the RTVI messages the agent emits: | Event | Meaning | | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `response` | The agent's reply. In audio mode this is a transcription of the agent's actual synthesized speech; in text mode it resolves to `llm_response`. Prefer this for content checks. | | `llm_response` | The LLM's text output for the turn. Available in both modes. | | `tts_response` | The text the TTS reports speaking, one segment at a time. Audio mode only. | | `llm_started` | The LLM began generating a response. | | `function_call` | The LLM called a function. | | `user_transcription` | The agent's STT finalized a transcription of the user. Audio mode only, except on [DTMF turns](#dtmf-keypresses-with-dtmf), where the aggregated keys become a transcription in either mode. | | `user_started_speaking` | The agent's VAD detected the start of user speech. Audio mode, or a [DTMF turn](#dtmf-keypresses-with-dtmf) in either mode. | | `user_stopped_speaking` | The agent's VAD detected the end of user speech. Audio mode, or a [DTMF turn](#dtmf-keypresses-with-dtmf) in either mode. | Use `response` for the agent's reply unless you have a reason not to. It's modality-agnostic: the same scenario judges LLM text in text mode and the transcription of real spoken audio in audio mode, so one file covers both. ## Assertions Each entry in `expect:` names an event and, optionally, asserts on its content or timing. ### Semantic judging with `eval:` The `eval:` field is a natural-language criterion that the event's text must satisfy, decided by the [judge LLM](#judging-with-judge): ```yaml - user: "What's 2 plus 2?" expect: - event: response eval: "the response says the answer is four" ``` The judge sees the whole conversation so far, so it can resolve terse or context-dependent replies (like "That's four"). It also understands that audio-mode responses come from a speech-to-text pass and judges intended meaning rather than exact spelling, so "for" transcribed instead of "four" still passes. The judge handles interim replies gracefully: if the agent says "Let me check on that." before the real answer, the harness keeps accumulating response text and re-judges until the criterion is met or the time budget runs out. `eval:` only makes sense on the agent's text output (`response`, `llm_response`, `tts_response`). ### Substring checks with `text_contains:` For exact content, `text_contains:` does a plain substring check, with no judge round-trip: ```yaml - user: "What is the capital of France?" expect: - event: response text_contains: "Paris" ``` ### Latency budgets with `within_ms:` `within_ms:` bounds how long after the turn's user send the event may arrive. All of a turn's expectations share that one anchor: ```yaml - user: "What is the capital of France?" expect: - event: llm_started within_ms: 2000 # the LLM must start responding within 2s - event: response text_contains: "Paris" ``` When omitted, an expectation defaults to a generous 60 second budget (configurable with `--timeout`), so timing is only asserted when you ask for it. Because every deadline is measured from the send, time spent matching earlier expectations counts against later ones. In the example above, if `llm_started` arrives at 1.5 seconds, the `response` (with the default 60 second budget) has 58.5 seconds left, and a turn that stalls completely fails within a single budget rather than one per expectation. ### Function calls A `function_call` expectation asserts that the turn invoked one or more tools. List the expected calls under `calls:`; they're matched by name in any order, and the expectation passes once all are found: ```yaml - user: "What's the weather in San Francisco? And recommend a restaurant." expect: - event: function_call calls: - name: get_current_weather args: { location: "San Francisco" } - name: get_restaurant_recommendation - event: response eval: "describes the weather and recommends a restaurant" ``` `args` is a subset check: every listed key/value must be present in the call's arguments, and extra arguments are ignored. A single expected call can use the `name:`/`args:` shorthand directly on the expectation, and a bare `function_call` with neither just asserts that some call happened. ## Next steps Let a coding assistant write agent code, run evals, and iterate automatically until the agent is better. # Eval Suites Source: https://docs.pipecat.ai/pipecat/evals/suites.md Spawn agents and run many scenarios concurrently from a single manifest. `pipecat eval run` tests scenarios against an agent you started yourself. A **suite** goes one step further: you list agents and scenarios in a manifest, and `pipecat eval suite` spawns each agent with its eval transport on its own port, runs its scenarios, tears it down, and aggregates the results, several runs at a time. Suites are the right tool when you have more than one agent, more than a handful of scenarios, or want a single command for CI. Pipecat's own release evals are a manifest with 100+ example agents plus this command. ## The manifest ```yaml manifest.yaml concurrency: 4 # how many runs execute at once runs_dir: eval-runs # logs + recordings go to // record: false # record conversation audio (audio-mode scenarios) scenarios_dir: scenarios # scenario names resolve to /.yaml # How to start each agent. {python}, {bot}, and {port} are substituted per run. spawn: "{python} {bot} -t eval --port {port}" suite: - bot: bots/support-agent.py scenarios: [greeting, capital_question, multi_turn] - bot: bots/sales-agent.py scenarios: [greeting, weather_function_call] - bot: bots/vision-agent.py runner_body: scenarios/vision-body.json # optional --runner-body data scenarios: [vision_describe] ``` Paths in the manifest (`bots_dir`, `scenarios_dir`, `runs_dir`, the `bot:` entries) resolve relative to the manifest file, so a manifest is portable: check it into your repo and run it from anywhere. Scenarios are reusable across agents. One `greeting` scenario can cover every agent in the suite. An optional `runner_body:` points at a JSON file passed to the agent as `--runner-body`. It supplies session data the agent would normally receive in a `/start` request body (for example, a vision agent's image path). ## Running a suite ```bash pipecat eval suite manifest.yaml ``` In a terminal, a live dashboard shows each run's status, a running tally, and total time. When piped (in CI, or driven by a coding assistant), it streams one plain result line per run instead. The command exits `0` only if every run passes. Useful flags: ```bash pipecat eval suite manifest.yaml -p support # only bots whose path contains "support" pipecat eval suite manifest.yaml -s greeting # only the greeting scenario pipecat eval suite manifest.yaml -c 8 # 8 runs at a time pipecat eval suite manifest.yaml -n nightly # output to eval-runs/nightly/ pipecat eval suite manifest.yaml -a # record conversation audio pipecat eval suite manifest.yaml -d # save full per-pipeline debug logs ``` Everything except the `suite:` list can live in the manifest or be passed on the command line (the command line wins), so a manifest can be as minimal as a `suite:` list. ## Run output Each invocation writes to `//` (a timestamp when `-n` is omitted): ``` eval-runs/20260610_142200/ logs/ bots_support-agent.py__greeting.log # the agent process output bots_support-agent.py__greeting.eval.log # the harness's decision trace bots_support-agent.py__greeting.debug.log # per-pipeline harness logs (-d only) recordings/ bots_support-agent.py__greeting.wav # conversation audio (record: true or -a) ``` When a run fails, start with the `.eval.log` decision trace: it's a timestamped record of every event the harness saw, what it matched, what the judge said, and why an assertion failed. The agent's own log sits next to it. ## Testing one agent with many scenarios If you just want to run a batch of scenarios against an agent you already have running, you don't need a manifest. `pipecat eval run` accepts multiple scenario files and shares the suite's dashboard and tally: ```bash pipecat eval run scenarios/*.yaml --bot-url ws://localhost:7860 ``` By default the agent is left running afterward so it can serve more evals; pass `--stop-bot` to shut it down when the batch finishes. ## Suites in CI The exit code makes suites CI-ready with no extra glue: ```yaml # e.g. GitHub Actions - name: Run behavioral evals run: pipecat eval suite manifest.yaml ``` For deterministic, key-free CI runs, prefer text-mode scenarios and an OpenAI-compatible judge endpoint you control. Audio-mode scenarios work in CI too, but need the harness's TTS and STT services available (local models by default, which also need more CPU). # Using the Library Source: https://docs.pipecat.ai/pipecat/evals/library.md Run, build, and orchestrate evals from Python with the pipecat.evals API. Everything the `pipecat eval` CLI does is available as a library under `pipecat.evals`. Use it to run evals from your own test runner (pytest, a CI script, a custom dashboard), to build scenarios in code instead of YAML, or to customize pieces like the judge LLM. ## Running a scenario `EvalScenario.load()` parses a scenario file, and `EvalSession.from_scenario()` builds a ready-to-run session, constructing the judge, user speech, and transcriber the scenario calls for: ```python import asyncio from pipecat.evals.harness import EvalSession from pipecat.evals.scenario import EvalScenario async def main(): scenario = EvalScenario.load("scenarios/capital_question.yaml") session = EvalSession.from_scenario(scenario, "ws://localhost:7860") result = await session.run() if result.passed: print(f"PASS ({result.duration_ms}ms)") else: for failure in result.failures: print(f" {failure}") asyncio.run(main()) ``` The agent must already be running with its eval transport (`python bot.py -t eval`), just as with `pipecat eval run`. ### The result `run()` returns an `EvalResult`: | Field | Description | | --------------- | ------------------------------------------------------------------------------------------- | | `scenario_name` | Name of the scenario that ran. | | `passed` | Whether every assertion passed. | | `failures` | The failed assertions, each with the turn index, expectation index, event name, and reason. | | `duration_ms` | Wall-clock time the run took. | | `events_seen` | Every semantic event observed, for diagnostics. | | `debug_log` | The harness's timestamped decision trace (what the CLI writes to `.eval.log`). | | `skipped` | Set (with a reason) when the scenario was not run; such a result is neither pass nor fail. | This maps cleanly onto a pytest test: ```python import pytest from pipecat.evals.harness import EvalSession from pipecat.evals.scenario import EvalScenario @pytest.mark.asyncio async def test_capital_question(): scenario = EvalScenario.load("scenarios/capital_question.yaml") result = await EvalSession.from_scenario(scenario, "ws://localhost:7860").run() assert result.passed, "\n".join(str(f) for f in result.failures) ``` ## Building scenarios in code Scenarios are plain dataclasses, so you can construct them programmatically, generating turns from a dataset, parameterizing a template, or skipping YAML entirely: ```python from pipecat.evals.scenario import EvalExpectation, EvalScenario, EvalTurn scenario = EvalScenario( name="capital_question", turns=[ EvalTurn( user="What is the capital of Germany?", expect=[ EvalExpectation( event="llm_response", eval="the response says the capital of Germany is Berlin", ) ], ) ], ) ``` The modality-agnostic `response` event is resolved while parsing YAML. When constructing scenarios in code, use `llm_response` for text mode directly (or `response` only when you also configure audio judging). ## Customizing the judge `from_scenario()` builds the judge from the scenario's `judge:` block, but you can inject your own. `EvalJudge` works with any Pipecat LLM service backed by an OpenAI-compatible API: ```python import os from pipecat.evals.harness import EvalSession from pipecat.evals.judge import EvalJudge from pipecat.services.openai.llm import OpenAILLMService llm = OpenAILLMService( api_key=os.environ["OPENAI_API_KEY"], settings=OpenAILLMService.Settings(model="gpt-4o-mini"), ) session = EvalSession.from_scenario( scenario, "ws://localhost:7860", judge=EvalJudge(llm), ) ``` The same injection points exist for the user's synthesized voice (`speech=`, wrapping any `TTSService` in an `EvalSpeech`) and the transcriber used for the agent's spoken audio (`transcriber=`, wrapping any `STTService` in an `EvalTranscriber`). The wrapped services can be local models or HTTP-based; WebSocket-streaming services are rejected, since they need a running pipeline to manage their connection lifecycle. ## Observing progress Pass `on_progress` to get a callback as each turn and expectation resolves, which is how the CLI implements its `--verbose` output: ```python from pipecat.evals.harness import EvalSession, EvalTurnProgress def show(p: EvalTurnProgress): print(f"turn {p.turn_index} [{p.status}] {p.event_name} {p.detail}") session = EvalSession.from_scenario(scenario, url, on_progress=show) ``` ## Orchestrating suites `EvalManifest` and `EvalSuite` are the library behind `pipecat eval suite`: the suite spawns each agent with its eval transport on its own port, runs its scenarios, and executes several runs concurrently: ```python import asyncio from pathlib import Path from pipecat.evals.suite import EvalManifest, EvalSuite async def main(): manifest = EvalManifest.load("manifest.yaml") suite = EvalSuite(manifest) # Optionally narrow the runs, like the CLI's -p / -s flags. suite.filter(pattern="support") await suite.run( Path("eval-runs/logs"), on_update=lambda run: print(run.bot, run.scenario, run.status), ) for run in suite.runs: verdict = run.error or ("passed" if run.result and run.result.passed else "failed") print(f"{run.bot} / {run.scenario}: {verdict}") asyncio.run(main()) ``` Each run is mutated in place as it executes (`status`, `result`, `error`, `duration_ms`), so a live display can render directly from `suite.runs`. `EvalManifest.load()` accepts keyword overrides for every manifest value (`concurrency`, `base_port`, `spawn`, `scenarios_dir`, and so on), mirroring the CLI flags. # The Eval Loop Source: https://docs.pipecat.ai/pipecat/evals/the-eval-loop.md Close the loop: Pipecat evals give an AI coding assistant a pass/fail signal it can read, so it writes agent code, runs evals, and iterates. Evals turn agent quality into a signal an AI coding assistant can read. That closes the loop: instead of asking an assistant to "improve the prompt" and judging the result by hand, you describe the desired behavior as a scenario and let the assistant iterate until the eval passes, and the agent gets better with every pass. Think of it as a REPL for agent behavior: the assistant writes a change, evals it, reads a pass/fail result, and loops, except the eval step already contains the judgment, so the cycle can close without a human reading the output. ## The loop 1. **Describe the behavior as a scenario.** A scenario file is an executable specification: the conversation, the expected events, and the criteria a response must meet. 2. **The assistant changes the agent.** A prompt edit, a new tool, a pipeline change. 3. **The assistant runs the evals.** One command, either against a running agent (`pipecat eval run`) or letting the suite spawn the agent itself (`pipecat eval suite`). 4. **The assistant reads the result.** A non-zero exit code, a per-assertion failure message ("turn 1 expectation 0 (llm_response): judge said no: ..."), and a full decision trace in `.eval.log`. 5. **Repeat until green.** Steps 2 through 5 need no human in the loop. You review the final diff with the evidence that it works attached. ## Why this works well for coding assistants The framework was built to be driven by tools, not just humans: - **One command, one exit code.** `pipecat eval run scenarios/*.yaml` exits `0` on success and `1` on failure, so an assistant knows mechanically whether it's done. - **Plain-text output when piped.** Outside a terminal the CLI streams one result line per scenario instead of rendering a live dashboard, which is exactly what an assistant running shell commands sees. - **Actionable failures.** Failures name the turn, the expectation, and the reason, including what the judge said. The `.eval.log` decision trace shows every event the harness observed, so "why did this fail" is answerable from files. - **Suites are self-contained.** `pipecat eval suite` spawns the agents itself, so an autonomous loop doesn't need to manage processes: edit, run one command, read the result. - **Text mode is fast and cheap.** Iterating on prompts and logic skips STT and TTS entirely, so an assistant can afford to run the evals after every change. ## Setting up your project Keep scenarios in the repo next to the agent and tell your assistant how to run them. For example, in your project's `CLAUDE.md` or `AGENTS.md`: ```markdown ## Behavioral evals Evals live in `scenarios/`. To verify any change to the agent's behavior: 1. Start the agent: `uv run bot.py -t eval` (serves ws://localhost:7860) 2. Run the evals: `pipecat eval run scenarios/*.yaml` The command exits non-zero on failure and prints each failed assertion. Each scenario writes a decision trace to `.eval.log`; read it to understand a failure before changing code. When you add or change agent behavior, add or update a scenario in `scenarios/` to cover it. ``` With that in place, a request like this becomes fully verifiable: > Add a `get_order_status` tool to the agent and make sure it gets called when the user asks where their order is. Add a scenario for it and run the evals until they pass. The assistant writes the tool, writes the scenario (a `function_call` assertion plus a judged response), runs `pipecat eval run`, reads any failure, and fixes its own work. ## Evals as acceptance criteria You can also run the loop in the other direction: write the scenario first, watch it fail, and hand the failure to the assistant. The scenario is the spec, and "make this pass" is the task. ```yaml order_status.yaml name: order_status turns: - user: "Where's my order? The number is 12345." expect: - event: function_call calls: - name: get_order_status args: { order_id: "12345" } - event: response eval: "tells the user the status of their order" ``` This is test-driven development for agent behavior, with the judge LLM absorbing the fuzziness that makes conversational output hard to assert on with string matching. ## Guardrails A few practices keep autonomous loops honest: - **Review scenario changes like code.** An assistant that can edit scenarios can also weaken them. Failing evals should usually be fixed in the agent, not in the scenario. - **Keep a regression set.** As behaviors accumulate, so should scenarios. Run the full set (or a suite) before merging, not just the scenario being worked on. - **Gate merges in CI.** `pipecat eval suite manifest.yaml` in CI makes "the evals pass" a property of the branch, whoever (or whatever) wrote it. See [Eval Suites](/pipecat/evals/suites). - **Use audio mode for the final check.** Iterate in text mode for speed, then run the audio variants before release to cover the full STT, LLM, and TTS path. ## Next steps Give your coding assistant access to Pipecat docs and source context. Layer in simulation platforms and observability once your agent is deployed. # Arize Source: https://docs.pipecat.ai/pipecat/evals/platforms/arize.md Observability and online evaluation for Pipecat voice agents, powered by OpenInference auto-instrumentation and OpenTelemetry. ## Overview [Arize](https://arize.com) provides AI observability and evaluation for agents in development and production. It comes in two products that share the same OpenTelemetry and OpenInference foundation: [Arize AX](https://arize.com/docs/ax), the hosted platform that gives AI engineers and product managers the tools to observe, improve, and evaluate their AI agents and applications, and [Phoenix](https://arize.com/docs/phoenix), the open-source AI observability platform for experimentation, evaluation, and troubleshooting. Arize maintains a Pipecat instrumentor, [`openinference-instrumentation-pipecat`](https://github.com/Arize-ai/openinference/tree/main/python/instrumentation/openinference-instrumentation-pipecat), that auto-traces a running pipeline. It's built on [OpenInference](https://github.com/Arize-ai/openinference), a set of OpenTelemetry-compatible semantic conventions for AI, so spans land in Arize AX, Phoenix, or any OpenTelemetry backend, complementing Pipecat's built-in [OpenTelemetry tracing](/api-reference/server/utilities/opentelemetry). See the [Pipecat tracing guide](https://arize.com/docs/ax/integrations/python-agent-frameworks/pipecat/pipecat-tracing) for the full integration. Pipecat conversation traces in Arize AX, with per-turn input/output, latency, and helpfulness evaluations Pipecat conversation traces in Arize AX, with per-turn input/output, latency, and helpfulness evaluations With Arize, you can: - Auto-instrument a Pipecat agent with a few lines at startup, no manual span code required - Trace every turn, with STT, LLM, TTS, and tool spans grouped by conversation - Align transcripts, tool calls, and per-stage latency in a single timeline to find bottlenecks - Run LLM-as-a-judge evaluations (hallucination, correctness, relevance, task completion) over live traffic - Track quality over time with dashboards and monitors, and alert on drift or regressions ## Connect your Pipecat agent Install the instrumentor plus the OTel SDK for your backend (`arize-otel` for Arize AX, or `arize-phoenix-otel` for Phoenix): ```bash # Arize AX pip install openinference-instrumentation-pipecat pipecat-ai arize-otel # Phoenix (open source) pip install openinference-instrumentation-pipecat pipecat-ai arize-phoenix-otel ``` Register a tracer provider and instrument Pipecat once at application startup, before you build your pipeline. Pass a `conversation_id` to `PipelineWorker` so spans are grouped per session. ```python Arize AX import os from arize.otel import register from openinference.instrumentation.pipecat import PipecatInstrumentor # Send traces to Arize AX tracer_provider = register( space_id=os.environ["ARIZE_SPACE_ID"], api_key=os.environ["ARIZE_API_KEY"], project_name="my-voice-agent", ) PipecatInstrumentor().instrument(tracer_provider=tracer_provider) # Build your pipeline as usual; spans now export to Arize AX. pipeline = Pipeline(...) worker = PipelineWorker(pipeline, conversation_id=conversation_id) ``` ```python Phoenix (open source) from phoenix.otel import register from openinference.instrumentation.pipecat import PipecatInstrumentor # Send traces to Phoenix (local or self-hosted) tracer_provider = register(project_name="my-voice-agent") PipecatInstrumentor().instrument(tracer_provider=tracer_provider) pipeline = Pipeline(...) worker = PipelineWorker(pipeline, conversation_id=conversation_id) ``` That's it. Run your agent and conversations show up in your Arize project. Because the instrumentor speaks OpenTelemetry, you can also point it at any other OTel-compatible collector by configuring the tracer provider accordingly. The instrumentor requires `pipecat-ai>=1.3` and Python 3.11+. Instrument before the pipeline is constructed so worker spans are captured from the first turn. ## What gets traced The instrumentor converts Pipecat's pipeline activity into OpenInference spans, so each conversation becomes a structured trace in Arize. As described in the [Pipecat tracing guide](https://arize.com/docs/ax/integrations/python-agent-frameworks/pipecat/pipecat-tracing), it captures: - **Conversation sessions**, grouping all turns that share a `conversation_id` - **Turn boundaries**, with each user-to-assistant exchange as a parent span - **LLM calls** with prompts, responses, token counts, and model metadata - **Speech-to-text and text-to-speech** spans with their input/output and latency - **Tool and function calls** with inputs, outputs, and duration - **End-to-end and per-stage latency**, with failures surfaced as span errors ## Online evaluation Beyond tracing, Arize runs evaluations on the traces it collects, the "evals" part of the workflow. You define an LLM-as-judge (a prompt plus an output label), and Arize scores spans automatically as traffic flows in: - Pre-built and custom judges for hallucination, correctness, relevance, and task completion - Continuous evaluation of live traffic, with scores attached back to the originating spans - Dashboards and monitors that track eval scores over time and alert on quality drift An LLM-as-judge helpfulness score on a Pipecat turn in Arize AX, with a label, score, and written explanation An LLM-as-judge helpfulness score on a Pipecat turn in Arize AX, with a label, score, and written explanation This complements Pipecat Evals: use Pipecat Evals for fast, scripted, pre-merge behavioral checks, and Arize for production-scale observability and online scoring of real conversations. ## Next steps Arize's official guide to tracing a Pipecat agent, including setup and what gets captured. Set up the hosted platform: projects, tracing, online evals, dashboards, and monitors. Self-host the open-source version for local tracing and evaluation of your Pipecat agent. The OpenTelemetry-compatible semantic conventions behind Arize's instrumentation. # Bluejay Source: https://docs.pipecat.ai/pipecat/evals/platforms/bluejay.md Simulation, observability, and evaluation platform for voice AI agents with native Pipecat integration. ## Overview [Bluejay](https://getbluejay.ai) is a simulation, observability, and evaluation platform purpose-built for voice AI agents. It provides no-code simulation testing and production call monitoring that integrate directly with Pipecat, whether you're running on Pipecat Cloud or self-hosting. With Bluejay, you can: - Run automated simulations that call your agent and evaluate its responses - Define test scenarios covering edge cases like interruptions, unexpected input, and multi-turn flows - Monitor every production call with automated quality scoring - Track evaluation metrics over time to catch regressions early ## Pipecat Cloud integration If your agent is deployed on [Pipecat Cloud](/pipecat-cloud/introduction), Bluejay offers two zero-configuration integration paths: Enter your Pipecat Cloud API key and agent name in Bluejay's dashboard. Bluejay connects directly to your agent's API to spin up simulation sessions with no code changes required. Enter your agent's phone number into Bluejay and start running simulations immediately. Bluejay calls your agent just like a real user would, testing end-to-end behavior over telephony. The telephony integration tests the full call stack, from phone network to your agent and back, making it ideal for catching issues that only surface in real call conditions. ## Self-hosted integration If you're running Pipecat on your own infrastructure, Bluejay integrates via a WebSocket connection. Point Bluejay at your agent's WebSocket endpoint and it will establish a session to run simulations against your agent directly. See the [Bluejay WebSocket integration guide](https://docs.getbluejay.ai/simulation-integrations/websockets) for setup instructions. ## Observability Simulations cover pre-deployment testing, but observability ensures your agent maintains quality with real users. Bluejay's [Evaluate API](https://docs.getbluejay.ai/api-reference/endpoint/evaluate) lets you submit any production call for automated evaluation. ```python import requests url = "https://api.getbluejay.ai/v1/evaluate" headers = {"X-API-Key": ""} payload = { "agent_id": "", "start_time_utc": "2025-03-31T18:30:00Z", "participants": [ {"role": "AGENT", "name": "Healthcare Agent Harry"}, {"role": "USER", "name": "John Doe"}, ], "recording_url": "https://s3.amazonaws.com/my-recordings/call-123.wav", } response = requests.post(url, json=payload, headers=headers) ``` Integrate the evaluate endpoint into your agent's session cleanup logic to automatically evaluate every production call without manual intervention. ## Traces Bluejay supports [tracing](https://docs.getbluejay.ai/core-concepts/traces) to monitor and observe your agent's execution flow, latency, and performance in real-time. Traces conform to the [OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/) standard, so you can use any compatible instrumentation library, including OpenInference, Langfuse, and OpenLLMetry. To send traces to Bluejay: 1. Instrument your application to export traces to Bluejay's OTLP endpoint 2. Link traces to call evaluations by including the `trace_id` in your [Evaluate API](https://docs.getbluejay.ai/api-reference/endpoint/evaluate) requests 3. View traces alongside your call evaluations in the Bluejay dashboard ### Example: OpenTelemetry setup Configure the OpenTelemetry SDK to export traces to Bluejay: ```python from opentelemetry.sdk import trace as trace_sdk from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.trace.export import SimpleSpanProcessor from opentelemetry.sdk.resources import SERVICE_NAME, Resource endpoint = "https://otlp.getbluejay.ai/v1/traces" resource = Resource.create({SERVICE_NAME: "my-pipecat-agent"}) tracer_provider = trace_sdk.TracerProvider(resource=resource) headers = { "X-API-KEY": "", } tracer_provider.add_span_processor( SimpleSpanProcessor(OTLPSpanExporter(endpoint, headers=headers)) ) ``` Once the tracer provider is configured, use it with any OpenTelemetry-compatible instrumentation. For example, to automatically trace LLM calls: ```python from openinference.instrumentation.openai import OpenAIInstrumentor OpenAIInstrumentor().instrument(tracer_provider=tracer_provider) ``` ## Next steps Full setup guides, API reference, and configuration options. Step-by-step guide for connecting Bluejay to your Pipecat agent. Pipecat's built-in behavioral testing framework. Capture conversation transcripts to use with evaluation tools. # Cekura Source: https://docs.pipecat.ai/pipecat/evals/platforms/cekura.md Simulation and production monitoring platform for Pipecat agents with flexible connection methods and metric-driven QA. ## Overview [Cekura](https://www.cekura.ai) is a simulation and production monitoring platform for voice AI agents. The broad idea is to use the same evaluation system across the full lifecycle of a Pipecat agent: simulate conversations before deployment, validate infrastructure continuously in CI, monitor real calls after deployment with metrics, dashboards, and alerts, and turn production issues back into new simulations so you can fix regressions and verify them before shipping again. With Cekura, you can: - Treat simulations and monitoring as one continuous QA workflow - Connect to Pipecat through telephony, SIP, or native Pipecat WebRTC flows - Run scenario-based tests from the dashboard or API with automated or manual session control - Track quality with predefined metrics and custom metrics tailored to your agent's workflow ## Simulations Testing in Cekura is built around scenarios, metrics, and repeatable runs. The main design choice is how you want Cekura to connect to your Pipecat agent: - Use **telephony or SIP** when you want to validate the full voice and network path - Use **native Pipecat connections** when you want faster, more direct WebRTC-based testing After you choose the connection method, the workflow is straightforward: create or select scenarios, run them from the dashboard or your CI pipeline, and review transcripts, recordings, run status, and metrics in Cekura. Telephony and SIP are useful when you want the full call path in the loop; native Pipecat connections are better when you want faster and more direct WebRTC-based simulation against your Pipecat runtime. ### Automated Pipecat connection For Pipecat Cloud agents, configure the following fields in Cekura: - **Provider**: Select `Pipecat` as the voice integration provider - **Assistant ID**: Optional Pipecat assistant identifier - **Pipecat Cloud API Key**: Used by Cekura to create and manage sessions - **Pipecat Agent Name**: Identifier used for your agent in Pipecat - **Agent Configuration (JSON)**: Optional request body sent to Pipecat's start endpoint - **Room Properties (JSON)**: Optional Daily room configuration for the WebRTC session After setup, you can run scenarios from the dashboard or API and let Cekura handle session creation, lifecycle management, execution, and cleanup automatically. This is the simplest path for repeatable Pipecat Cloud simulations. Configure Pipecat Cloud credentials once and let Cekura create and manage sessions for each test run. Provide your own room URL and token when you want explicit control over Pipecat session creation and lifecycle. ### Manual Pipecat connection If you want to control the session yourself, Cekura also supports a [manual Pipecat connection flow](https://docs.cekura.ai/documentation/integrations/pipecat/manual). In this mode, you provide the Pipecat room URL and token for each run, and Cekura joins the session you created instead of provisioning one automatically. This mode is useful when you need custom session orchestration, want to test nonstandard environments, or already manage Pipecat rooms in your own infrastructure. The automated flow is the best default for Pipecat Cloud. Use the manual flow when you need explicit control over room creation and access tokens. ### Infrastructure Suite For infrastructure-focused validation, Cekura provides an [Infrastructure Suite](https://docs.cekura.ai/documentation/guides/testing-agents/infrastructure-suite) with 18+ pre-built scenarios for latency, audio quality, interruption handling, language support, and failure cases such as packet loss and rapid-fire speech. Instead of building these tests from scratch, you can add the suite to your project from the Cekura dashboard and run the scenarios as a group. This makes it a good fit for validating the parts of your stack around the LLM itself, such as transport behavior, turn-taking, silence handling, and voice pipeline stability. Add Cekura's predefined infrastructure scenarios to your project with the appropriate metric mappings already set up. Run tagged infrastructure scenarios automatically on pull requests, pushes, or scheduled workflows. Cekura documents this suite as a good fit for CI/CD. A practical setup is to tag those scenarios as `infrastructure-suite` and execute them on every pull request: ```yaml name: Cekura Infrastructure Tests on: pull_request: types: [opened, synchronize] jobs: infrastructure-tests: runs-on: ubuntu-latest steps: - name: Run Cekura Infrastructure Suite uses: cekura-ai/cekura-github-actions@v1.0.0 with: agent_id: ${{ vars.AGENT_ID }} tags: "infrastructure-suite" api_key: ${{ secrets.CEKURA_API_KEY }} ``` This gives you a straightforward CI gate for infrastructure regressions before you ship changes to your Pipecat agent. ## Production observability Cekura can also monitor production calls after they complete. For Pipecat, the core pattern is simple: send completed calls to Cekura, use metrics and dashboards to track quality, infrastructure health, and workflow performance over time, then turn failed or interesting real calls into new simulations so the same issues are covered in your future test suite. ### Send calls to Cekura If you want observability for Pipecat traffic, send recordings and transcripts to Cekura's observability API: ```bash POST https://api.cekura.ai/observability/v1/observe/ X-CEKURA-API-KEY: Content-Type: application/json ``` The observability API accepts `transcript_type: "pipecat"` along with either an `agent` ID or `assistant_id`, plus a `voice_recording` or `voice_recording_url`. ### Example: send a Pipecat production call ```python import requests headers = { "X-CEKURA-API-KEY": "", "Content-Type": "application/json", } payload = { "call_id": "call_123", "agent": 2421, "transcript_type": "pipecat", "voice_recording_url": "https://storage.example.com/recordings/call_123.mp3", "transcript_json": [ { "role": "user", "content": "I need help with my order", "timestamp": "2026-04-03T20:15:01Z", }, { "role": "assistant", "content": "Sure, let me look that up for you.", "timestamp": "2026-04-03T20:15:03Z", }, ], "call_ended_reason": "customer-ended-call", "timestamp": "2026-04-03T20:15:45Z", } response = requests.post( "https://api.cekura.ai/observability/v1/observe/", headers=headers, json=payload, ) result = response.json() print(result) ``` Once calls are ingested, Cekura stores them for review in its observability workflow, where you can inspect call logs, transcripts, recordings, and evaluation output. ### Metrics and performance insights Cekura's metrics system can be used in both simulations and production monitoring. For reference, see the [metrics overview](https://docs.cekura.ai/documentation/key-concepts/metrics/overview), [predefined metrics](https://docs.cekura.ai/documentation/key-concepts/metrics/pre-defined-metrics), and [custom metrics](https://docs.cekura.ai/documentation/key-concepts/metrics/custom-metrics). Relevant predefined metrics for production monitoring include: - **Latency** with percentile statistics for response-time monitoring - **Infrastructure Issues** to detect failures to respond within a configured timeout - **Transcription Accuracy** for STT quality checks - **AI Interrupting User** and **Stop Time After User Interruption** for turn-taking quality - **Gibberish Detection**, **Average Pitch**, and other speech-quality metrics for audio analysis - **CSAT**, **Sentiment**, and **Dropoff Node** for agent-performance and user-experience insights Beyond the predefined set, Cekura also supports custom metrics so you can evaluate whether your agent followed the right workflow, completed key business logic, or handled domain-specific tasks correctly. That is useful for measuring the logical performance of the agent itself, not just infrastructure or audio quality. For production monitoring, audio recordings improve what Cekura can measure. Several of the speech-quality and turn-taking metrics in their docs require audio, and some require stereo recordings for the most precise results. For day-to-day operations, Cekura's observability tooling is designed around dashboards, detailed call logs, and alerting for critical issues and anomalies. In practice, that gives you voice-quality metrics, agent-performance insights, and infrastructure-failure visibility in one monitoring loop. ## Next steps Step-by-step setup for running automated Pipecat tests in Cekura. Start with Cekura's broader guide to scenario creation, execution, and testing workflows. Learn how Cekura structures production monitoring, observability, and call review workflows. Review predefined metrics and custom metrics for workflow, logic, speech, and infrastructure evaluation. # Coval Source: https://docs.pipecat.ai/pipecat/evals/platforms/coval.md AI-native simulation and evaluation platform for voice agents, trusted by QA, Engineering, Operations, AI, and Executive teams. ## Overview [Coval](https://www.coval.dev) is an AI-native simulation, evaluation, and production monitoring platform for voice agents — trusted by QA, Engineering, Operations, AI, and Executive teams to test and improve voice AI before and after it ships. Coval simulations view showing recent runs and voice metrics Coval simulations view showing recent runs and voice metrics With Coval, you can: - Spin up realistic voice simulations against your Pipecat agent and score them on voice-grade metrics - Drive evals end-to-end from Claude Code, Cursor, or any MCP-compatible client - Monitor production calls with the same metric suite you use in simulations - Catch regressions with scheduled runs, CI integration, and live dashboards ## Agent-native by design Coval is built so your AI coding assistant can run, inspect, and iterate on evals on your behalf, without you stepping out of your editor. Three surfaces work together: Drive every Coval resource from your terminal with structured JSON output. Built for scripting and CI pipelines. Expose Coval as MCP tools to Claude Code, Cursor, and any MCP-compatible client. Your agent reads state, launches runs, and surfaces failures inline. Pre-built skills your coding agent invokes to onboard a new Pipecat agent, configure metrics, and triage failing conversations. Claude Desktop calling Coval MCP tools to inspect evaluation runs In practice, your agent can create personas, launch a run against a new Pipecat build, identify which conversations failed which metrics, fetch transcripts and audio for the failures, and propose prompt fixes — all without you leaving your editor. Prefer a web UI? The same workflows are available in the [Coval dashboard](https://app.coval.dev) for teammates who'd rather click than type. ## Connect your Pipecat agent Coval supports two paths into Pipecat, depending on how you're deployed: - **Pipecat Cloud agents** — Coval's first-class [Pipecat Cloud connection](https://docs.coval.dev/concepts/agents/connections/pipecat) authenticates with your Pipecat API key, calls your agent directly, and runs simulations end-to-end. Configuration-only — no code changes required. - **Self-hosted Pipecat agents** — Expose your agent over a [WebSocket transport](https://docs.coval.dev/concepts/agents/connections/websocket) and point Coval at the endpoint. Either way, the workflow is the same once you're connected: pick a persona, point at a test set, and launch a run from the dashboard, CLI, MCP, or Agent Skills. ## Personas and scenarios Realistic voice evaluations need realistic users. Coval ships a [persona system](https://docs.coval.dev/concepts/personas/overview) with configurable voices, background environments, interruption rates, and emotional progression — so you can stress-test your agent against an impatient caller in a noisy café as easily as a calm one in a quiet room. [Test sets](https://docs.coval.dev/concepts/test-sets/overview) tell your simulated users what to do, say, and how to behave. Author them as freeform prompts for generative simulations, or as exact transcripts and pre-recorded audio for deterministic regression runs. ## Voice-native metrics Coval's [built-in metric catalog](https://docs.coval.dev/concepts/metrics/built-in-metrics) is purpose-built for voice. Evaluate conversations on: - **Latency** — LLM, STT, and TTS time-to-first-byte, plus end-to-end turn latency - **Turn-taking** — interruption rate, agent-fails-to-respond, repeated turns - **Audio quality** — natural tone detection, pause analysis, speech tempo - **STT accuracy** — Word Error Rate against a reference transcript - **Sentiment** — audio sentiment per segment and transcript-level tone - **Conversation flow** — end-reason classification (completed, hangup, max-turns, error) - **[Trace metrics](https://docs.coval.dev/concepts/metrics/built-in-metrics#trace-metrics)** — LLM, STT, and TTS span data, tool call counts, and token usage extracted from your OpenTelemetry traces - **LLM Judge & Composite Evaluation** — behavioral checks against your own criteria - **[Custom metrics](https://docs.coval.dev/concepts/metrics/prompting) and [custom trace metrics](https://docs.coval.dev/concepts/metrics/custom-trace-metrics)** — author your own LLM-judge criteria or pull numerical values straight from any OTel span Coval run results showing the voice metric breakdown for a single conversation Coval run results showing the voice metric breakdown for a single conversation ## CI and scheduled runs Wire Coval into your release process so every Pipecat change is evaluated before it ships. Coval ships a [GitHub Actions integration](https://docs.coval.dev/getting-started/github-actions-tutorial) for running eval suites on pull requests, pushes, or merges, plus [scheduled runs](https://docs.coval.dev/guides/scheduled-runs) for recurring nightly or weekly regression coverage. Both are driven by the [Coval CLI](https://docs.coval.dev/cli/overview), so the same commands work locally, in CI, and from your AI coding agent. ## Production monitoring and alerts Coval [monitors](https://docs.coval.ai/agents/overview) live Pipecat traffic with the same metric suite you use in simulations. Push completed conversations — transcript-only or with stereo audio — and Coval scores them automatically. Configure default metrics that run on every call, or conditional rules that fire additional metrics when results or metadata look off. Send [OpenTelemetry traces alongside your monitoring calls](https://docs.coval.dev/concepts/simulations/traces/opentelemetry#tracing-for-monitoring-calls) and the same trace metrics light up on real production traffic. When a monitor crosses an alert threshold, Coval can notify your team in Slack or Microsoft Teams, open a ticket in Linear for the failing conversation, or POST to any HTTP endpoint via [webhooks](https://docs.coval.ai/welcome). Failing production calls flow straight back into your test sets, so the same regression is covered the next time you ship. Stereo recordings (agent on the left channel, user on the right) unlock the full audio metric set. See [Coval's monitoring guide](https://docs.coval.ai/agents/overview) for ingestion options. ## Human review and continuous improvement Automated metrics catch most issues; the long tail needs a human in the loop. Coval ships a [human review workflow](https://docs.coval.dev/concepts/metrics/human-review/human-review) for QA and Operations teams to label conversations, correct automated scoring, and feed ground truth back into your metrics: - Manual annotation with binary, numerical, categorical, sentiment, and per-message labels - Collaborative review queues or per-reviewer assignments, with keyboard shortcuts for high-volume triage - Programmatic review project creation via the [Human Review API](https://docs.coval.dev/guides/human-review-api) - Annotated calls flow back into your metric definitions, tightening agreement scores over time QA, Engineering, Operations, AI, and Executive teams align around a shared feedback loop: automated scoring drives human correction, which improves metrics and strengthens regression coverage. ## Next steps Full setup guides, API reference, and configuration options. Connect a Pipecat Cloud agent to Coval and run end-to-end voice simulations. Wire up Coval as MCP tools in Claude Code, Cursor, or any MCP client. Label conversations, correct automated scoring, and feed ground truth back into your metrics. # Roark Eval Platform Source: https://docs.pipecat.ai/pipecat/evals/platforms/roark.md Test and monitor Pipecat voice agents with Roark: automated simulations with personas, plus observability, tracing, and metrics. ## Overview [Roark](https://roark.ai) is an end-to-end platform for testing and monitoring voice AI agents. It runs automated simulations against your [Pipecat Cloud](/pipecat-cloud/introduction) agent over the Daily transport, captures every production call, traces each turn end to end, and scores everything with the same library of metrics — so the behavior you test before shipping is the behavior you measure in production. Roark dashboard — Customer flows With Roark, you can: - Run automated simulations that call your agent over the Daily transport using lifelike **personas** and branching **flows** - Score every call — simulated or live — against **built-in metrics** plus your own LLM-graded custom metrics - **Trace** each turn (STT → LLM → TTS, tool calls, latency) via OpenTelemetry and inspect it on the call - Monitor production traffic continuously and turn any real call into a repeatable regression test ## How the integration works Roark connects to Pipecat through a single observer, then layers testing and monitoring on top: 1. **Sync and capture with the observer.** The [`pipecat-roark`](https://pypi.org/project/pipecat-roark/) package adds a `RoarkObserver` to your pipeline. It registers ("syncs") your agent to Roark the first time it runs, then streams every call's transcript, tool calls, and a stereo recording to Roark. See the [Roark analytics observer](/api-reference/server/services/analytics/roark) reference for full setup. 2. **Simulate over the Daily transport.** Once your Pipecat Cloud credentials are connected, Roark starts your agent on Pipecat Cloud, joins the Daily room as a simulated caller, drives your flow with a chosen persona, and collects the transcript, recording, traces, and metrics automatically — no code changes beyond the observer. ## Getting started Sign up at [roark.ai](https://roark.ai) and create an API key on the **API keys** page in your project. You'll pass it as `api_key` in the steps below. Add the [`pipecat-roark`](https://pypi.org/project/pipecat-roark/) package and drop `RoarkObserver` into your pipeline. The first call registers your agent with Roark automatically, so it appears under the Pipecat source filter and becomes available for simulations and reports. ```bash pip install pipecat-roark ``` ```python import os from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat_roark import RoarkObserver roark = RoarkObserver( api_key=os.environ["ROARK_API_KEY"], agent_id="support-bot-v3", agent_name="Support Bot v3", agent_prompt=SYSTEM_PROMPT, ) pipeline = Pipeline([ transport.input(), stt, context_aggregator.user(), llm, tts, transport.output(), roark.audio_processor, # after transport.output() — L=user, R=bot context_aggregator.assistant(), ]) task = PipelineTask(pipeline, params=PipelineParams(observers=[roark])) ``` See the [Roark analytics observer](/api-reference/server/services/analytics/roark) reference for the full configuration, including audio capture and the OpenTelemetry correlation used by [Tracing](#tracing). In the Roark dashboard, create a Pipecat integration and enter your **Pipecat Cloud API key** and the **Pipecat agent name** (the name of your deployed Pipecat Cloud agent — this is what Roark starts for each run, distinct from the Roark agent name you set on the observer). Roark uses these to start your agent and target it for simulation runs. Assemble a run plan — a test matrix of flows, personas, and metrics — then run it from the dashboard or on a schedule. Roark provisions each simulation automatically and handles session creation, execution, and cleanup for every run. ## Simulations When you start a run, Roark starts your agent on Pipecat Cloud, which boots it into a Daily room, and Roark joins that room as a simulated caller over the Daily transport. Because Pipecat Cloud returns the room URL and token up front, Roark connects without any SDP handshake and manages the full session lifecycle for you. A run is defined by a **run plan** — a test matrix of flows × personas × metrics. You can run 1–100 iterations per test case, run on a schedule, and compare runs over time to catch regressions. ### Personas Personas are the simulated callers that drive each conversation — configurable characters that let you test how your agent handles a realistic range of real-world callers, not just the happy path: - **Voice**: language and accent (US, British, Indian, and more), gender, and optional background noise (e.g. office) - **Speech**: pace, clarity (clear, vague, rambling), and natural disfluencies like "um" and "uh" - **Behavior**: base emotion (neutral, cheerful, confused, frustrated, skeptical, rushed), intent clarity, confirmation style, and memory reliability - **Context**: an optional backstory and custom key-value properties (account number, membership status, and so on) Start with 3–5 core personas that mirror your most common callers, then reuse them across flows. ### Flows A flow defines the conversation path a simulation follows — what the caller says, how the agent is expected to respond, and where the conversation can branch. Flows come in two modes: - **Scripted** — a visual graph of nodes and edges that lays out an exact path. Nodes cover customer turns, agent turns, silence, DTMF keypad input, and voicemail; edges let the conversation branch or loop back, so one flow can cover many outcomes. - **Improv** — a plain-language brief the simulated caller improvises from, for open-ended conversations you don't want to script turn by turn. Build a flow three ways: - **Generate from your agent prompt** — Roark drafts flows from a description of what your agent does - **Generate from a real call** — turn a production conversation into a repeatable flow - **Build it yourself** — lay out the graph (scripted) or write the brief (improv) in the visual editor Each flow can carry **variants** — edge-case variations like an interrupting, confused, or hostile caller — that reuse the flow while swapping in a different persona. ## Metrics Every call — simulated or live — is scored against the same metric library, so a fix you verify in simulation is measured the same way in production. - **Built-in metrics** are collected automatically with no configuration, spanning latency and response time, interruptions and overlap, sentiment and emotion, speech quality (DNSMOS), and compliance checks like PII handling and prompt-injection resistance. - **Custom metrics** are defined in natural language and graded by Roark Prism, Roark's model built for scoring voice conversations — for example identity verification (boolean), empathy (scale), call reason (classification), or upsell attempts (count). - **Thresholds** turn any metric into a pass/fail outcome (e.g. `Response Time < 1000ms`), applied per call or aggregated across a run. Because simulations use the same `RoarkObserver` capture path as production calls, the two are scored identically — making it straightforward to reproduce a production issue as a repeatable, metric-gated test. ## Tracing Roark ingests [OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/) traces from Pipecat's built-in tracing, so each call's full turn tree — STT, LLM, TTS, and tool calls, with latency at every span — shows up on the **Tracing** tab of the call. Configure the tracer provider **before** constructing `PipelineTask` (Pipecat grabs the provider at task-init time), and pass the **same** call ID to both `RoarkObserver(pipecat_call_id=...)` and `PipelineTask(conversation_id=...)` so Roark can link the trace to the call. ```python import os import uuid from opentelemetry import trace from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.resources import Resource from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat_roark import RoarkObserver def setup_roark_tracing() -> None: """Must run BEFORE constructing PipelineTask.""" resource = Resource.create({ "deployment.environment": os.getenv("ENVIRONMENT", "production"), # Optional: tag spans for the trace explorer. "roark.project.tag.env": os.getenv("ENVIRONMENT", "production"), }) provider = TracerProvider(resource=resource) provider.add_span_processor( BatchSpanProcessor( OTLPSpanExporter( endpoint="https://api.roark.ai/v1/traces", headers={"Authorization": f"Bearer {os.environ['ROARK_API_KEY']}"}, timeout=30, ) ) ) trace.set_tracer_provider(provider) setup_roark_tracing() # Generate ONE id and pass it to both sides so Roark can correlate. call_id = str(uuid.uuid4()) roark = RoarkObserver( api_key=os.environ["ROARK_API_KEY"], agent_id="support-bot-v3", pipecat_call_id=call_id, # appears on the Roark record as pipecatCallId ) task = PipelineTask( pipeline, params=PipelineParams(observers=[roark]), enable_tracing=True, conversation_id=call_id, # set as the conversation.id span attribute ) ``` If the two IDs don't match, the call still records — but its **Tracing** tab will be empty because Roark can't link the trace to the call. ## Observability Simulations cover pre-deployment testing; the `RoarkObserver` keeps that coverage running against real traffic. Because the observer is already in your pipeline, every production call flows to Roark automatically — transcript, tool calls, stereo recording, traces, and metrics — with no extra wiring. Watch calls live, track metrics over time on dashboards, and turn any interesting production call into a new flow so the same case is covered in your next run. See the [Roark analytics observer](/api-reference/server/services/analytics/roark) reference for capture details and configuration. ## Next steps Full setup guides, API reference, and configuration options. Step-by-step guide for connecting Roark to your Pipecat agent. Personas, flows, run plans, and how simulations execute. The `RoarkObserver` reference for capturing production calls. # Krisp VIVA Source: https://docs.pipecat.ai/pipecat/features/krisp-viva.md Integrate Krisp VIVA into Pipecat: voice isolation, noise filtering, and turn detection from the Krisp VIVA SDK. ## Overview Krisp's VIVA SDK provides four capabilities for Pipecat applications: - **Voice Isolation** — Filter out background noise and voices from the user's audio input stream, yielding clearer audio for fewer false interruptions and better transcription. - **Turn Detection** — Determine when a user has finished speaking using Krisp's streaming turn detection model, as an alternative to the [Smart Turn model](/api-reference/server/utilities/turn-detection/smart-turn-overview). - **Interruption Prediction** — Distinguish genuine user interruptions from backchannels (e.g. "uh-huh", "yeah"), preventing the bot from being interrupted by brief acknowledgements. - **Voice Activity Detection** — Detect speech in audio streams using Krisp's VAD model, supporting sample rates from 8kHz to 48kHz. You can use any combination of these features together. API reference for voice isolation API reference for turn detection API reference for interruption prediction API reference for voice activity detection Complete example with Krisp features Get the Krisp SDK and API key ## Prerequisites To complete this setup, you will need access to a Krisp developers account, where you can download the Python SDK, models, and generate an API key. Get started on the [Krisp developers website](https://krisp.ai/developers). ## Setup ### Download the Python SDK and Models 1. Log in to the [Krisp developer portal](https://sdk.krisp.ai/) 2. Navigate to the `Server SDK Version` Tab 3. Find the latest version of the Python SDK: - Download the SDK - Download the Voice Isolation models (for voice isolation) - Download the Turn Detection models (for turn detection) ### Install the Python wheel file 1. First, unzip the SDK files you downloaded in the previous step. In the unzipped folder, you will find a `dist` folder containing the Python wheel file you will need to install. 2. Install the Python wheel file that corresponds to your platform. For example, a macOS ARM64 platform running Python 3.12 would install the following: ```bash uv pip install /PATH_TO_DOWNLOADED_SDK/krisp-viva-uar-python-sdk-1.8.0/dist/krisp_audio-1.8.0-cp312-cp312-macosx_12_0_arm64.whl ``` ### Generate an API key 1. In the [Krisp developer portal](https://sdk.krisp.ai/), generate an API key for your application. The `KRISP_VIVA_API_KEY` is required for Krisp SDK v1.6.1 and later. For older SDK versions, this is not required. ### Set up environment variables 1. Unzip the models you downloaded in the first step. 2. For voice isolation, choose a model: - `krisp-viva-pro`: Mobile, Desktop, Browser (WebRTC, up to 32kHz) - `krisp-viva-tel`: Telephony, Cellular, Landline, Mobile, Desktop, Browser (up to 16kHz) Note: the full model name will be in the format of `krisp-viva-tel-v2.kef`. 3. In your .env file, add the environment variables for the features you're using: ```bash # Krisp SDK API key (required for SDK v1.6.1+) KRISP_VIVA_API_KEY=your_api_key_here # Voice isolation model path KRISP_VIVA_FILTER_MODEL_PATH=/PATH_TO_UNZIPPED_MODELS/krisp-viva-vi-tel-v2.kef # Turn detection model path KRISP_VIVA_TURN_MODEL_PATH=/PATH_TO_UNZIPPED_MODELS/krisp-viva-tp-v3.kef # Interruption prediction model path KRISP_VIVA_IP_MODEL_PATH=/PATH_TO_UNZIPPED_MODELS/krisp-viva-ip-v1.kef # Voice activity detection model path (optional) KRISP_VIVA_VAD_MODEL_PATH=/PATH_TO_UNZIPPED_MODELS/krisp-viva-vad-v2.kef ``` Each feature uses a **different model**. Set `KRISP_VIVA_FILTER_MODEL_PATH` for voice isolation, `KRISP_VIVA_TURN_MODEL_PATH` for turn detection, `KRISP_VIVA_IP_MODEL_PATH` for interruption prediction, and `KRISP_VIVA_VAD_MODEL_PATH` for voice activity detection. ## Test the integration You're ready to test the integration! Try running the [Krisp VIVA foundation example](https://github.com/pipecat-ai/pipecat/blob/main/examples/voice/voice-krisp-viva.py), which demonstrates both voice isolation and turn detection together. Learn how to [run foundational examples](https://github.com/pipecat-ai/pipecat/blob/main/examples/README.md) in Pipecat. ## Voice Isolation `KrispVivaFilter` isolates the user's voice by filtering out background noise and other voices in real-time audio streams. Add it to any transport via the `audio_in_filter` parameter. ```python from pipecat.audio.filters.krisp_viva_filter import KrispVivaFilter from pipecat.transports.base_transport import TransportParams transport = SmallWebRTCTransport( webrtc_connection=webrtc_connection, params=TransportParams( audio_in_enabled=True, audio_in_filter=KrispVivaFilter(), # Enable Krisp voice isolation audio_out_enabled=True, ), ) ``` See the [KrispVivaFilter reference](/api-reference/server/services/audio-filters/krisp-viva-filter) for configuration options. ## Turn Detection `KrispVivaTurn` uses Krisp's streaming turn detection model to determine when a user has finished speaking. Unlike the [Smart Turn model](/api-reference/server/utilities/turn-detection/smart-turn-overview) which analyzes audio in batches, `KrispVivaTurn` processes each audio frame in real time. Configure it as a user turn stop strategy: ```python from pipecat.audio.turn.krisp_viva_turn import KrispVivaTurn from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.processors.aggregators.llm_response_universal import ( LLMContextAggregatorPair, LLMUserAggregatorParams, ) from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy from pipecat.turns.user_turn_strategies import UserTurnStrategies user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, user_params=LLMUserAggregatorParams( user_turn_strategies=UserTurnStrategies( stop=[TurnAnalyzerUserTurnStopStrategy( turn_analyzer=KrispVivaTurn() )] ), vad_analyzer=SileroVADAnalyzer(), ), ) ``` See the [KrispVivaTurn reference](/api-reference/server/utilities/turn-detection/krisp-viva-turn) for configuration options. ## Interruption Prediction `KrispVivaIPUserTurnStartStrategy` uses Krisp's Interruption Prediction (IP) model to distinguish genuine user interruptions from backchannels. When VAD detects user speech, the IP model analyzes the audio and outputs a probability indicating whether the speech is a real interruption or a brief acknowledgement (e.g., "uh-huh", "yeah"). This prevents the bot from being interrupted unnecessarily by short utterances. Configure it as a user turn start strategy: ```python from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.processors.aggregators.llm_response_universal import ( LLMContextAggregatorPair, LLMUserAggregatorParams, ) from pipecat.turns.user_start import ( KrispVivaIPUserTurnStartStrategy, TranscriptionUserTurnStartStrategy, ) from pipecat.turns.user_turn_strategies import UserTurnStrategies user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, user_params=LLMUserAggregatorParams( user_turn_strategies=UserTurnStrategies( start=[ KrispVivaIPUserTurnStartStrategy(threshold=0.5), TranscriptionUserTurnStartStrategy(), # Fallback ], ), vad_analyzer=SileroVADAnalyzer(), ), ) ``` See the [KrispVivaIPUserTurnStartStrategy reference](/api-reference/server/utilities/turn-management/user-turn-strategies#krispvivaipuserturnstartstrategy) for configuration options. ## Voice Activity Detection `KrispVivaVadAnalyzer` detects speech in audio streams using Krisp's VAD model. It supports sample rates from 8kHz to 48kHz, making it suitable for a wide range of applications including telephony and high-quality audio. Configure it as a VAD analyzer: ```python from pipecat.audio.vad.krisp_viva_vad import KrispVivaVadAnalyzer from pipecat.audio.vad.vad_analyzer import VADParams from pipecat.processors.aggregators.llm_response_universal import ( LLMContextAggregatorPair, LLMUserAggregatorParams, ) user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, user_params=LLMUserAggregatorParams( vad_analyzer=KrispVivaVadAnalyzer(params=VADParams(stop_secs=0.2)), ), ) ``` See the [KrispVivaVadAnalyzer reference](/api-reference/server/services/vad/krisp-viva-vad-analyzer) for configuration options. # WhatsApp Business Calling API Source: https://docs.pipecat.ai/pipecat/features/whatsapp.md Receive WhatsApp voice calls in a Pipecat application using the WhatsApp Business Calling API and WhatsAppTransport. ## Overview This guide explains how to integrate WhatsApp Business voice calling into your Pipecat application using the **WhatsApp Cloud API** and **SmallWebRTC**. Once configured, your Pipecat bot will be able to receive and handle real-time voice calls directly from WhatsApp users. ## Prerequisites ### WhatsApp Business Setup Before proceeding, ensure you have completed the following steps: 1. **Facebook Account** — Create one at [facebook.com](https://facebook.com) 2. **Facebook Developer Account** — Register at [developers.facebook.com](https://developers.facebook.com) 3. **WhatsApp Business App** — Create a new app via [Meta for Developers](https://developers.facebook.com/apps) 4. **Phone Number** — Add and verify a WhatsApp Business phone number 5. **Business Verification** — Required for production environments 6. **Webhook Configuration** — You’ll need a public webhook URL to receive call events > **Note**: Voice calling is only available for WhatsApp Business API numbers with calling capability enabled. For reference: - [WhatsApp Cloud API Getting Started](https://developers.facebook.com/docs/whatsapp/cloud-api/get-started/) - [Voice Calling API Documentation](https://developers.facebook.com/docs/whatsapp/cloud-api/calling/) - [Webhook Setup](https://developers.facebook.com/docs/whatsapp/webhooks/) ## WhatsApp Configuration > **Note**: For these settings you must be logged into your **Meta Developer Console** and inside your **App**. ### 1. Enable Voice Calls To enable voice calls for your number: - Go to **WhatsApp** → **Configuration → Phone Numbers → Manage Phone Numbers** - Select your WhatsApp number - Under the **Calls** tab, enable **”Allow voice calls”** - Save the configuration > For development, Meta provides a **free test number** valid for **90 days**. **Test number showing “Unverified”? The Calls tab won't appear.** If your phone number status is **Unverified** in WhatsApp Manager, the **Calls** tab will not be available. You must register the number first: ```bash curl -X POST “https://graph.facebook.com/v22.0//register” \ -H “Authorization: Bearer ” \ -H “Content-Type: application/json” \ -d '{“messaging_product”: “whatsapp”, “pin”: “000000”}' ``` - `` — found under **WhatsApp → API Setup** - `` — from [Graph API Explorer](https://developers.facebook.com/tools/explorer) or **WhatsApp → API Setup** - `pin` — any 6-digit value (sets the 2FA PIN for the number) Once done, the phone number status in [WhatsApp Manager → Phone Numbers](https://business.facebook.com/latest/whatsapp_manager/phone_numbers) should change to **Connected** and the **Calls** tab will appear. Generate a new access token and update your `WHATSAPP_TOKEN`. ### 2. Configure Webhook Set up your webhook to receive **WhatsApp call events**. 1. In the **Meta Developer Console**, go to **WhatsApp → Configuration → Webhooks** 2. **Set your callback URL** — this is where WhatsApp will send incoming call notifications. #### For local development - Make sure your local Pipecat server is running (e.g. `http://localhost:7860`). - Use **ngrok** (or a similar tunneling tool) to expose your local server: ```bash ngrok http --domain=YOUR_NGROK_DOMAIN http://localhost:7860 ``` - Copy the generated HTTPS URL and set your webhook to: `https://YOUR_NGROK_DOMAIN/whatsapp` - ✅ **Important:** Always include the `/whatsapp` path at the end of your webhook URL. #### For Pipecat Cloud - Ensure your Pipecat agent is already **deployed and running**. - Use the following format for your webhook URL: ``` https://api.pipecat.daily.co/v1/public/webhooks/$ORGANIZATION_ID/$AGENT_NAME/whatsapp ``` - ✅ **Important:** Always include the `/whatsapp` path at the end of your webhook URL. 3. **Enter your Verify Token** - For **local development**, set the environment variable `WHATSAPP_WEBHOOK_VERIFICATION_TOKEN` to match the verify token you enter in Meta. - For **Pipecat Cloud**, set the verify token to your **public API key**. 4. Click **Verify and Save** to confirm your webhook setup. 5. Under **Webhook Fields**, enable the following event type: - `calls` → _(required to receive voice call events)_ ### 3. Configure Access Token 1. Go to **WhatsApp → API Setup** 2. Click **Generate Access Token** 3. Use the generated token as your `WHATSAPP_TOKEN` environment variable (or secret) 4. Note your **Phone Number ID** and configure it as `PHONE_NUMBER_ID` > This will create a temporary access token, which usually expires in less than 02 hours. > To create a permanent access token, you can follow [this document](https://developers.facebook.com/blog/post/2022/12/05/auth-tokens/) from Meta. ### 4. Configure App secret 1. Go to **App settings → Basic** 2. Click to show your **App secret** 3. Use the app secret as your `WHATSAPP_APP_SECRET` environment variable (or secret) > This secret is used to verify that the webhooks you receive are actually from WhatsApp. ## Pipecat Integration WhatsApp works with any bot using `SmallWebRTCTransport`. All you need to do is configure WhatsApp as described above and provide the correct environment variables when starting your Pipecat bot. Make sure your `.env` file includes: ```bash WHATSAPP_TOKEN= WHATSAPP_PHONE_NUMBER_ID= WHATSAPP_APP_SECRET= # Only needed for local development WHATSAPP_WEBHOOK_VERIFICATION_TOKEN=pk_** ``` > If you are using Pipecat Cloud, you must configure these values as secrets. ### Example ```python transport = SmallWebRTCTransport( webrtc_connection=webrtc_connection, params=TransportParams( audio_in_enabled=True, audio_out_enabled=True, ), ) ``` ## Testing the Integration 1. From the WhatsApp app, call your registered WhatsApp Business number. 2. The call will be routed to your Pipecat application. 3. Your bot should automatically answer and begin the conversation. **Meta test numbers and country restrictions** Meta's sandbox test numbers are US-based. Due to a [recent Meta policy change](https://developers.facebook.com/community/threads/2215489342276947/), these numbers are **restricted from messaging or calling users in Brazil and several other countries** (error code 130497). If you're testing from one of those countries and calls never arrive, this is the likely cause. Use a production WhatsApp Business number from a supported country, or test from a phone number in a country not subject to the restriction. ## Full demo Pipecat Cloud WhatsApp Demo ## Troubleshooting Common issues and their solutions: 1. **Expired WhatsApp Token** ```bash Error validating access token ``` - Generate a new access token under **WhatsApp → API Setup**, then update your `WHATSAPP_TOKEN` and redeploy your agent. 2. **Webhook Verification Failed** - Ensure the verify token in Meta Developer Console matches your environment variable. - If using Pipecat Cloud, it must be your public API key. - Confirm that your webhook URL is publicly accessible. 3. **Bot Not Answering Calls** - Verify your phone number has calling enabled. - Check that your environment variables are correctly set. - Ensure Pipecat is running and reachable. 4. **Calls never arrive (error 130497 — country restriction)** Meta's sandbox test numbers are US-based and restricted from contacting users in Brazil and other countries due to a [recent policy change](https://developers.facebook.com/community/threads/2215489342276947/). Use a production WhatsApp Business number from a supported country, or test from a phone number in a country not subject to the restriction. ## Notes - Voice calling requires WhatsApp Business API access. - Test phone numbers are valid for 90 days in development mode. - Production deployment requires verified business credentials. ## References - [WhatsApp Cloud API – Getting Started](https://developers.facebook.com/docs/whatsapp/cloud-api/get-started/) - [Voice Calling API Docs](https://developers.facebook.com/docs/whatsapp/cloud-api/calling/) - [Webhook Configuration](https://developers.facebook.com/docs/whatsapp/webhooks/) - [SDP Overview & Examples](https://developers.facebook.com/docs/whatsapp/cloud-api/calling/reference#sdp-overview-and-sample-sdp-structures) # Building with Gemini Live Source: https://docs.pipecat.ai/pipecat/features/gemini-live.md Create real-time voice AI agents using Google's Gemini Live API and Pipecat Gemini Live is Google's speech-to-speech API that enables natural, real-time voice conversations with AI. With Pipecat, you can build production-ready voice agents that leverage Gemini Live for telephony, web, and mobile applications. Gemini Live service documentation Scaffold and deploy projects ## Capabilities Pipecat's Gemini Live integration supports multiple modalities and deployment targets: Real-time speech-to-speech conversations with natural turn-taking and voice activity detection Process video and screenshare alongside audio for multimodal interactions Build phone-based voice agents with Twilio WebSocket integration Function calling support for external integrations and dynamic responses ### Architecture Pipecat manages connections between your client and Gemini Live: ![Gemini Live Architecture](/images/gemini-live-architecture.png) The Pipecat server handles media streaming with clients via WebRTC (web/mobile) or WebSockets (telephony), while maintaining a persistent connection to Gemini Live for real-time AI processing. ## Quick Start The fastest way to start building is with the Pipecat CLI: ```bash # Install the CLI uv tool install "pipecat-ai[cli]" # Start a new project pipecat init ``` When `init` asks how you want to build, choose **Scaffold a runnable bot now**. The wizard will guide you through selecting: - **Bot type**: Gemini Live (speech-to-speech) - **Transport**: Daily WebRTC, Twilio, or others - **Deployment target**: Local development or Pipecat Cloud All CLI commands can use either `pipecat` or the shorter `pc` alias. ## Starter Projects These complete examples demonstrate Gemini Live in production scenarios. Each includes local development setup and Pipecat Cloud deployment configuration. ### Phone Bot (Twilio) A telephone-based voice agent using Gemini Live with Twilio WebSockets. The demo plays "Two Truths and a Lie" to showcase natural conversation flow. Build a production phone agent with Twilio integration **Try it now**: Call **1-970-LIVE-API** (1-970-548-3274) to talk to a live demo. **What you'll learn**: - Twilio WebSocket transport configuration - Google STT/TTS integration alongside Gemini Live - TwiML setup for incoming calls - Pipecat Cloud deployment with telephony ### Web Bot (Vision) A browser-based agent with screensharing and vision capabilities, built with the Pipecat Voice UI Kit and Daily WebRTC transport. Build a web agent with vision and screensharing **What you'll learn**: - Daily WebRTC transport for web clients - Vision/screenshare processing with Gemini Live - Next.js client with Voice UI Kit components - Resizable panels and event logging ## Deployment Both starter projects include configuration for [Pipecat Cloud](/pipecat-cloud/introduction), which handles scaling, monitoring, and global deployment. ```bash # Authenticate with Pipecat Cloud pipecat cloud auth login # Deploy your agent pipecat cloud deploy ``` Each starter includes a `pcc-deploy.toml` file with sensible defaults for agent configuration and scaling. Learn more about deploying to production ## Next Steps Add external integrations and dynamic responses Build custom web interfaces Deep dive into phone integrations Understand Pipecat pipelines and processors # Building With OpenAI Audio Models and APIs Source: https://docs.pipecat.ai/pipecat/features/openai-audio-models-and-apis.md Build voice agents with OpenAI audio models in Pipecat: STT, TTS, and the Realtime API, and when to use each. This guide provides an overview of the audio capabilities OpenAI offers via their APIs. We'll also link to Pipecat sample code. ## Two Ways To Build Voice-to-voice You can build voice-to-voice applications in two ways: 1. The cascaded models approach, using separate models for transcription, the LLM, and voice generation. ![OpenAI Cascaded Pipeline](/images/openai-cascade.jpg) A cascaded pipeline looks like this, in Pipecat code. Here's a [single-file example that uses a cascaded pipeline](https://github.com/pipecat-ai/pipecat/blob/main/examples/voice/voice-openai.py). (See below for an overview of Pipecat core concepts.) ```python pipeline = Pipeline( [ transport.input(), speech_to_text, context_aggregator.user(), llm, text_to_speech, context_aggregator.assistant(), transport.output(), ] ) ``` 2. Using a single, speech-to-speech model. This is conceptually much simpler. Though note that most applications also need to implement things like function calling, retrieval-augmented search, context management, and integration with existing systems. So the core pipeline is only part of an app's complexity. ![OpenAI S2S Pipeline](/images/openai-s2s.jpg) Here's a speech-to-speech pipeline in Pipecat code. And here's a [single-file example that uses the OpenAI Realtime API](https://github.com/pipecat-ai/pipecat/blob/main/examples/realtime/realtime-openai.py). ```python pipeline = Pipeline( [ transport.input(), context_aggregator.user(), speech_to_speech_llm, context_aggregator.assistant(), transport.output(), ] ) ``` Which approach should you choose? - The cascaded models approach is preferable if you are implementing a complex workflow and need the best possible instruction following performance and function calling reliability. The `gpt-4o` model operating in text-to-text mode has the strongest instruction following and function calling performance. - The speech-to-speech approach offers better audio understanding and human-like voice output. If your application is primarily free-form, open-ended conversation, these attributes might be more important than instruction following and function calling performance. Note also that `gpt-4o-audio-preview` and the OpenAI Realtime API are currently beta products. ## OpenAI Audio Models and APIs ### Transcription API - Models: `gpt-realtime-whisper` (default for streaming), `gpt-4o-transcribe`, `gpt-4o-mini-transcribe` - Pipecat services: `OpenAISTTService`, `OpenAIRealtimeSTTService` ([reference docs](/api-reference/server/services/stt/openai)) - OpenAI endpoint: `/v1/audio/transcriptions` ([docs](https://platform.openai.com/docs/api-reference/audio/createTranscription)) ### Chat Completions API - Models: `gpt-4o`, `gpt-4o-mini`, `gpt-4o-audio-preview` - Pipecat service: `OpenAILLMService` ([reference docs](/api-reference/server/services/llm/openai)) - OpenAI endpoint: `/v1/chat/completions` ([docs](https://platform.openai.com/docs/api-reference/chat)) ### Realtime API - Models: `gpt-realtime-2`, `gpt-realtime-1.5`, `gpt-realtime` - Pipecat service: `OpenAIRealtimeLLMService` ([reference docs](/api-reference/server/services/s2s/openai)) - OpenAI docs ([overview](https://platform.openai.com/docs/guides/realtime)) ### Speech API - Models: `gpt-4o-mini-tts` - Pipecat service: `OpenAITTSService` ([reference docs](/api-reference/server/services/tts/openai)) - OpenAI endpoint: `/v1/audio/speech` ([docs](https://platform.openai.com/docs/api-reference/audio/createSpeech)) ## Sample code and starter kits _If you have a code example or starter kit you would like this doc to link to, please let us know. We can add examples that help people get started with the OpenAI audio models and APIs._ ### Single-file examples A complete implementation demonstrating the cascaded approach with OpenAI services A speech-to-speech implementation using OpenAI's Realtime API ### OpenAI + Twilio + Pipecat Cloud [This starter kit](https://github.com/daily-co/pcc-openai-twilio/) is a complete telephone voice agent that can talk about the NCAA March Madness basketball tournaments and look up realtime game information using function calls. ![OpenAI Twilio](/images/openai-twilio.png) The starter kit includes two bot configurations: cascaded model and speech-to-speech. The code can be packaged for deployment to Pipecat Cloud, a commercial platform for Pipecat agent hosting. # Pipecat Telephony Overview Source: https://docs.pipecat.ai/pipecat/telephony/overview.md Give Pipecat bots phone capabilities: dial-in and dial-out over PSTN and SIP with Daily, Twilio, Telnyx, Plivo, and Exotel. ## Introduction You can dial-in to your Pipecat bots, and have them dial-out too, across both PSTN and SIP. The technical implementation will depend on your chosen transport and phone number vendor; each will likely have their own methods and events to consider. ## Key Terms **PSTN (Public Switched Telephone Network)**: The traditional phone network consisting of physical phone lines, cables, and transmission links. PSTN operates on a one-user-per-line basis. **SIP (Session Initiation Protocol)**: A signaling protocol used for voice and video calls over IP networks. SIP can handle multiple users per line and enables advanced call control features like transfers, forwarding, and multi-party calls. ## Telephony Connection Options ### WebSocket Connections **Best for:** Simple telephony workflows and quick prototypes - **How it works:** Real-time audio streaming over WebSocket connections - **Call control:** Basic, managed by the telephony provider - **Supported providers:** Twilio, Telnyx, Plivo, Exotel - **Limitations:** No advanced call center features like transfers or reconnects **When to use:** - Simple inbound/outbound calling - Integration with existing Twilio Studio or Flex workflows - Quick setup with minimal configuration ### WebRTC Connections **Best for:** When connecting web clients to telephony systems or for more complex transfer scenarios - **How it works:** Peer-to-peer real-time communication optimized for varying network conditions - **Call control:** Advanced, includes track-level control - **Supported providers:** Daily (with PSTN integration) - **Benefits:** Designed for scale, handles network variations gracefully **When to use:** - When connecting web or mobile clients to telephony systems - Applications requiring high-quality audio/video - Users on devices with varying network conditions ### SIP Connections **Best for:** Enterprise telephony and advanced call control scenarios - **How it works:** Industry-standard protocol for voice/video calls over IP - **Call control:** Full control, including transfers, forwarding, multi-party, etc. - **Integration:** Works with legacy call centers and telephony systems - **Vendor flexibility:** Supports multiple telephony vendors without platform-specific code **When to use:** - Multi-agent or multi-party calls - Integration with legacy call centers - Advanced features: warm transfers, agent assist, call forwarding - Enterprise telephony requirements **Multiple providers:** You can configure your Pipecat bots to handle multiple vendors simultaneously. For example, use both Daily and Twilio as phone number providers concurrently. ## Telephony Provider Options ### Daily PSTN (WebRTC) Direct PSTN connectivity through Daily's platform with WebRTC transport. - **Setup:** Purchase phone numbers directly through Daily - **Transport:** WebRTC for optimal audio quality - **Use case:** A range of use cases, from simple to complex including cold and warm transfers [**Daily PSTN Guide →**](./daily-pstn) ### Daily + SIP Integration Combine Daily's WebRTC transport with SIP-based telephony providers. - **Setup:** Use existing Twilio numbers with Daily transport - **Transport:** WebRTC with SIP forwarding - **Use case:** Leverage existing Twilio workflows with Daily's transport benefits [**Daily + Twilio SIP Guide →**](./twilio-daily-sip) ### WebSocket Providers Direct integration with telephony providers using WebSocket connections. Media Streams integration with Twilio's telephony platform {" "} Real-time media streaming with Telnyx services {" "} Voice streaming API integration with Plivo WebSocket integration with Exotel telephony platform ## Next Steps Choose the telephony option that best fits your use case and follow the corresponding guide. Each option provides complete setup instructions and example code to get you started quickly. # Daily Phone Numbers Source: https://docs.pipecat.ai/pipecat/telephony/daily-phone-numbers.md Purchase and manage phone numbers for Daily's PSTN services to give your Pipecat bots dial-in and dial-out. Use Daily's [REST API](https://docs.daily.co/reference/rest-api/phone-numbers) to find, buy, list, and release phone numbers programmatically. You can also purchase phone numbers through the Pipecat Cloud Dashboard by going to `Settings` > `Telephony` and following the UI. ### Search for available phone numbers We offer several [search filters](https://docs.daily.co/reference/rest-api/phone-numbers/list-available-numbers), such as, `region`, `city`, `areacode`, etc that can help narrow down the search space for a phone number. The response contains phone numbers that meet the chosen criteria. For simplicty, in the below examples, we will use `+19499870006` as an example `phone_number`. ```bash Request curl --request GET \ --url 'https://api.daily.co/v1/list-available-numbers?region=CA' \ --header 'Authorization: Bearer $DAILY_API_KEY' ``` ```js Success { "total_count": 1, "data": [{ "number": "+19499800001", "region": "CA" }] } ``` ### Buy a phone number To purchase the chosen number, make a POST request to the [buy-phone-number](https://docs.daily.co/reference/rest-api/phone-numbers/buy-phone-number) and pass the chosen phone number in the data field. The response returns the phone number and a unique id. The unique id is needed when releasing the chosen phone number, for example. ```bash Request curl --request POST \ --url 'https://api.daily.co/v1/buy-phone-number' \ --header 'Authorization: Bearer $DAILY_API_KEY' \ --header 'Content-Type: application/json' \ --data '{ "number": "+19499800001" }' ``` ```js Success { "id": "5ccc01ca-f448-4f17-b7b4-cf354b7b8256", "number": "+19499800001" } ``` Use the id to select the desired phone number for dial-out. Pass it as `callerId` in the `start_dialout` API or include it in the `dialout_settings` JSON. ### Release a phone number To release a phone number associated with your account. Successfully deleting the number takes the phone number out of operation. For compliance reasons, a phone number cannot be released until 14 days after purchase. A successful release operation cannot be undone, and the phone number will be taken out of operation. ```bash Request curl -H "Content-Type: application/json" \ -H "Authorization: Bearer $DAILY_API_KEY" \ -XDELETE \ https://api.daily.co/v1/release-phone-number/5ccc01ca-f448-4f17-b7b4-cf354b7b8256 ``` ```js Success { "deleted": true, "id": "0cb313e1-211f-4be0-833d-8c7305b19902", } ``` ```js Failure { ("Failed to release new number, try again in 14 days"); } ``` ### List all purchased phone numbers You can also list all your purchased phone numbers using the following [API](https://docs.daily.co/reference/rest-api/phone-numbers/purchased-phone-numbers) ```bash Request curl --request GET \ --url 'https://api.daily.co/v1/purchased-phone-numbers' \ --header 'Authorization: Bearer $DAILY_API_KEY' ``` ```js Success { "total_count": 1, "data": [{ "name": null, "id": "5ccc01ca-f448-4f17-b7b4-cf354b7b8256", "created_date": "Mon, 19 Aug 2024 18:50:34 +0000", "type": "purchased_phone_number", "number": "+19499800001", "status": "verified", "verified": true }] } ``` ### CNAM Registration Please contact `help@daily.co` for adding Caller Name (CNAM) to the purchased phone numbers. Daily will need your Name or Company Name to register with the United States CNAM Registry. Once registered, the Caller ID will display the appropriate name on the calling party's device. We need the following information: - All phone numbers and the corresponding CNAM entry. A CNAM entry can only be 14 letter long, including spaces. - Company documents - An employee's ID/driver's license It can take up to 48 hours for the CNAM to reflect on the national caller ID name database and can take longer to update for major carriers, who check the national database periodically. # Daily PSTN Source: https://docs.pipecat.ai/pipecat/telephony/daily-pstn.md Complete guide to Daily's PSTN capabilities including dial-in, dial-out, and call transfers ## Things you'll need - An active [Daily](https://www.daily.co) developer account with API key - At least one phone number purchased through Daily (covered below) - For local development: ngrok to expose your bot to the Internet Complete dial-in implementation with the Pipecat development runner Outbound calling with Daily PSTN integration Cold transfer implementation for advanced call routing ## Phone Number Management Before setting up dial-in or dial-out, you'll need to purchase phone numbers through Daily. Complete guide to searching, purchasing, and managing phone numbers with Daily's REST API ## Dial-in Dial-in allows users to call your phone number and connect directly to your Pipecat bot. ### How It Works Here's the sequence of events when someone calls your Daily phone number: 1. **Daily receives an incoming call** to your phone number 2. **Daily calls your webhook endpoint** 3. **The webhook creates a Daily room** with SIP configuration 4. **The webhook starts your bot** with the room details and caller information 5. **The caller is put on hold** with music 6. **The bot joins the Daily room** and signals readiness 7. **Daily forwards the call** to the Daily room 8. **The caller and bot are connected** for the conversation ### Local Development The Pipecat development runner provides built-in webhook handling for Daily PSTN dial-in, eliminating the need for a separate webhook server. #### 1. Run your bot with dial-in support ```bash uv run bot.py -t daily --dialin ``` This starts a FastAPI server on port 7860 with a `/daily-dialin-webhook` endpoint. `--dialin` no longer requires `-t daily`; passing `-t daily` simply restricts the runner to the Daily transport. #### 2. Expose your bot to the internet ```bash ngrok http 7860 ``` Copy the ngrok URL (e.g., `https://abc123.ngrok.io`). Use `ngrok http 7860 --subdomain your-subdomain` for a reusable URL. #### 3. Configure your Daily phone number Set your phone number's `room_creation_api` webhook to: ``` https://your-ngrok-url.ngrok.io/daily-dialin-webhook ``` Instructions for configuring the webhook URL for your phone number ```shell Create pinless dial-in config curl --location 'https://api.daily.co/v1' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer YOUR_DAILY_API_KEY' \ --data '{ "properties": { "pinless_dialin": [ { "phone_number": "DAILY_PROVISIONED_NUMBER_HERE", "room_creation_api": "https://your-ngrok-url.ngrok.io/daily-dialin-webhook" } ] } }' ``` Ensure your `DAILY_API_KEY` has the phone number associated with it. This is required for the `pinlessCallUpdate` API call that connects the caller to the room. ### Configure your Pipecat bot for dial-in The bot receives dial-in information through `RunnerArguments` containing: - `room_url`: Daily room URL for the call - `token`: Daily room token for authentication - `body`: Contains `DailyDialinRequest` with call details **Bot Entry Point**: ```python bot.py from pipecat.runner.types import DailyDialinRequest, RunnerArguments from pipecat.transports.daily import DailyTransport, DailyParams, DailyDialinSettings async def bot(runner_args: RunnerArguments): # Parse dial-in request from runner request = DailyDialinRequest.model_validate(runner_args.body) # Configure dial-in settings with webhook data daily_dialin_settings = DailyDialinSettings( call_id=request.dialin_settings.call_id, call_domain=request.dialin_settings.call_domain ) # Create transport with dial-in configuration transport = DailyTransport( runner_args.room_url, runner_args.token, "Voice Bot", DailyParams( api_key=request.daily_api_key, api_url=request.daily_api_url, dialin_settings=daily_dialin_settings, audio_in_enabled=True, audio_out_enabled=True, ) ) # Your bot setup and pipeline creation here... if __name__ == "__main__": from pipecat.runner.run import main main() ``` See the full bot.py with argument handling, transport setup, and pipeline configuration ### Customize Your Bot with Caller Information Use the caller's phone number to personalize the conversation: ```python async def bot(runner_args: RunnerArguments): # Parse dial-in request request = DailyDialinRequest.model_validate(runner_args.body) # Get caller's phone number caller_phone = request.dialin_settings.From # Look up customer information from your database customer = await get_customer_by_phone(caller_phone) # Customize the system prompt messages = [ { "role": "developer", "content": f"You are a helpful assistant for {customer.name}. " f"Their account status is {customer.status}. " "Keep responses concise and conversational." } ] # Use the customized context in your bot... ``` ### Run the Example See the full README with step-by-step setup, environment variables, and troubleshooting tips ## Dial-out Dial-out allows your bot to initiate calls to phone numbers. Unlike dial-in, your bot starts the call rather than waiting for incoming calls. You must contact Daily to enable dial-out for your account. Submit a support request [here](https://docs.google.com/forms/d/1EHy2wsX20a2HjljAsJkyPqvATIM4WCvm7JzYKypGYis/edit). ### How It Works Here's the sequence of events for dial-out calls: 1. **Your application triggers a dial-out** (via API call or user action) 2. **Server creates a Daily room** with dial-out capabilities enabled 3. **Bot joins the Daily room** and sets up the pipeline 4. **Bot initiates the dial-out call** to the target phone number 5. **Daily connects the call** to the phone number 6. **The recipient answers** and is connected to your bot 7. **The bot handles the conversation** with the called party ### Set up your server for dial-out The dial-out server is simpler than dial-in since you're initiating calls rather than receiving webhooks. Your server needs to: 1. **Create Daily rooms** with dial-out enabled 2. **Start bot processes** with the target phone number 3. **Handle API requests** to trigger outbound calls See the full FastAPI server code with room creation and dial-out triggering ### Configure your Pipecat bot for dial-out The dial-out bot receives the target phone number and creates a transport without dial-in settings: **Bot Entry Point**: The `bot()` method receives `RunnerArguments` containing: - `room_url`: Daily room URL for the call - `token`: Daily room token for authentication - `phone_number`: Target phone number to call **Transport Creation**: These arguments are used to configure the `DailyTransport`: ```python bot.py from pipecat.transports.daily import DailyTransport, DailyParams async def bot(args: RunnerArguments): # Create transport for dial-out (no dial-in settings needed) transport = DailyTransport( args.room_url, args.token, "Voice Bot", DailyParams( api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"), api_key=os.getenv("DAILY_API_KEY"), audio_in_enabled=True, audio_out_enabled=True, video_out_enabled=False, transcription_enabled=True, ) ) # Start the dial-out call await transport.start_dialout(args.phone_number) # Pass transport to your bot pipeline # Your bot setup and pipeline creation here... ``` See the full bot.py with dial-out configuration and pipeline setup ### Run the Example To test dial-out functionality: 1. **Start your server**: Run your FastAPI server 2. **Trigger a call**: Make an API request to start a dial-out call 3. **Answer your phone**: The bot will call the specified number 4. **Talk to your bot**: Have a conversation with your AI agent See the full README with step-by-step setup, API usage, and configuration details ## Call Transfers Daily supports two transfer patterns: - **Cold transfer** — the bot hands the caller off to another number and leaves. The caller is connected directly to the destination; the bot is no longer involved. - **Warm transfer** — the bot stays on the call, dials a specialist, briefs them while the caller is on hold, then bridges everyone together. The bot is the room owner, so when it leaves, the Daily room terminates and any remaining PSTN legs drop — there's no "bot leaves, the two callers keep talking" mode (`max_idle_timeout_sec` only holds a bot-less room open for a fixed countdown). To keep a caller and a human on Daily together, use a warm transfer (the bot stays as the bridge); to drop the bot entirely, use a cold transfer off Daily via SIP REFER. ## Cold transfer A cold transfer hands the caller off to another number and the bot exits. ### How cold transfer works 1. **Bot receives transfer request** (via function call or user input) 2. **Bot informs the caller** about the transfer 3. **Bot initiates SIP call transfer** to the destination number 4. **Daily connects the transfer** to the destination 5. **Bot leaves the call** (cold transfer) 6. **Caller and destination** continue the conversation ### Implementing a cold transfer Call transfers are typically implemented as LLM function calls that your bot can invoke: ```python bot.py async def dial_operator(transport: BaseTransport, params: FunctionCallParams): """Function the bot can call to transfer to an operator.""" operator_number = os.getenv("OPERATOR_NUMBER", "+1234567890") # Inform the user about the transfer content = "I'm transferring you to a supervisor now. Please hold while I connect you." message = {"role": "developer", "content": content} await params.llm.push_frame(LLMMessagesAppendFrame([message], run_llm=True)) # Execute the SIP call transfer transfer_params = {"toEndPoint": operator_number} await transport.sip_call_transfer(transfer_params) ``` ### Handling Transfer Events Your bot should handle transfer-related events to manage the call flow: ```python bot.py @transport.event_handler("on_dialout_answered") async def on_dialout_answered(transport, data): logger.info(f"Transfer successful: {data}") # Cold transfer: bot leaves, caller and operator continue await worker.queue_frames([EndFrame()]) ``` See the full call transfer example with LLM function calls, event handling, and error management ## Warm transfer A warm transfer connects the caller to a human specialist after the bot briefs that specialist on the call. Unlike a cold transfer, **the bot stays in the room for the entire call and acts as the audio bridge** between the caller and the specialist. ### How warm transfer works 1. **Caller dials in** and talks to the bot. 2. **Caller asks for a specialist.** The bot calls an `initiate_warm_transfer` LLM function with the target and a short summary of the caller's issue. 3. **Bot puts the caller on hold** — it plays a hold message, enables hold music, and gates the caller's audio so they don't hear the briefing. 4. **Bot dials the specialist** with `start_dialout`. 5. **Specialist answers** (`on_dialout_answered`). The bot briefs the specialist while the caller remains on hold. 6. **Everyone is connected.** The bot takes the caller off hold and stays in the room, bridging the audio. 7. **Call ends** when the specialist hangs up after a successful connection, or when a participant leaves — the bot ends its own session with an `EndWorkerFrame`. ### Room configuration The room only needs dial-out and SIP enabled. No `max_idle_timeout_sec` or other keep-alive setting is required, because the bot stays in the room: ```python server_utils.py from pipecat.transports.daily.utils import DailyRoomProperties, DailyRoomSipParams room_properties = DailyRoomProperties( enable_dialout=True, sip=DailyRoomSipParams(display_name=call_data.from_phone), ) ``` ### Key components The example builds the flow from a few reusable pieces: - **A coordinator processor** (`TransferCoordinator`) — a `FrameProcessor` that drives the transfer as a small state machine: start transfer → hold caller and dial specialist → connect on answer → end on hang-up. Keeping the state in one processor avoids scattering call-flow logic across event handlers. - **Hold music** via `SoundfileMixer`, toggled with `MixerEnableFrame(True/False)` so the caller hears music while on hold and the specialist's ringing once dial-out connects. - **Audio gating** with a mute strategy and a `CustomerHoldFrame`, so the caller's audio is held during the briefing and released once everyone is connected. See the full warm transfer example: the transfer coordinator, hold music, audio gating, the LLM transfer function, and dial-out event handling. ## Deployment ### Pipecat Cloud For production deployment without managing your own infrastructure, use Pipecat Cloud. Pipecat Cloud handles all webhook infrastructure, room creation, and bot scaling automatically. Complete guide for deploying dial-in bots with automatic webhook handling Complete guide for deploying dial-out bots with caller ID management ### Self-Hosted Deployment For self-hosted production deployment, ensure your servers are: - Publicly accessible with HTTPS - Able to handle concurrent requests - Properly configured with your Daily API credentials ## Next Steps - Explore the [complete examples](https://github.com/pipecat-ai/pipecat/tree/main/examples) for full implementations - Learn about [Daily's SIP integration](./twilio-daily-sip) for more advanced telephony scenarios - Check out [Daily's REST API documentation](https://docs.daily.co/reference) for additional configuration options # Daily SIP Source: https://docs.pipecat.ai/pipecat/telephony/daily-sip.md Use Daily as the SIP provider for dial-in and dial-out with any SIP-capable telephony carrier (Twilio, Telnyx, Plivo, etc.). This guide covers the **`provider="daily"`** SIP mode, where Daily directly connects its SIP leg to your telephony carrier (Twilio, Telnyx, Five9, Genseys, Cisco, …). This SIP mode with `provider="daily"` gives you: - **Static egress IPs** you can allow-list on the carrier side for tighter ACLs. - **One SIP configuration** that works across carriers — the bot code doesn't change when you swap providers. The reference implementation lives in the Twilio SIP examples because Twilio is the most common carrier, but the same bot code works with any SIP-capable provider — only the carrier-side webhook / SIP URI changes. Inbound calls routed from a carrier into a Daily room via `provider="daily"` SIP Outbound calls initiated from Daily and routed out through the carrier ## Things you'll need - A Daily API key. - An account with a SIP-capable carrier (Twilio, Telnyx, Plivo, Exotel, …) with at least one provisioned number. ## Environment Setup ```shell .env DAILY_API_KEY=... DAILY_API_URL=https://api.daily.co/v1 OPENAI_API_KEY=... CARTESIA_API_KEY=... ``` ## Creating a Daily room with `provider="daily"` The key difference from the carrier-default SIP flow is a single field on the room SIP config: ```python server_utils.py sip_config = await configure( session, sip_caller_phone=call_data.from_phone, sip_provider="daily", # <-- routes SIP through Daily's own infrastructure enable_dialout=True, room_geo="us-east-1", # <-- optional, this anchors the Daily Room to a location ) ``` This returns a `sip_endpoint` you hand to your carrier for call forwarding, and (for dial-out) a room configured to accept `start_dialout` with `{"provider": "daily", ...}`. Daily's SIP address are of the format: `sip:$roomName.$index@$domainName.sip-us.daily.co`. ## Dial-in Dial-in lets a caller reach your bot by calling a carrier-owned phone number. The carrier forwards the audio into Daily over SIP. ### Flow 1. Carrier receives the incoming call and hits your webhook server. 2. Your server creates a Daily room with `sip_provider="daily"` and spawns the bot. 3. Your server responds to the carrier with hold music. So the caller isn't in silence while the bot boots. 4. The bot fires `on_dialin_ready` with a Daily SIP endpoint URI. 5. The bot calls the carrier's API to update the in-progress call, forwarding its audio to that SIP endpoint. 6. Caller and bot are connected; Daily carries the media over WebRTC. ### Bot configuration ```python bot.py from pipecat.transports.daily import DailyTransport, DailyParams async def run_bot(room_url: str, token: str, call_sid: str, sip_endpoint: str): transport = DailyTransport( room_url, token, "Voice Bot", DailyParams( audio_in_enabled=True, audio_out_enabled=True, video_out_enabled=False, transcription_enabled=True, ), ) call_already_forwarded = False @transport.event_handler("on_dialin_ready") async def on_dialin_ready(transport, cdata): nonlocal call_already_forwarded if call_already_forwarded: return # Forward the carrier's in-progress call to Daily's SIP endpoint. # This example uses Twilio; swap for your carrier's call-update API. twilio_client.calls(call_sid).update( twiml=f"{sip_endpoint}" ) call_already_forwarded = True ``` Full FastAPI server, bot, and README with setup instructions ## Dial-out Dial-out initiates an outbound call from Daily, routed out through the carrier. ### Flow 1. Your app triggers a dial-out (API call, agent decision, scheduled task). 2. Server creates a Daily room with `sip_provider="daily"` and `enable_dialout=True`. 3. Bot joins the Daily room and sets up the WebRTC transport. 4. Bot calls `transport.start_dialout(...)` with `provider="daily"` and a SIP URI pointing at the carrier. 5. Carrier places the PSTN call to the destination number. 6. Recipient answers; media flows through Daily's WebRTC transport to the bot. ### Bot configuration ```python bot.py from pipecat.transports.daily import DailyTransport, DailyParams async def run_bot(room_url: str, token: str, target_number: str, sip_uri: str): transport = DailyTransport( room_url, token, "Voice Bot", DailyParams( audio_in_enabled=True, audio_out_enabled=True, video_out_enabled=False, transcription_enabled=True, ), ) @transport.event_handler("on_joined") async def on_joined(transport, data): await transport.start_dialout( { "sipUri": sip_uri, "displayName": "Pipecat Bot", "provider": "daily", } ) @transport.event_handler("on_dialout_connected") async def on_dialout_connected(transport, data): logger.info(f"Dial-out connected: {data}") @transport.event_handler("on_dialout_stopped") async def on_dialout_stopped(transport, data): logger.info(f"Dial-out stopped: {data}") await worker.cancel() @transport.event_handler("on_dialout_warning") async def on_dialout_warning(transport, data): logger.warning(f"Dial-out warning: {data}") ``` Full FastAPI server, bot, and README with setup instructions ## DTMF When `provider="daily"` is in use, Daily surfaces DTMF tones from the connected carrier leg and lets you send tones back out over the same session. ### Receiving DTMF Register `on_dtmf_event` on the transport. The event fires once per keypress and the payload contains the `sessionId` of the calling leg and the pressed `tone`: ```python bot.py @transport.event_handler("on_dtmf_event") async def on_dtmf_event(transport, data): logger.info(f"DTMF event: {data}") # data = {"sessionId": "...", "tone": "1", ...} ``` Internally, Pipecat also pushes each inbound digit as an `InputDTMFFrame` into the pipeline, so processors like `DTMFAggregator` can collect a sequence and feed it to an LLM as context; use that path if you want the bot to "hear" DTMF alongside speech. ### Sending DTMF To send DTMF tones back to the caller, send them through Daily's native DTMF channel. Use the session id from the inbound event (or the dial-out session id captured in `on_dialout_connected`) as the target. The default `method`to send the DTMF tones is `auto`, determined in the SIP offer/answer negotiation. However, if you already know what the remote party supports, then pick the appropriate option: `telephone-events` which are in-band RTP packets (RFC2833/4733) or as a `sip-info` message. ```python bot.py @transport.event_handler(“on_dialout_answered”) async def on_dialout_answered(transport, data): logger.info(f”Dial-out answered: {data}”) # Wait briefly for the IVR prompt before sending the PIN await asyncio.sleep(1) err = await transport.send_dtmf({ “sessionId”: data[“sessionId”], # the dialed-out participant “tones”: “1234#”, # full sequence in one call “digitDurationMs”: 100, # “method”: “sip-info” | “telephone-event” | “auto” }) if err: logger.error(f”send_dtmf failed: {err}”) ``` You can also push `OutputDTMFFrame` / `OutputDTMFUrgentFrame` through the pipeline for cases where the tones are part of your bot's conversational logic (IVR navigation, confirming a menu choice) rather than a direct invocation: ```python # Single key (backward compatible) await transport.queue_frame(OutputDTMFUrgentFrame(button=KeypadEntry.ONE)) # Multi-key dial string await transport.queue_frame(OutputDTMFFrame.from_string("1234#")) # Daily with explicit session + method await transport.queue_frame( DailyOutputDTMFFrame.from_string( "1234#", session_id="abc", digit_duration_ms=80, method="sip-info", ) ) ``` ## Best Practices ### Guard against duplicate forwarding `on_dialin_ready` can fire more than once when you have multiple sip endpoints defined on a room, for example for supervisory actions (silent monitoring, barge-in). In this case, you will need to keep track of which sip endpoint is assigned to which incoming call. Daily's SIP address are of the format: `sip:$roomName.$index@$domainName.sip-us.daily.co`. ### Allow-list Daily's static IPs One of the main reasons to pick `provider="daily"` is that Daily publishes static IPs that you can allow-list on the carrier side, tightening SIP ACLs. Fetch the [current list](https://ip-info.daily.co/ips/ip-info.json) from Daily's [Networking Guide](https://docs.daily.co/guides/privacy-and-security/corporate-firewalls-nats-allowed-ip-list) and configure your SIP trunk / IP access control list accordingly. There are different Static IP addresses for SIP signaling and media traffic. ## Next Steps - See [Daily + Twilio SIP](./twilio-daily-sip) for the carrier-specific walkthrough (Twilio webhooks, TwiML, call forwarding). - See [Daily PSTN](./daily-pstn) to skip the carrier entirely and have Daily provision the number. - For simpler carrier-hosted telephony (no SIP), see [Twilio WebSockets](./twilio-websockets), [Telnyx WebSockets](./telnyx-websockets), [Plivo WebSockets](./plivo-websockets), or [Exotel WebSockets](./exotel-websockets). # Daily + Twilio SIP Source: https://docs.pipecat.ai/pipecat/telephony/twilio-daily-sip.md Complete guide to using Daily's WebRTC transport with Twilio's SIP services for dial-in and dial-out ## Things you'll need - An active [Twilio](https://www.twilio.com) developer account with API credentials - One or more Twilio provisioned phone numbers - Daily API key for WebRTC transport - The Twilio Python client library (`uv add twilio`) Complete dial-in implementation using Twilio SIP with Daily WebRTC transport Outbound calling using Daily WebRTC transport with Twilio SIP routing ## Phone Number Setup You'll need Twilio phone numbers for both dial-in and dial-out functionality: - Visit [console.twilio.com](https://console.twilio.com) and purchase phone numbers - Ensure your numbers support Voice capabilities - Configure webhook URLs for dial-in numbers (covered below) ## Environment Setup Configure your environment variables for both Twilio and Daily: ```shell .env DAILY_API_KEY=... DAILY_API_URL=https://api.daily.co/v1 TWILIO_ACCOUNT_SID=... TWILIO_AUTH_TOKEN=... OPENAI_API_KEY=... CARTESIA_API_KEY=... ``` ## Dial-in Dial-in allows users to call your Twilio number and connect to your Pipecat bot via Daily's WebRTC transport. ### How It Works Here's the sequence of events when someone calls your Twilio number: 1. **Twilio receives an incoming call** to your phone number 2. **Twilio calls your webhook server** (`/call` endpoint) 3. **Your server creates a Daily room** with SIP capabilities 4. **Your server starts the bot process** with the room details 5. **Your server responds to Twilio** with TwiML that puts the caller on hold with music 6. **Upon receiving the `on_dialin_ready` event**, the bot forwards the call to the Daily SIP endpoint 7. **The caller and bot are connected**, and the bot handles the conversation ### Set up your webhook server Your server acts as the orchestrator between Twilio's telephony infrastructure and Daily's WebRTC transport. When Twilio receives a call, it sends a webhook to your server. Your server then: 1. **Extracts call information** from Twilio's webhook (CallSid, caller details) 2. **Creates a Daily room** with SIP capabilities enabled 3. **Spawns a bot process** with the room details and call information 4. **Responds to Twilio** with TwiML that puts the caller on hold with music 5. **Coordinates the connection** between Twilio SIP and Daily WebRTC The server handles the complex orchestration between two different telephony systems, ensuring seamless audio quality through Daily's WebRTC transport while leveraging Twilio's robust SIP infrastructure. See the full FastAPI server code with Twilio webhook handling and Daily room creation ### Configure your Pipecat bot for dial-in The bot receives arguments from the server process and uses them to coordinate between Daily's WebRTC transport and Twilio's SIP infrastructure. The key components are: **Bot Entry Point**: The bot receives arguments containing: - `room_url`: Daily room URL for the WebRTC connection - `token`: Daily room token for authentication - `call_sid`: Twilio call identifier for SIP forwarding - `sip_endpoint`: Daily's SIP endpoint URL for call routing **Transport Creation**: These arguments are used to configure the `DailyTransport` for WebRTC: ```python bot.py from pipecat.transports.daily import DailyTransport, DailyParams async def run_bot(room_url: str, token: str, call_sid: str, sip_endpoint: str): # Create Daily WebRTC transport transport = DailyTransport( room_url, token, "Voice Bot", DailyParams( audio_in_enabled=True, audio_out_enabled=True, video_out_enabled=False, transcription_enabled=True, ) ) # Handle when Daily SIP endpoint is ready for call forwarding @transport.event_handler("on_dialin_ready") async def on_dialin_ready(transport, cdata): # Forward the Twilio call to Daily's SIP endpoint twilio_client.calls(call_sid).update( twiml=f"{sip_endpoint}" ) logger.info("Call forwarded from Twilio to Daily SIP endpoint") # Your bot pipeline setup here... ``` **Call Forwarding Flow**: The `on_dialin_ready` event is crucial. It signals when Daily's SIP infrastructure is ready to receive the call. At this point, the bot uses Twilio's API to update the call with new TwiML that forwards the audio from Twilio's SIP to Daily's SIP endpoint, completing the connection. See the full bot.py with Daily transport setup and Twilio call forwarding ### Set up Twilio webhook Configure your Twilio phone number to use your server's webhook URL: 1. Go to the [Twilio Console](https://console.twilio.com) 2. Navigate to Phone Numbers → Manage → Active Numbers 3. Click on your phone number 4. Under "Configure", set "A Call Comes In" to: - Webhook: `https://your-server.com/call` (your server's URL) - HTTP Method: POST ### Run the Example For local development, you can use [ngrok](https://ngrok.com/) to expose your local server: ```shell # Start your server python server.py # In another terminal, start ngrok ngrok http 8000 # Use the ngrok URL (e.g., https://a1b2c3.ngrok.io/call) as your webhook ``` See the full README with step-by-step setup, webhook configuration, and troubleshooting ## Dial-out Dial-out allows your bot to initiate calls through Twilio's SIP infrastructure using Daily's WebRTC transport. ### How It Works Here's the sequence of events for dial-out calls: 1. **Your application triggers a dial-out** (via API call or user action) 2. **Server creates a Daily room** with SIP capabilities 3. **Bot joins the Daily room** and sets up the WebRTC transport 4. **Bot initiates SIP call** through Twilio to the target number 5. **Twilio routes the call** to the destination phone number 6. **The recipient answers** and is connected to your bot via Daily WebRTC 7. **The bot handles the conversation** with high-quality WebRTC audio ### Set up your server for dial-out The dial-out server orchestrates outbound calls by coordinating between your application, Daily's WebRTC transport, and Twilio's SIP infrastructure. When you trigger a dial-out call, your server: 1. **Receives the dial-out request** with target phone number and call parameters 2. **Creates a Daily room** with SIP capabilities and dial-out enabled 3. **Spawns a bot process** with the room details and target phone number 4. **Initiates the SIP call** through Twilio to the destination number 5. **Manages the connection** between Daily WebRTC and Twilio SIP routing 6. **Handles call events** like answered, busy, or failed connections The server simplifies outbound calling by abstracting the complexity of coordinating between WebRTC transport and SIP telephony, providing high-quality audio through Daily while leveraging Twilio's reliable call routing. See the full FastAPI server code with Daily room creation and SIP dial-out management ### Configure your Pipecat bot for dial-out The dial-out bot receives arguments from the server process and uses them to initiate outbound calls through Twilio's SIP infrastructure via Daily's WebRTC transport. The key components are: **Bot Entry Point**: The bot receives arguments containing: - `room_url`: Daily room URL for the WebRTC connection - `token`: Daily room token for authentication - `target_number`: Phone number to call via Twilio SIP - `sip_uri`: Twilio SIP URI for routing the outbound call **Transport Creation**: These arguments are used to configure the `DailyTransport` for dial-out: ```python bot.py from pipecat.transports.daily import DailyTransport, DailyParams async def run_bot(room_url: str, token: str, target_number: str, sip_uri: str): # Create Daily WebRTC transport transport = DailyTransport( room_url, token, "Voice Bot", DailyParams( audio_in_enabled=True, audio_out_enabled=True, video_out_enabled=False, transcription_enabled=True, ) ) # Initiate the outbound call through Twilio SIP await transport.start_dialout(sip_uri) # Your bot pipeline setup here... ``` **Dial-out Flow**: The `start_dialout()` method initiates the call by connecting Daily's WebRTC transport to Twilio's SIP infrastructure. The `sip_uri` parameter contains the Twilio SIP endpoint configured with the target phone number, allowing Daily to route the call through Twilio's telephony network to reach the destination. See the full bot.py with WebRTC transport setup and SIP dial-out configuration ### Run the Example To test dial-out functionality: 1. **Start your server**: Run your FastAPI server 2. **Trigger a call**: Make an API request to start a dial-out call 3. **Answer your phone**: The bot will call the specified number via Twilio 4. **Talk to your bot**: Have a conversation with high-quality WebRTC audio See the full README with step-by-step setup, API usage, and SIP configuration ## Best Practices ### Hold Music for Initialization Always respond to Twilio's initial webhook with hold music to give your bot time to initialize: ```python resp = VoiceResponse() resp.play(url="https://your-hold-music.mp3", loop=10) return str(resp) ``` **Avoid using ``** - Twilio's pause duration is limited and may not provide enough time for Daily SIP setup. ### Handle Multiple Events Use a flag to ensure you only forward calls once when handling multiple `on_dialin_ready` events: ```python call_already_forwarded = False @transport.event_handler("on_dialin_ready") async def on_dialin_ready(transport, cdata): nonlocal call_already_forwarded if call_already_forwarded: return # Forward the call... call_already_forwarded = True ``` ## Deployment ### Local Development Use [ngrok](https://ngrok.com/) to expose your local server for testing: ```shell python server.py ngrok http 8000 # Use the ngrok URL as your Twilio webhook ``` ### Pipecat Cloud Deployment For production deployment on Pipecat Cloud, your webhook server calls the `/{agent}/start` endpoint instead of spawning local bot processes: ```python # Instead of subprocess.Popen(bot_cmd) response = await httpx.post( f"https://api.pipecat.daily.co/v1/public/{agent_id}/start", headers={"Authorization": f"Bearer {pipecat_api_key}"}, json={ "createDailyRoom": True, "dailyRoomProperties": { "sip": {"sip_mode": "dial-in", "num_endpoints": 1} }, "body": { "room_url": room_url, "token": token, "call_sid": call_sid, # For dial-in "sip_endpoint": sip_endpoint, # Or for dial-out: # "target_number": target_number, # "sip_uri": sip_uri } } ) ``` The room information and call parameters are passed in the `body` field, which your Pipecat Cloud bot receives as arguments. ### Self-Hosted Production Deployment For self-hosted production deployment, ensure your servers are: - Publicly accessible with HTTPS - Able to handle concurrent webhook requests - Properly configured with both Twilio and Daily API credentials ## Next Steps - Explore the [complete examples](https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot) for full implementations - See [Daily SIP](./daily-sip) for the provider-agnostic `provider="daily"` flow that works with any SIP carrier and includes DTMF send/receive - Learn about [Daily PSTN integration](./daily-pstn) for direct phone number provisioning - Check out [Twilio WebSocket integration](./twilio-websockets) for simpler telephony workflows # Twilio WebSocket Integration Source: https://docs.pipecat.ai/pipecat/telephony/twilio-websockets.md Complete guide to using Twilio Media Streams with Pipecat for dial-in and dial-out functionality ## Things you'll need - An active [Twilio](https://www.twilio.com) account with API credentials - One or more Twilio provisioned phone numbers - A tunneling service like [ngrok](https://ngrok.com/) for local development - API keys for speech-to-text, text-to-speech, and LLM services Complete dial-in implementation using Twilio Media Streams over WebSocket Outbound calling using Twilio Media Streams with WebSocket transport ## Phone Number Setup You'll need Twilio phone numbers for both dial-in and dial-out functionality: - Visit [console.twilio.com](https://console.twilio.com) and purchase phone numbers - Ensure your numbers support Voice capabilities - Configure webhook URLs for dial-in numbers (covered below) ## Environment Setup Configure your environment variables for Twilio and AI services: ```shell .env TWILIO_ACCOUNT_SID=... TWILIO_AUTH_TOKEN=... OPENAI_API_KEY=... DEEPGRAM_API_KEY=... CARTESIA_API_KEY=... ``` ## Dial-in Dial-in allows users to call your Twilio number and connect to your Pipecat bot via WebSocket Media Streams. ### How It Works Here's the sequence of events when someone calls your Twilio number: 1. **Twilio sends WebSocket messages**: Twilio processes the associated TwiML Bin and starts a WebSocket stream to your bot (local or Pipecat Cloud) 2. **Parse the WebSocket messages**: Your bot parses the WebSocket connection messages to set up the corresponding Pipecat transport 3. **(Optional) Look up the caller**: Optionally, look up the caller using Twilio's REST API to retrieve custom information about the call and personalize your bot's behavior 4. **Bot starts responding**: Once the pipeline is started, your bot will initiate the conversation ### Setting Up Twilio #### 1. Create a TwiML Bin A TwiML Bin tells Twilio how to handle incoming calls. You'll create one that establishes a WebSocket connection to your bot. 1. Go to the [Twilio Console](https://console.twilio.com) 2. Navigate to **TwiML Bins** → **My TwiML Bins** 3. Click the **+** to create a new TwiML Bin 4. Name your bin and add the TwiML: ```xml ``` Replace `your-url.ngrok.io` with your ngrok URL. ```xml ``` Replace: - `AGENT_NAME` with the name of the agent you deployed to Pipecat Cloud - `ORGANIZATION_NAME` with the name of your Pipecat Cloud organization 5. Click **Save** #### 2. Assign TwiML Bin to Your Phone Number 1. Navigate to **Phone Numbers** → **Manage** → **Active Numbers** 2. Click on your Twilio phone number 3. In the "Voice Configuration" section: - Set "A call comes in" to **TwiML Bin** - Select the TwiML Bin you created 4. Click **Save configuration** ### Running the Example You'll need two terminal windows to run the example. 1. In your first terminal, start ngrok to expose your local server: ```shell ngrok http 7860 ``` > Tip: Use the `--subdomain` flag for a reusable ngrok URL. 2. In your second terminal, install dependencies: ```shell uv sync ``` 3. Then run the bot: ```shell uv run bot.py -t twilio ``` `-t twilio` restricts the runner to Twilio so it registers the TwiML webhook. The runner otherwise serves all transports at once when `-t` is omitted. 4. Call your bot by placing a call to the number associated with your bot. The bot will answer and start the conversation. ### Personalizing Your Bot with Caller Information When a call comes in, your bot receives a Call SID from Twilio. You can use this Call SID to fetch caller information from Twilio's REST API, including the caller's phone number. With this information, you can: - Look up customer information in your database - Personalize greetings and responses based on caller identity - Route calls to different bot behaviors (e.g., VIP handling, support vs sales) - Implement custom business logic based on the caller The example bot demonstrates how to fetch caller information asynchronously using `aiohttp` to avoid blocking the event loop, ensuring your bot remains responsive even with multiple concurrent calls. See the full implementation with WebSocket parsing, REST API integration, and caller personalization ## Dial-out Dial-out allows your bot to initiate calls to phone numbers using Twilio's outbound calling capabilities with WebSocket Media Streams. ### How It Works Here's the sequence of events for dial-out calls: 1. **Your application triggers a dial-out** (via API call or user action) 2. **Server initiates a Twilio call** to the target phone number 3. **Twilio establishes the call** and opens a WebSocket connection 4. **Your bot joins the WebSocket** and sets up the pipeline 5. **The recipient answers** and is connected to your bot 6. **The bot handles the conversation** with real-time audio streaming ### Set up your server for dial-out The dial-out server creates outbound calls and manages WebSocket connections. When you trigger a dial-out call, your server: 1. **Receives the dial-out request** with target phone number and parameters 2. **Creates a Twilio call** using the REST API with TwiML that establishes WebSocket 3. **Accepts the WebSocket** connection from Twilio 4. **Parses call data** including any custom parameters 5. **Runs the Pipecat bot** with the call information See the full FastAPI server code with outbound call creation and WebSocket handling ### Configure your Pipecat bot for dial-out The dial-out bot configuration is similar to dial-in, but you can pass custom parameters when creating the outbound call: **Custom Parameters for Dial-out**: Pass information to your bot through TwiML parameters: ```python server.py # When creating the outbound call, include custom parameters twiml = f""" """ # Create the outbound call call = twilio_client.calls.create( to=target_number, from_=your_twilio_number, twiml=twiml ) ``` **Bot Configuration**: The bot receives and uses these parameters: ```python bot.py # Parse WebSocket data (same as dial-in) transport_type, call_data = await parse_telephony_websocket(websocket) # Access custom parameters custom_params = call_data["body"] campaign_id = custom_params.get("campaign_id") customer_id = custom_params.get("customer_id") # Customize bot behavior for outbound calls if campaign_id == "summer_sale": greeting = f"Hi! I'm calling about our summer sale promotion..." ``` See the full bot.py with outbound call handling and parameter usage ### Run the Example To test dial-out functionality: 1. **Start your server**: Run your FastAPI server 2. **Trigger a call**: Make an API request to start an outbound call 3. **Answer your phone**: The bot will call the specified number 4. **Talk to your bot**: Have a conversation with your AI agent See the full README with step-by-step setup, API usage, and outbound call configuration ## Key Features ### Audio Format and Sample Rate Twilio Media Streams uses 8kHz mono audio with 16-bit PCM encoding. To avoid resampling, set the audio input and output sample rate to 8000 Hz: ```python worker = PipelineWorker( pipeline, params=PipelineParams( audio_in_sample_rate=8000, audio_out_sample_rate=8000, ), ) ``` ### Automatic Call Termination `TwilioFrameSerializer` ends the call when your pipeline ends. This is on by default (`auto_hang_up=True`) and requires `call_sid`, `account_sid`, and `auth_token` — the serializer raises a `ValueError` at construction if any are missing: ```python serializer = TwilioFrameSerializer( stream_sid=stream_id, call_sid=call_id, account_sid=os.getenv("TWILIO_ACCOUNT_SID"), auth_token=os.getenv("TWILIO_AUTH_TOKEN"), ) ``` On an `EndFrame` or `CancelFrame`, the serializer calls Twilio's hangup API for the call, ending the call leg immediately. If the call needs to outlive your bot — for example, you've transferred the caller to a human agent — disable it: ```python serializer = TwilioFrameSerializer( stream_sid=stream_id, call_sid=call_id, params=TwilioFrameSerializer.InputParams(auto_hang_up=False), ) ``` With `auto_hang_up=False`, Pipecat no longer ends the call — your TwiML document does. When the WebSocket closes, `` completes and TwiML resumes at the next verb. If the document runs out of verbs, Twilio ends the call, including a leg you've transferred to a human. Make sure your TwiML keeps the document alive for as long as the call should last. See Twilio's [`` documentation](https://www.twilio.com/docs/voice/twiml/connect) for the `action` attribute and TwiML call flow control. For the full parameter list, see the [`TwilioFrameSerializer` reference](/api-reference/server/services/serializers/twilio). ## Deployment ### Pipecat Cloud Deployment For production deployment without managing your own infrastructure, use Pipecat Cloud: Deploy Twilio WebSocket bots with automatic scaling and managed infrastructure ### Self-Hosted Production Deployment For fully self-hosted production deployment, ensure your servers are: - Publicly accessible with HTTPS - Able to handle concurrent WebSocket connections - Properly configured with Twilio API credentials - Implementing proper error handling and logging ## Next Steps - Explore the [complete examples](https://github.com/pipecat-ai/pipecat-examples/tree/main/twilio-chatbot) for full implementations - Learn about [Daily + Twilio SIP integration](./twilio-daily-sip) for advanced telephony scenarios - Check out [Daily PSTN integration](./daily-pstn) for direct phone number provisioning # Telnyx WebSocket Integration Source: https://docs.pipecat.ai/pipecat/telephony/telnyx-websockets.md Complete guide to using Telnyx Media Streaming with Pipecat for dial-in and dial-out functionality ## Things you'll need - An active [Telnyx](https://telnyx.com) account with API credentials - One or more Telnyx provisioned phone numbers - A public-facing server or tunneling service like [ngrok](https://ngrok.com/) (for dial-out only) - API keys for speech-to-text, text-to-speech, and LLM services Complete dial-in implementation using Telnyx Media Streaming over WebSocket Outbound calling using Telnyx Media Streaming with WebSocket transport ## Phone Number Setup You'll need Telnyx phone numbers for both dial-in and dial-out functionality: - Visit [telnyx.com](https://telnyx.com) and purchase phone numbers - Ensure your numbers support Voice capabilities - Configure TeXML applications for dial-in numbers (covered below) ## Environment Setup Configure your environment variables for Telnyx and AI services: ```shell .env TELNYX_API_KEY=... OPENAI_API_KEY=... DEEPGRAM_API_KEY=... CARTESIA_API_KEY=... ``` ## Dial-in Dial-in allows users to call your Telnyx number and connect to your Pipecat bot via WebSocket Media Streaming. Unlike Twilio, Telnyx automatically provides caller information (to/from numbers) in the WebSocket messages, so no custom server is needed for basic dial-in functionality. ### How It Works Here's the sequence of events when someone calls your Telnyx number: 1. **Telnyx receives an incoming call** to your phone number 2. **Telnyx executes your TeXML application** which establishes a WebSocket connection 3. **Telnyx opens a WebSocket** to your server with real-time audio and call metadata 4. **Your bot processes the audio** using the Pipecat pipeline 5. **The bot responds with audio** sent back to Telnyx over WebSocket 6. **Telnyx plays the audio** to the caller in real-time ### Set up your TeXML application Telnyx uses TeXML (Telnyx Extensible Markup Language) to control call flow. For dial-in, you create a TeXML application that establishes a WebSocket connection directly to your bot: ```xml TeXML for dial-in ``` The `bidirectionalMode="rtp"` parameter enables real-time audio streaming in both directions. ### Personalizing Your Bot with Caller Information Telnyx automatically includes caller information (to/from numbers) in the WebSocket messages. You can use the caller's phone number to: - Look up customer information in your database - Personalize greetings and responses based on caller identity - Route calls to different bot behaviors (e.g., VIP handling, support vs sales) - Implement custom business logic based on the caller The example bot demonstrates how to extract caller information from Telnyx's WebSocket messages and use it to personalize the conversation. ### Configure your Pipecat bot for dial-in Your bot receives the WebSocket connection and automatically gets call information from Telnyx's WebSocket messages. The key components are: **WebSocket Parsing**: Pipecat's built-in parser extracts call data from Telnyx's WebSocket messages: ```python bot.py from pipecat.runner.utils import parse_telephony_websocket async def run_bot(websocket: WebSocket): # Parse Telnyx WebSocket data transport_type, call_data = await parse_telephony_websocket(websocket) # Extract call information (automatically provided by Telnyx) # Data format: { # "stream_id": str, # "call_control_id": str, # "outbound_encoding": str, # "from": str, # "to": str, # } stream_id = call_data["stream_id"] call_control_id = call_data["call_control_id"] outbound_encoding = call_data["outbound_encoding"] from_number = call_data["from"] # Caller's number to_number = call_data["to"] # Your Telnyx number ``` **Transport Creation**: Configure the WebSocket transport with Telnyx serialization: ```python bot.py # Create Telnyx serializer with call details serializer = TelnyxFrameSerializer( stream_id=stream_id, call_control_id=call_control_id, outbound_encoding=outbound_encoding, inbound_encoding="PCMU", api_key=os.getenv("TELNYX_API_KEY"), ) # Configure WebSocket transport transport = FastAPIWebsocketTransport( websocket=websocket, params=FastAPIWebsocketParams( audio_in_enabled=True, audio_out_enabled=True, add_wav_header=False, serializer=serializer, ), ) # Your bot pipeline setup here... ``` See the full bot.py with WebSocket parsing, transport setup, and pipeline configuration ### Set up Telnyx TeXML application Configure your Telnyx phone number to use your TeXML application: 1. Go to the [Telnyx Portal](https://portal.telnyx.com) 2. Navigate to Voice → Programmable Voice → TeXML Applications 3. Create a new TeXML Application with your WebSocket URL 4. Assign the TeXML Application to your phone number See the full README with step-by-step setup, TeXML configuration, and testing ## Dial-out Dial-out allows your bot to initiate calls to phone numbers using Telnyx's outbound calling capabilities with WebSocket Media Streaming. ### How It Works Here's the sequence of events for dial-out calls: 1. **Your application triggers a dial-out** (via API call or user action) 2. **Server initiates a Telnyx call** using the Call Control API 3. **Telnyx establishes the call** and opens a WebSocket connection 4. **Your bot joins the WebSocket** and sets up the pipeline 5. **The recipient answers** and is connected to your bot 6. **The bot handles the conversation** with real-time audio streaming ### Set up your server for dial-out The dial-out server creates outbound calls and manages WebSocket connections. When you trigger a dial-out call, your server: 1. **Receives the dial-out request** with target phone number and parameters 2. **Creates a Telnyx call** using the Call Control API with a webhook URL 3. **Accepts the WebSocket** connection from Telnyx 4. **Parses call data** including call control information 5. **Runs the Pipecat bot** with the call information See the full FastAPI server code with outbound call creation and WebSocket handling ### Passing Custom Data to Your Bot For dial-out calls, you often need to pass custom data to personalize the conversation—such as user information, campaign details, or context from your application. The dial-out example supports passing a `body` object that becomes available to your bot via `runner_args.body`. **Triggering a call with custom data:** ```bash curl -X POST https://your-server.com/start \ -H "Content-Type: application/json" \ -d '{ "phone_number": "+1234567890", "body": { "user": { "id": "user123", "firstName": "John", "lastName": "Doe", "accountType": "premium" }, "campaign": "renewal_reminder", "context": "subscription expiring in 3 days" } }' ``` **How it works:** 1. The `/start` endpoint receives the request with `phone_number` and optional `body` 2. The `body` is base64-encoded and passed through the TeXML URL chain 3. When the WebSocket connects, the server decodes the body and passes it to your bot via `runner_args.body` 4. Your bot accesses the custom data to personalize the conversation **Accessing custom data in your bot:** ```python bot.py async def bot(runner_args: RunnerArguments): body_data = runner_args.body or {} first_name = body_data.get("user", {}).get("firstName", "there") # Use first_name to personalize greetings, prompts, etc. ``` This approach works consistently in both local development and Pipecat Cloud production deployments. ### Configure your Pipecat bot for dial-out The dial-out bot configuration is similar to dial-in, with Telnyx automatically providing call information. **Call Information**: Telnyx provides call details in the WebSocket messages: ```python bot.py # Parse WebSocket data (same as dial-in) transport_type, call_data = await parse_telephony_websocket(websocket) # Extract call information stream_id = call_data["stream_id"] call_control_id = call_data["call_control_id"] from_number = call_data["from"] # Your Telnyx number to_number = call_data["to"] # Target number you're calling # Customize bot behavior for outbound calls greeting = f"Hi! This is an automated call from {from_number}. How are you today?" ``` **Custom Parameters for Dial-out**: You can pass custom data to your bot by adding query parameters to the WebSocket URL in your TeXML: ```python server.py # When creating the TeXML for outbound calls, include custom parameters texml = f""" """ ``` These parameters are accessible in your server code and can be used to customize your bot's behavior based on the specific call context. See the full bot.py with outbound call handling and call information usage ### Run the Example To test dial-out functionality: 1. **Start your server**: Run your FastAPI server 2. **Trigger a call**: Make an API request to start an outbound call 3. **Answer your phone**: The bot will call the specified number 4. **Talk to your bot**: Have a conversation with your AI agent See the full README with step-by-step setup, API usage, and outbound call configuration ## Key Features ### Audio Format and Sample Rate Telnyx Media Streaming uses 8kHz mono audio with 16-bit PCM encoding. Configure your pipeline accordingly: ```python worker = PipelineWorker( pipeline, params=PipelineParams( audio_in_sample_rate=8000, audio_out_sample_rate=8000, ), ) ``` ### Automatic Call Termination `TelnyxFrameSerializer` ends the call when your pipeline ends. This is on by default (`auto_hang_up=True`) and requires both `call_control_id` and a Telnyx API key — the serializer raises a `ValueError` at construction if either is missing: ```python serializer = TelnyxFrameSerializer( stream_id=stream_id, call_control_id=call_control_id, outbound_encoding=outbound_encoding, inbound_encoding="PCMU", api_key=os.getenv("TELNYX_API_KEY"), ) ``` On an `EndFrame` or `CancelFrame`, the serializer calls Telnyx's hangup action for the call, ending the call leg immediately. If the call needs to outlive your bot — for example, you've transferred the caller to a human agent — disable it: ```python serializer = TelnyxFrameSerializer( stream_id=stream_id, call_control_id=call_control_id, outbound_encoding=outbound_encoding, inbound_encoding="PCMU", params=TelnyxFrameSerializer.InputParams(auto_hang_up=False), ) ``` With `auto_hang_up=False`, Pipecat no longer ends the call — your TeXML document does. When the WebSocket closes, `` completes and TeXML resumes at the next verb. If the document runs out of verbs, Telnyx ends the call, including a leg you've transferred to a human. Make sure your TeXML keeps the document alive for as long as the call should last. See Telnyx's [`` documentation](https://developers.telnyx.com/docs/voice/programmable-voice/texml-verbs/connect) for the `action` attribute and TeXML call flow control. For the full parameter list, see the [`TelnyxFrameSerializer` reference](/api-reference/server/services/serializers/telnyx). ### Built-in Call Information Unlike other providers, Telnyx automatically includes caller information (to/from numbers) in the WebSocket messages, eliminating the need for custom webhook servers in basic dial-in scenarios. ## Deployment ### Pipecat Cloud Deployment For production deployment without managing your own infrastructure, use Pipecat Cloud: Deploy Telnyx WebSocket bots with automatic scaling and managed infrastructure ### Self-Hosted Production Deployment For fully self-hosted production deployment, ensure your servers are: - Publicly accessible with HTTPS - Able to handle concurrent WebSocket connections - Properly configured with Telnyx API credentials - Implementing proper error handling and logging ## Next Steps - Explore the [complete examples](https://github.com/pipecat-ai/pipecat-examples/tree/main/telnyx-chatbot) for full implementations - Learn about [Daily + Twilio SIP integration](./twilio-daily-sip) for advanced telephony scenarios - Check out [Daily PSTN integration](./daily-pstn) for direct phone number provisioning # Plivo WebSocket Integration Source: https://docs.pipecat.ai/pipecat/telephony/plivo-websockets.md Complete guide to using Plivo Media Streaming with Pipecat for dial-in and dial-out functionality ## Things you'll need - An active [Plivo](https://www.plivo.com) account with API credentials - One or more Plivo provisioned phone numbers - A public-facing server or tunneling service like [ngrok](https://ngrok.com/) - API keys for speech-to-text, text-to-speech, and LLM services Complete dial-in implementation using Plivo Media Streaming over WebSocket Outbound calling using Plivo Media Streaming with WebSocket transport ## Phone Number Setup You'll need Plivo phone numbers for both dial-in and dial-out functionality: - Visit [plivo.com](https://www.plivo.com) and purchase phone numbers - Ensure your numbers support Voice capabilities - Configure XML applications for dial-in numbers (covered below) ## Environment Setup Configure your environment variables for Plivo and AI services: ```shell .env PLIVO_AUTH_ID=... PLIVO_AUTH_TOKEN=... OPENAI_API_KEY=... DEEPGRAM_API_KEY=... CARTESIA_API_KEY=... ``` ## Dial-in Dial-in allows users to call your Plivo number and connect to your Pipecat bot via WebSocket Media Streaming. ### How It Works Here's the sequence of events when someone calls your Plivo number: 1. **Plivo receives an incoming call** to your phone number 2. **Plivo calls your XML server** with call details 3. **Your server responds with XML** that establishes a WebSocket connection 4. **Plivo opens a WebSocket** to your server with real-time audio 5. **Your bot processes the audio** using the Pipecat pipeline 6. **The bot responds with audio** sent back to Plivo over WebSocket 7. **Plivo plays the audio** to the caller in real-time ### Set up your XML server Your server handles Plivo webhooks and WebSocket connections for real-time audio streaming. When Plivo receives a call, your server: 1. **Receives the webhook** with call details (CallUUID, caller information) 2. **Returns XML** that establishes a WebSocket connection to your server 3. **Accepts the WebSocket** connection from Plivo 4. **Parses call data** from the WebSocket messages 5. **Runs the Pipecat bot** with the parsed call information The server orchestrates the real-time audio streaming between Plivo's telephony infrastructure and your Pipecat bot pipeline. ### Custom Data with Query Parameters You can pass custom data to your bot by adding query parameters to the WebSocket URL in your XML response. This works for both dial-in and dial-out scenarios: ```python server.py # Example: Adding custom parameters to the WebSocket URL xml = f""" wss://your-server.com/ws?user_id={user_id}&session_type=support&campaign_id=summer_sale """ ``` These parameters are accessible in your server code and can be used to customize your bot's behavior based on the specific call context. See the full FastAPI server code with XML generation and WebSocket handling ### Configure your Pipecat bot for dial-in Your bot receives the WebSocket connection and parses the call data to extract call information. The key components are: **WebSocket Parsing**: Pipecat's built-in parser extracts call data from Plivo's WebSocket messages: ```python bot.py from pipecat.runner.utils import parse_telephony_websocket async def run_bot(websocket: WebSocket): # Parse Plivo WebSocket data transport_type, call_data = await parse_telephony_websocket(websocket) # Extract call information stream_id = call_data["stream_id"] call_id = call_data["call_id"] # Plivo's CallUUID ``` **Transport Creation**: Configure the WebSocket transport with Plivo serialization: ```python bot.py # Create Plivo serializer with call details serializer = PlivoFrameSerializer( stream_id=stream_id, call_id=call_id, auth_id=os.getenv("PLIVO_AUTH_ID"), auth_token=os.getenv("PLIVO_AUTH_TOKEN"), ) # Configure WebSocket transport transport = FastAPIWebsocketTransport( websocket=websocket, params=FastAPIWebsocketParams( audio_in_enabled=True, audio_out_enabled=True, add_wav_header=False, serializer=serializer, ), ) # Your bot pipeline setup here... ``` See the full bot.py with WebSocket parsing, transport setup, and pipeline configuration ### Set up Plivo XML application Configure your Plivo phone number to use your XML application: 1. Go to the [Plivo Console](https://console.plivo.com) 2. Navigate to Voice → XML Applications 3. Create a new XML Application with your server's webhook URL 4. Assign the XML Application to your phone number ### Run the Example For local development, use [ngrok](https://ngrok.com/) to expose your local server: ```shell # Start your server python server.py # In another terminal, start ngrok ngrok http 8000 # Use the ngrok URL in your Plivo XML Application ``` See the full README with step-by-step setup, XML configuration, and testing ## Dial-out Dial-out allows your bot to initiate calls to phone numbers using Plivo's outbound calling capabilities with WebSocket Media Streaming. ### How It Works Here's the sequence of events for dial-out calls: 1. **Your application triggers a dial-out** (via API call or user action) 2. **Server initiates a Plivo call** to the target phone number 3. **Plivo establishes the call** and opens a WebSocket connection 4. **Your bot joins the WebSocket** and sets up the pipeline 5. **The recipient answers** and is connected to your bot 6. **The bot handles the conversation** with real-time audio streaming ### Set up your server for dial-out The dial-out server creates outbound calls and manages WebSocket connections. When you trigger a dial-out call, your server: 1. **Receives the dial-out request** with target phone number and parameters 2. **Creates a Plivo call** using the Voice API with XML that establishes WebSocket 3. **Accepts the WebSocket** connection from Plivo 4. **Parses call data** from the WebSocket messages 5. **Runs the Pipecat bot** with the call information See the full FastAPI server code with outbound call creation and WebSocket handling ### Passing Custom Data to Your Bot For dial-out calls, you often need to pass custom data to personalize the conversation—such as user information, campaign details, or context from your application. The dial-out example supports passing a `body` object that becomes available to your bot via `runner_args.body`. **Triggering a call with custom data:** ```bash curl -X POST https://your-server.com/start \ -H "Content-Type: application/json" \ -d '{ "phone_number": "+1234567890", "body": { "user": { "id": "user123", "firstName": "John", "lastName": "Doe", "accountType": "premium" }, "campaign": "renewal_reminder", "context": "subscription expiring in 3 days" } }' ``` **How it works:** 1. The `/start` endpoint receives the request with `phone_number` and optional `body` 2. The `body` is base64-encoded and passed through the TeXML URL chain 3. When the WebSocket connects, the server decodes the body and passes it to your bot via `runner_args.body` 4. Your bot accesses the custom data to personalize the conversation **Accessing custom data in your bot:** ```python bot.py async def bot(runner_args: RunnerArguments): body_data = runner_args.body or {} first_name = body_data.get("user", {}).get("firstName", "there") # Use first_name to personalize greetings, prompts, etc. ``` This approach works consistently in both local development and Pipecat Cloud production deployments. ### Configure your Pipecat bot for dial-out The dial-out bot configuration is similar to dial-in, with Plivo automatically providing call information. **Call Information**: Extract call details from Plivo's WebSocket messages: ```python bot.py # Parse WebSocket data (same as dial-in) transport_type, call_data = await parse_telephony_websocket(websocket) # Extract call information stream_id = call_data["stream_id"] call_id = call_data["call_id"] # Plivo's CallUUID # Access custom data from runner args body_data = runner_args.body or {} user_name = body_data.get("user", {}).get("firstName", "there") # Customize bot behavior for outbound calls greeting = f"Hi {user_name}! This is an automated call. How can I help you today?" ``` **Transport Configuration**: Same transport setup as dial-in: ```python bot.py # Create Plivo serializer serializer = PlivoFrameSerializer( stream_id=stream_id, call_id=call_id, auth_id=os.getenv("PLIVO_AUTH_ID"), auth_token=os.getenv("PLIVO_AUTH_TOKEN"), ) # Configure transport transport = FastAPIWebsocketTransport( websocket=websocket, params=FastAPIWebsocketParams( audio_in_enabled=True, audio_out_enabled=True, add_wav_header=False, serializer=serializer, ), ) ``` See the full bot.py with outbound call handling and call information usage ### Run the Example To test dial-out functionality: 1. **Start your server**: Run your FastAPI server 2. **Trigger a call**: Make an API request to start an outbound call 3. **Answer your phone**: The bot will call the specified number 4. **Talk to your bot**: Have a conversation with your AI agent See the full README with step-by-step setup, API usage, and outbound call configuration ## Key Features ### Audio Format and Sample Rate Plivo Media Streaming uses 8kHz mono audio with μ-law encoding. Configure your pipeline accordingly: ```python worker = PipelineWorker( pipeline, params=PipelineParams( audio_in_sample_rate=8000, audio_out_sample_rate=8000, ), ) ``` ### Automatic Call Termination `PlivoFrameSerializer` ends the call when your pipeline ends. This is on by default (`auto_hang_up=True`) and requires `call_id`, `auth_id`, and `auth_token` — the serializer raises a `ValueError` at construction if any are missing: ```python serializer = PlivoFrameSerializer( stream_id=stream_id, call_id=call_id, auth_id=os.getenv("PLIVO_AUTH_ID"), auth_token=os.getenv("PLIVO_AUTH_TOKEN"), ) ``` To leave the call up after your pipeline ends, disable it with `params=PlivoFrameSerializer.InputParams(auto_hang_up=False)`. The `keepCallAlive="true"` attribute in your Plivo XML keeps the call from being torn down when the stream ends, so the call stays up until something else ends it. For the full parameter list, see the [`PlivoFrameSerializer` reference](/api-reference/server/services/serializers/plivo). ## Deployment ### Local Development Use [ngrok](https://ngrok.com/) to expose your local server for testing: ```shell python server.py ngrok http 8000 # Use the ngrok URL in your Plivo XML Application ``` ### Pipecat Cloud Deployment For production deployment without managing your own infrastructure, use Pipecat Cloud: Deploy Plivo WebSocket bots with automatic scaling and managed infrastructure ### Self-Hosted Production Deployment For fully self-hosted production deployment, ensure your servers are: - Publicly accessible with HTTPS - Able to handle concurrent WebSocket connections - Properly configured with Plivo API credentials - Implementing proper error handling and logging ## Next Steps - Explore the [complete examples](https://github.com/pipecat-ai/pipecat-examples/tree/main/plivo-chatbot) for full implementations - Learn about [Daily + Twilio SIP integration](./twilio-daily-sip) for advanced telephony scenarios - Check out [Daily PSTN integration](./daily-pstn) for direct phone number provisioning # Exotel WebSocket Integration Source: https://docs.pipecat.ai/pipecat/telephony/exotel-websockets.md Complete guide to using Exotel Voice Streaming with Pipecat for dial-in and dial-out functionality ## Things you'll need - An active [Exotel](https://exotel.com) account with voice streaming enabled - One or more Exotel provisioned phone numbers - A public-facing server or tunneling service like [ngrok](https://ngrok.com/) (for dial-out only) - API keys for speech-to-text, text-to-speech, and LLM services Complete dial-in implementation using Exotel Voice Streaming over WebSocket Outbound calling using Exotel Voice Streaming with WebSocket transport ## Phone Number Setup You'll need Exotel phone numbers for both dial-in and dial-out functionality: - Visit [exotel.com](https://exotel.com) and purchase phone numbers - Complete KYC verification if required - Ensure voice streaming is enabled on your account (contact Exotel support if needed) ## Environment Setup Configure your environment variables for Exotel and AI services: ```shell .env EXOTEL_ACCOUNT_SID=... EXOTEL_API_KEY=... EXOTEL_API_TOKEN=... OPENAI_API_KEY=... DEEPGRAM_API_KEY=... CARTESIA_API_KEY=... ``` ## Dial-in Dial-in allows users to call your Exotel number and connect to your Pipecat bot via WebSocket Voice Streaming. Unlike other providers, Exotel automatically provides caller information (to/from numbers) in the WebSocket messages, so no custom server is needed for basic dial-in functionality. ### How It Works Here's the sequence of events when someone calls your Exotel number: 1. **Exotel receives an incoming call** to your phone number 2. **Exotel executes your App Bazaar flow** which includes a Voicebot applet 3. **Exotel opens a WebSocket** to your server with real-time audio and call metadata 4. **Your bot processes the audio** using the Pipecat pipeline 5. **The bot responds with audio** sent back to Exotel over WebSocket 6. **Exotel plays the audio** to the caller in real-time ### Set up your App Bazaar flow Exotel uses App Bazaar to configure call flows. For dial-in, you create an app with a Voicebot applet that establishes a WebSocket connection directly to your bot: 1. **Navigate to App Bazaar** in your Exotel dashboard 2. **Create a new app** or edit an existing one 3. **Add a Voicebot applet** (not "Stream" or "Passthru") 4. **Configure the WebSocket URL**: `wss://your-server.com/ws` 5. **Add a Hangup applet** at the end to properly terminate calls Your flow should look like: `Call Start → [Voicebot Applet] → [Hangup Applet]` ### Personalizing Your Bot with Caller Information Exotel automatically includes caller information (to/from numbers) in the WebSocket messages. You can use the caller's phone number to: - Look up customer information in your database - Personalize greetings and responses based on caller identity - Route calls to different bot behaviors (e.g., VIP handling, support vs sales) - Implement custom business logic based on the caller The example bot demonstrates how to extract caller information from Exotel's WebSocket messages and use it to personalize the conversation. ### Configure your Pipecat bot for dial-in Your bot receives the WebSocket connection and automatically gets call information from Exotel's WebSocket messages. The key components are: **WebSocket Parsing**: Pipecat's built-in parser extracts call data from Exotel's WebSocket messages: ```python bot.py from pipecat.runner.utils import parse_telephony_websocket async def run_bot(websocket: WebSocket): # Parse Exotel WebSocket data transport_type, call_data = await parse_telephony_websocket(websocket) # Extract call information (automatically provided by Exotel) # Data format: { # "stream_id": str, # "call_id": str, # "account_sid": str, # "from": str, # "to": str, # } stream_id = call_data["stream_id"] call_id = call_data["call_id"] # Exotel's CallSid account_sid = call_data["account_sid"] from_number = call_data["from"] # Caller's number to_number = call_data["to"] # Your Exotel number ``` **Transport Creation**: Configure the WebSocket transport with Exotel serialization: ```python bot.py # Create Exotel serializer with call details serializer = ExotelFrameSerializer( stream_id=stream_id, call_id=call_id, account_sid=account_sid, api_key=os.getenv("EXOTEL_API_KEY"), api_token=os.getenv("EXOTEL_API_TOKEN"), ) # Configure WebSocket transport transport = FastAPIWebsocketTransport( websocket=websocket, params=FastAPIWebsocketParams( audio_in_enabled=True, audio_out_enabled=True, add_wav_header=False, serializer=serializer, ), ) # Your bot pipeline setup here... ``` See the full bot.py with WebSocket parsing, transport setup, and pipeline configuration ### Set up Exotel phone number Configure your Exotel phone number to use your App Bazaar flow: 1. Go to the [Exotel Dashboard](https://exotel.com/) 2. Navigate to ExoPhones 3. Find your phone number and click the edit icon 4. Under "App", select your App Bazaar flow 5. Save the configuration ### Run the Example For dial-in, you can run your bot directly without a separate server: ```shell # Start your bot directly python bot.py # For local development, use ngrok to expose your WebSocket ngrok http 8000 # Use the ngrok WebSocket URL (wss://abc123.ngrok.io/ws) in your App Bazaar flow ``` See the full README with step-by-step setup, App Bazaar configuration, and testing ## Dial-out Dial-out allows your bot to initiate calls to phone numbers using Exotel's outbound calling capabilities with WebSocket Voice Streaming. ### How It Works Here's the sequence of events for dial-out calls: 1. **Your application triggers a dial-out** (via API call or user action) 2. **Server initiates an Exotel call** using the Voice API 3. **Exotel establishes the call** and opens a WebSocket connection 4. **Your bot joins the WebSocket** and sets up the pipeline 5. **The recipient answers** and is connected to your bot 6. **The bot handles the conversation** with real-time audio streaming ### Set up your server for dial-out The dial-out server creates outbound calls and manages WebSocket connections. When you trigger a dial-out call, your server: 1. **Receives the dial-out request** with target phone number and parameters 2. **Creates an Exotel call** using the Voice API with a callback URL 3. **Accepts the WebSocket** connection from Exotel 4. **Parses call data** from the WebSocket messages 5. **Runs the Pipecat bot** with the call information See the full FastAPI server code with outbound call creation and WebSocket handling ### Configure your Pipecat bot for dial-out The dial-out bot configuration is similar to dial-in, with Exotel automatically providing call information. **Call Information**: Extract call details from Exotel's WebSocket messages: ```python bot.py # Parse WebSocket data (same as dial-in) transport_type, call_data = await parse_telephony_websocket(websocket) # Extract call information stream_id = call_data["stream_id"] call_id = call_data["call_id"] account_sid = call_data["account_sid"] from_number = call_data["from"] # Your Exotel number to_number = call_data["to"] # Target number you're calling # Customize bot behavior for outbound calls greeting = f"Hi! This is an automated call from {from_number}. How are you today?" ``` **Custom Data Limitations for Dial-out**: Currently, there are limitations with passing custom data through query parameters in Exotel's WebSocket URLs. While the platform theoretically supports query parameters, they may be stripped during call processing. This is an ongoing limitation that may be addressed in future Exotel updates. If you need to pass custom data for dial-out, consider these alternatives: - Use different WebSocket endpoints for different call types - Configure multiple App Bazaar flows for different scenarios - Handle customization based on the called number (available in WebSocket messages) See the full bot.py with outbound call handling and call information usage ### Run the Example To test dial-out functionality: 1. **Start your server**: Run your FastAPI server 2. **Trigger a call**: Make an API request to start an outbound call 3. **Answer your phone**: The bot will call the specified number 4. **Talk to your bot**: Have a conversation with your AI agent See the full README with step-by-step setup, API usage, and outbound call configuration ## Key Features ### Audio Format and Sample Rate Exotel Voice Streaming uses 8kHz mono audio with 16-bit PCM encoding. Configure your pipeline accordingly: ```python worker = PipelineWorker( pipeline, params=PipelineParams( audio_in_sample_rate=8000, audio_out_sample_rate=8000, ), ) ``` ### Built-in Call Information Like Telnyx, Exotel automatically includes comprehensive call information (to/from numbers, account details) in the WebSocket messages, eliminating the need for custom webhook servers in basic dial-in scenarios. ## Deployment ### Local Development Use [ngrok](https://ngrok.com/) to expose your local server for testing: ```shell python server.py # For dial-out # OR python bot.py # For dial-in ngrok http 8000 # Use the ngrok URL in your App Bazaar Voicebot applet ``` ### Pipecat Cloud Deployment For production deployment without managing your own infrastructure, use Pipecat Cloud: Deploy Exotel WebSocket bots with automatic scaling and managed infrastructure ### Self-Hosted Production Deployment For fully self-hosted production deployment, ensure your servers are: - Publicly accessible with HTTPS - Able to handle concurrent WebSocket connections - Properly configured with Exotel API credentials - Implementing proper error handling and logging ## Next Steps - Explore the [complete examples](https://github.com/pipecat-ai/pipecat-examples/tree/main/exotel-chatbot) for full implementations - Learn about [Daily + Twilio SIP integration](./twilio-daily-sip) for advanced telephony scenarios - Check out [Daily PSTN integration](./daily-pstn) for direct phone number provisioning # Pipecat Deployment Overview Source: https://docs.pipecat.ai/pipecat/deployment/overview.md Run Pipecat bots locally and in production: process model, hosting patterns, and choosing a deployment approach. A Pipecat bot is a Python process. Running one locally is straightforward; running the right number of them on demand for real users is a separate problem, and the section that follows is a map for both. ## Two questions, not one It's helpful to separate two things that often get tangled together: - **Running a bot during development** — getting a bot file to start up, accept a connection, and let you talk to it. Pipecat ships its own server for this; you usually don't need to write one. See [Running Bots Locally](./running-bots-locally). - **Hosting bots in production** — deciding what serves session-start requests at scale, how bot processes get spawned, and how the operational surface (capacity, lifecycle, observability) is handled. This is genuinely several distinct problems and the right answer depends on your bot, your traffic, and what you already know how to operate. See [Running Bots in Production](./running-bots-in-production). ## What you'll typically build Most Pipecat deployments end up with some version of these pieces: - **A bot** — your `bot.py`. A Python function (`async def bot(runner_args)`) that joins a media session, runs your pipeline, and exits when the session ends. - **Something that serves session-start requests** — an HTTP service that receives "start a session" requests from your client, sets up the media transport, and causes a bot process to come into existence ready to accept the user. In development this is the [built-in runner](./running-bots-locally). In production this is a hand-rolled dispatcher, a managed agent runtime, or [Pipecat Cloud](/pipecat-cloud/introduction) — the development runner is a local tool and isn't built for production. See [Running Bots in Production](./running-bots-in-production) for the tradeoffs. - **A media transport** — WebRTC, WebSockets, SIP, or whatever the user connects through. Sometimes a service you host, more often a third-party provider (Daily, Twilio, etc.). The development runner deliberately mimics Pipecat Cloud's session-start API (`POST /start`, `/sessions/{id}/...`), so a bot file that runs locally can run against either with no code changes — and an HTTP service that fronts your fleet in production can offer the same shape if you want clients to be portable. ## Where to go from here If you're starting fresh: The bot entry point, the built-in runner, and the supported transports. What plays the runner's role at scale, and how to think about the choices. If you already know the basics and want concrete patterns: Dispatcher calls a cloud machines API to spawn a fresh VM per session. Pre-allocated resources and a worker pool on a single host, replenished on use. Hand the bot lifecycle off to a runtime that owns scaling and dispatch. Webhook-driven dispatch, SIP gotchas, and where telephony differs. If you want a worked example on a specific platform: A worked vm-per-session deployment. Containerized FastAPI + Pipecat with optional GPU functions. Serverless CPU/GPU infra for Pipecat agents. ## A note on Pipecat Cloud [Pipecat Cloud](/pipecat-cloud/introduction) is Daily's managed service for running Pipecat bots. It exists because the operational surface in [running bots in production](./running-bots-in-production) is genuinely large, and many teams would rather not build it themselves — but it's one option among several. The pages in this section are intended to be useful whether you build the infrastructure yourself, use Pipecat Cloud, or end up with a hybrid. Some patterns described here come directly from how Pipecat Cloud is built. Where that's true the page calls it out, usually with the caveat that PCC made a specific choice for reasons (multi-tenant isolation, billing, SLA) that may not apply to a single-tenant self-hosted deployment. # Running Bots Locally Source: https://docs.pipecat.ai/pipecat/deployment/running-bots-locally.md The bot entry point, the built-in development runner, and the transports it supports. Pipecat ships a built-in development runner (`pipecat.runner.run`) that handles the server-side glue most bots need during development: creating Daily rooms, accepting WebRTC offers, terminating telephony WebSockets, and serving a prebuilt UI to talk to your bot. You write the bot; the runner does everything around it. This page covers the canonical shape: a bot file with a single async entry point, run via `pipecat.runner.run.main()`. The development runner is a **local development tool** — it isn't built or supported for production use. See [The runner is a development tool](#the-runner-is-a-development-tool-not-a-production-server) below, and [Running Bots in Production](./running-bots-in-production) for what to reach for instead. ## The bot entry point Every bot is a Python file with an async `bot()` function that takes a `RunnerArguments`. The runner calls this function once per session, passing in everything the bot needs to connect (room URL, token, WebRTC connection, WebSocket, etc.) as fields on the argument object. ```python bot.py from pipecat.runner.types import RunnerArguments async def bot(runner_args: RunnerArguments): """Entry point. The runner calls this once per session.""" # ... configure the transport from runner_args, build a pipeline, run it. ... if __name__ == "__main__": from pipecat.runner.run import main main() ``` The bot is encapsulated: it doesn't know how the request to start a session arrived, who authenticated it, or whether it's running locally or in production. Everything it needs to do its job is on `runner_args`. This is the property that makes the same bot file portable across the development runner, Pipecat Cloud, and most production self-hosting setups. ## Choosing a transport `RunnerArguments` has transport-specific subclasses; the runner instantiates the right one based on how the session was initiated. A typical multi-transport bot pattern-matches: ```python from pipecat.runner.types import ( DailyRunnerArguments, RunnerArguments, SmallWebRTCRunnerArguments, WebSocketRunnerArguments, ) async def bot(runner_args: RunnerArguments): match runner_args: case DailyRunnerArguments(): transport = DailyTransport(runner_args.room_url, runner_args.token, "Bot", ...) case SmallWebRTCRunnerArguments(): transport = SmallWebRTCTransport(webrtc_connection=runner_args.webrtc_connection, ...) case WebSocketRunnerArguments(): transport = FastAPIWebsocketTransport(websocket=runner_args.websocket, ...) await run_pipeline(transport) ``` A bot can support a single transport or many. In both the local development runner and Pipecat Cloud, the client chooses how to start the session by sending a `transport` value in the `/start` request (`webrtc`, `daily`, `twilio`, `telnyx`, `plivo`, `exotel`, or `websocket`); `-t/--transport` only restricts the local runner to one transport and sets that default. For working files demonstrating each transport, see [`examples/runner-examples/`](https://github.com/pipecat-ai/pipecat-examples/tree/main/runner-examples) — the `01-` through `04-` files step from a single-transport bot through to a factory-driven multi-transport setup. ## Installing and running ```bash uv add "pipecat-ai[runner]" ``` Run the bot file directly: ```bash uv run bot.py ``` By default, the development runner starts a local server on `localhost:7860`, serves the prebuilt client UI at `/client`, and lets clients start sessions through `POST /start`. Clients can request the transport in that start request; `-t/--transport` is available when you want to restrict the local runner to one transport. For the full `/start` request shape, transport-specific behavior, and CLI flags, see the [Development Runner guide](/api-reference/server/utilities/runner/guide). ## What the runner does for you Behind `uv run bot.py` the runner is doing a fair amount of work depending on the transport the client requests: - **WebRTC (`smallwebrtc`)** — mounts a prebuilt UI at `/client`, accepts `POST /api/offer` (with ICE candidates via `PATCH /api/offer`), and bridges the resulting `SmallWebRTCConnection` to your `bot()` function. Also exposes a `POST /start` endpoint that returns a `sessionId`, mimicking Pipecat Cloud's start API. - **Daily** — calls Daily's REST API to create a room and issue tokens, then either redirects the browser to the room (via `GET /daily`) or returns the room URL + token via `POST /start`. - **Telephony** — returns the carrier-specific XML stub (TwiML for Twilio, the equivalent for Telnyx/Plivo/Exotel) on `POST /`, then accepts the bidirectional media WebSocket at `/ws` and hands it to your bot wrapped in a `WebSocketRunnerArguments`. - **Plain WebSocket** — for non-telephony clients (e.g. browser apps using protobuf framing). `POST /start` with `"transport": "websocket"` returns a `wsUrl` for the `/ws-client` endpoint; the bot receives a `WebSocketRunnerArguments` with `transport_type="websocket"`. - **Daily PSTN dial-in** (`--dialin`) — handles Daily's pinless dial-in webhook, creates a SIP-enabled room, and dispatches the bot with full dial-in context. `GET /` redirects to the prebuilt client UI at `/client/`, and `GET /status` reports which transports the running instance accepts. The `POST /start` and `/sessions/{id}/...` endpoints exposed by the runner are deliberately shaped the same way as [Pipecat Cloud](/pipecat-cloud/introduction)'s session API. That means a client built against the development runner works against PCC unchanged, and a custom production dispatcher can offer the same contract if you want clients to remain portable. ## Adding your own routes The runner exports its FastAPI app as a module-level attribute, so you can add custom routes before calling `main()`: ```python from pipecat.runner.run import app, main @app.get("/healthz") async def healthz(): return {"status": "ok"} if __name__ == "__main__": main() ``` This is the simplest way to extend the runner — for things like health checks, internal status endpoints, or accepting metadata alongside session-start requests — without forking the dispatcher itself. ## The runner is a development tool, not a production server The development runner is exactly that — a tool for local development. It's deliberately simple: its `POST /start` endpoint accepts session-start requests from anyone, some transport webhooks (like Daily's dial-in webhook) are likewise unauthenticated, and there's no rate limiting or backpressure anywhere. It's also a single process with no lifecycle management. Those are the right tradeoffs on your own machine, and the wrong ones on a public address, where a single request can spin up a bot and create paid rooms on your account. For anything real, put a proper dispatcher or a managed runtime in front of your bots instead of exposing the runner. The next page, [Running Bots in Production](./running-bots-in-production), walks through those options — including how to package the bot into a container image — and, if you only need something reachable for a prototype or alpha test, the minimum you'd want in front of the runner and where that stops being enough. # Running Bots in Production Source: https://docs.pipecat.ai/pipecat/deployment/running-bots-in-production.md What plays the role of the development runner when you run bots in production, and how to think about the choices. The [development runner](./running-bots-locally) gets you a working bot you can talk to on your own machine. Production raises a separate question: _what serves the session-start request, decides where the bot process runs, and manages its lifecycle once it's running?_ **The development runner (`pipecat.runner.run`) is not built for production, and is not supported there.** It's a local development tool. Its `POST /start` endpoint accepts session-start requests from anyone, and some transport webhooks (like Daily's dial-in webhook) are likewise unauthenticated — so a single request can spawn a bot and create paid resources (rooms) on your account. It has no rate limiting or backpressure, and it's a single process with no lifecycle management. Running bots in production means choosing one of the options below — not exposing the runner to real traffic. If you're only prototyping, [Exposing the runner for prototyping](#exposing-the-runner-for-prototyping) covers the minimum you'd want in front of it, and where that stops being enough. There isn't one right answer for the production version. It depends on traffic shape, isolation requirements, what operational surface your team already runs, and how much of the dispatcher you want to own. This page walks through the answer space at one paragraph per option, links out to concrete patterns, and ends with the concerns that self-hosting puts on you. ## What plays the runner's role The two options below can roughly be summarized as "run everything yourself" and "let a managed runtime do it for you". Both are presented together here because the decision is the same one ("what serves session-start and runs the bot?"), and most teams evaluating self-hosting are also weighing whether they want to be in that business at all. ### Option 1 — Build your own dispatcher You write the HTTP service that receives session-start requests, and you decide how bot processes get into existence. This is the most flexible option and the most work. The two patterns most teams reach for here are: - **[VM per session](./patterns/vm-per-session)** — your dispatcher calls a cloud provider's machines API (Fly.io, AWS, GCP) to spin up a fresh VM with the room/token baked into its entrypoint. Strong isolation, easy scale-out, no warm capacity to manage. Pays cold-start latency on every session. - **[Warm pool with subprocess workers](./patterns/warm-pool-subprocess)** — pre-allocate transport resources (e.g. Daily rooms) and a pool of bot subprocesses on one or more long-lived hosts. Replenish on use. Very low session-start latency; constrained by what one host can hold. These aren't the only shapes. Some teams build longer-lived orchestrated fleets on Kubernetes (with HPA or KEDA scaling on a custom session-count metric), or use a serverless platform's function-per-invocation model. The right shape is the one whose latency / cost / isolation profile matches the bot you're shipping and the traffic you're seeing. ### Option 2 — Use a managed agent runtime This option steps out of self-hosting: a managed platform runs the bot process for you. You supply the bot file (and usually a thin proxy that forwards session-start traffic), and the platform owns the runtime, pool, and lifecycle. This minimizes the operational surface you carry but trades for vendor lock-in and whatever feature ceiling the runtime has. See [Managed Agent Runtime](./patterns/managed-runtime). [Pipecat Cloud](/pipecat-cloud/introduction) is one example: Daily's managed runtime, purpose-built for Pipecat. The development runner's session-start API is deliberately shaped the same way as PCC's (`POST /start`, `/sessions/{id}/...`), which means a bot file and client built against the runner work against PCC unchanged. Other managed runtimes — like AWS Bedrock AgentCore — are the same shape, with their own platform-specific tradeoffs. ## Exposing the runner for prototyping Sometimes you need something reachable before you've built either option above — a demo, an alpha test, a prototype a handful of people will hit. If you're going to put the development runner on a public address at all, treat it as **prototyping-only**. We recommend putting a reverse proxy in front of it that adds, at a minimum: - **Authentication on `POST /start`** — the runner accepts start requests from anyone. A shared secret or bearer token checked at the proxy keeps strangers from spawning bots (and paid rooms) on your account. - **Rate limiting** — the runner has no backpressure. A per-IP and global request cap at the proxy bounds both load and cost if the address is discovered. - **TLS termination** — the runner speaks plain HTTP. Note that the runner binds to `localhost` by default; reaching it from anywhere else (including from outside a container) means starting it with `--host 0.0.0.0`, which is exactly the exposure this section is about — so keep the reverse proxy, not the runner, as the public entry point. Notably, none of the above addresses the following — which is why this stays prototyping-only: - **The transport webhooks have a different caller than `POST /start`.** Endpoints like the Daily dial-in webhook are invoked by the transport provider, not your app, so a single proxy secret doesn't fit them — they'd need provider-appropriate verification (signature checks, IP allowlists) that the runner's dial-in webhook doesn't perform. Enabling a telephony transport exposes these endpoints too. - **None of this addresses lifecycle, capacity, or isolation.** You still have a single point of failure, no graceful drain, and no way to scale past one host. If you find yourself building this out, that's the signal you've outgrown the runner — move to Option 1 or Option 2. ## Self-hosting considerations If you build your own dispatcher (Option 1), these concerns are yours to handle. A managed runtime (Option 2) handles most of them for you — though a couple, like bot images and region selection, apply either way. None have a single right answer; the goal here is just to name them so they don't surprise you. ### Request authentication Whatever serves session-start traffic in production needs some form of authentication, both to keep strangers from spinning up bots on your dime and to identify the user the bot is being started for. The shape varies: signed JWTs from your app server, bearer tokens, mutual-TLS, etc. If you're using a managed runtime this is usually handled by the runtime; if you're rolling your own dispatcher you'll need to add it. (This is the same gap that makes the development runner unsafe to expose — see the warning above.) ### Secret delivery Bots typically need a handful of API keys (Daily, your TTS/STT/LLM providers, your own service tokens). In development these come from a `.env` file. In production they should arrive through whatever mechanism your runtime supports — environment variables injected by a secrets manager, mounted files, an external secrets service. Avoid baking keys into images. ### Bot images and model discipline In production the bot usually runs as a **per-session process**, not the always-on development server. Your dispatcher passes the room URL and token into the container's entrypoint (VM per session) or spawns the bot as a subprocess with those arguments (warm pool); the process runs one conversation and exits. The image's job is to be a ready-to-run bot — bot file, dependencies, and any cached model weights — not to serve session-start itself. Pipeline models (Silero VAD, any local TTS/STT/turn-detection models) want to be cached at build time, not download time, so the first session doesn't pay a download. A minimal bot image bakes them in: ```dockerfile FROM python:3.11-slim-bookworm WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY bot.py . # Bake model weights (e.g. Silero VAD) into the image so first-run is fast. RUN python -c "import torch; torch.hub.load('snakers4/silero-vad', 'silero_vad', force_reload=True)" # Pre-download NLTK data used for sentence aggregation. RUN python -m nltk.downloader punkt_tab -d /usr/local/share/nltk_data && \ python -c "import nltk; nltk.data.find('tokenizers/punkt_tab')" ``` The launch command is deliberately left off — it depends on the pattern, since the dispatcher supplies the room and token when it starts the container or subprocess. For anything substantial in local model weights, decide between baking them into the image (faster cold start, larger image) and mounting them from a network volume (smaller image, slower first-touch). Build dispatcher and bot images separately if the two have meaningfully different dependencies. Pipecat Cloud's [base image](https://github.com/daily-co/pipecat-cloud-images) (`dailyco/pipecat-base`) is what most of the [examples](https://github.com/pipecat-ai/pipecat-examples) build from; it runs your bot as `app.py` and provides a sensible runtime environment. It's a reasonable starting point for self-hosted bot images too, and you can use it without using PCC. ### Session lifecycle This is arguably the most distinctive operational fact about hosting bots and worth dwelling on. Most production HTTP workloads are short-lived: a request comes in, you do some work, you send a response, the worker is free again. Bot sessions are the opposite. A single session ties up a process (often a whole container or VM) for the entire duration of a conversation — typically minutes, sometimes hours — and that work can't be paused, migrated, or interrupted without dropping the user mid-sentence. That shape has consequences that ripple through every other operational choice: - **Autoscaling can't be reactive.** Traditional HPA-style "CPU is high, add more pods" works because new pods can immediately absorb new requests. With long-lived sessions, scale-up has to be predictive (anticipate demand, warm capacity ahead of it) and scale-down has to be patient (you can't terminate a pod that's mid-call; you have to stop sending it new sessions and wait for the existing ones to finish naturally). KEDA scaling on a custom "active sessions" metric is a closer fit than HPA on CPU, but neither solves the "wait for sessions to drain" problem. - **Pod and VM shutdowns need long graceful periods.** Kubernetes' default `terminationGracePeriodSeconds` is 30 seconds. For a typical bot host that might have sessions running for 10+ minutes, you need to either crank that up dramatically or accept that deploys will drop in-flight calls. Pair this with a `preStop` hook (or equivalent) that stops the host from accepting new sessions so the existing ones can drain. - **Readiness gating.** When do you tell the client "your bot is ready"? Returning a join URL the moment you've issued the dispatch is fast but lets the client try to connect before the bot has joined. Waiting until the bot has actually joined the transport is more reliable but adds latency to session start. Most teams pick somewhere in the middle: return optimistically and let the client poll or wait for a "bot joined" event from the transport provider. - **Crash handling.** Bots crash. Decide what the client sees, whether you retry, and whether the user-facing transport (Daily room, Twilio call) is reused or recreated. The right answer depends on how long sessions tend to last and how the user perceives a mid-call failure. - **Graceful drain on deploy.** Concretely, this usually looks like: mark the old version as not-accepting-new-sessions, redirect new traffic to the new version, wait for the old version's sessions to finish (with a deadline), then terminate. Without this you drop in-flight calls every deploy. With it, you accept that "rolling deploys" can take as long as your longest session. ### Observability Bots are ephemeral and often distributed across hosts. Correlating events across "the dispatcher that accepted the request, the bot that ran the session, the transport provider that carried the audio" requires designing for it. A single session ID propagated through every log line (and ideally through your transport provider's session identifier as well) is the load-bearing piece. Beyond that, the metrics most teams find useful for voice bots are active-session count, dispatcher latency to "client ready", session duration, and pipeline-level metrics from the bot itself (Pipecat's `PipelineParams(enable_metrics=True, enable_usage_metrics=True)` gives you a starting point). ### Networking and placement Most production deployments hit one or more of: - **Region selection for latency** — voice is latency-sensitive. Bots placed far from users introduce audible delay even when individual services are fast. - **Service quotas** — your cloud provider's machines API, your transport provider's rate limits, and the LLM/TTS providers your pipeline calls all have ceilings you'll find at scale. - **Telephony-specific networking** — SIP and PSTN setups can carry their own constraints depending on the carrier. See [Telephony in Production](./telephony-in-production). ## Where to go next Dispatcher calls a cloud machines API to spawn a fresh VM per session. Pre-allocated resources and a worker pool on a single host, replenished on use. Hand the bot lifecycle off to a runtime that owns scaling and dispatch. Webhook-driven dispatch, SIP gotchas, and where telephony differs. # VM per Session Source: https://docs.pipecat.ai/pipecat/deployment/patterns/vm-per-session.md A dispatcher spawns a fresh cloud VM for each session via the provider's machines API. A small HTTP service ("the dispatcher") receives session-start requests from clients. For each request it calls a cloud provider's machines API to spawn a fresh VM running your bot image, with the room URL and token passed in via the entrypoint command. The VM exits when the session ends, the provider tears it down, and you pay for exactly the time the bot was running. This is the shape used in the [Fly.io example](https://github.com/pipecat-ai/pipecat-examples/tree/main/deployment/flyio-example). The pattern generalizes to any cloud with a machines API: AWS Fargate, Google Cloud Run jobs, Azure Container Instances, Fly Machines, and so on. ## When it fits - You want strong per-session isolation — each bot runs in its own VM, so misbehavior in one session can't affect another. - Your traffic is bursty enough that maintaining a warm pool would mostly burn money on idle capacity. - You're comfortable with cold-start latency in the seconds range on each new session (mitigated by image size discipline and the provider's start-up speed). - You don't want to operate a long-running fleet — the provider's machines API is the only thing you talk to. ## When it doesn't - Cold-start latency dominates your UX. If users expect a bot to answer within a second of pressing "call", per-session VMs are usually too slow without significant work (small images, warm reserves, optimistic client connect + poll). - Your concurrency exceeds the provider's per-account API rate limits or instance-count quotas. These are routinely raised on request but worth checking before committing. - Your bots are extremely lightweight — at some point the VM-per-session overhead dominates the actual work. ## How it usually looks The dispatcher: 1. Receives `POST /start` (or whatever your equivalent is). 2. Authenticates the request. 3. Creates whatever transport-side resources the bot will need (e.g. a Daily room and tokens). 4. Calls the cloud provider's machines API with the bot image and a command that passes `--room-url` / `--token` / etc. into the bot. 5. Waits for the machine to enter a "started" state (most providers expose a synchronous wait endpoint). 6. Returns the join URL to the client. The bot is a normal Pipecat bot file — its entrypoint parses the room URL and token, builds the transport, and runs the pipeline. The dispatcher does not need to know anything about Pipecat internals. ## Tradeoffs worth being explicit about - **Cold-start vs. cost.** No warm capacity means low idle cost and slow first-byte. Mitigations: keep images small, pre-cache pipeline models at build time, optimistically return the join URL once dispatch has been requested (and let the client poll for "bot ready"). - **Isolation vs. response time.** Fresh VM per session is the cleanest possible isolation model, but every session pays the cloud provider's full instance-startup latency on the way in. There's no way to amortize that across sessions without giving up the per-session isolation that's the whole point. - **Rate limits and quotas.** Your machines-API quota becomes your real concurrency ceiling. Worth knowing in advance. - **Image discipline.** Image size directly affects cold-start time. Multi-stage builds, baking only what the bot needs at runtime, and keeping VAD/STT models cached are all material here. ## See also - [Fly.io worked example](../platforms/fly) — end-to-end walkthrough of the pattern on Fly Machines. - [Modal](../platforms/modal) — similar pattern on Modal's function infrastructure. - [Cerebrium](../platforms/cerebrium) — similar with GPU support if your pipeline needs it. # Warm Pool with Subprocess Workers Source: https://docs.pipecat.ai/pipecat/deployment/patterns/warm-pool-subprocess.md Pre-allocate transport resources and bot subprocesses on a long-lived host; replenish on use. A long-lived dispatcher pre-creates the things that take time to set up — typically Daily rooms with tokens — and keeps them in an in-memory pool. When a session-start request arrives, the dispatcher pops a pre-warmed entry out of the pool, spawns a bot subprocess against it, and triggers a background task to replenish the pool back to its target size. The bot subprocess runs to completion in its own process; when it exits, the dispatcher cleans up the room. This is the shape used in the [`instant-voice`](https://github.com/pipecat-ai/pipecat-examples/tree/main/instant-voice) example. See `server/src/server.py` for a working `RoomPool` plus `BotManager` implementation with replenish-on-pop and process-exit cleanup. ## When it fits - You care about session-start latency more than anything else. Popping from a pre-warmed pool and forking a subprocess on a host that already has the bot image loaded is fast. - Your traffic shape is steady enough that maintaining warm capacity isn't wasteful. - Your bot fits comfortably in a subprocess on a single host, and you're willing to scale across hosts by running multiple instances of the dispatcher. ## When it doesn't - Your concurrency outgrows a single host. Subprocess fan-out is bounded by the host's CPU / memory / file descriptors / `pipe(2)` limits. Run this pattern across N hosts and you'll need a layer in front to route to the right one — at which point you may want a different shape. - You need strong per-session isolation (e.g. you're hosting bots for multiple customers and a crash in one must absolutely not affect another). Subprocesses share a kernel and a host; that's usually fine but isn't VM-level isolation. - Bots have large per-process memory footprints. The pool model is most efficient when one host can hold many bots. ## How it usually looks The dispatcher (a single long-running process per host) has two pieces: 1. **A resource pool.** At startup it pre-creates N transport resources (e.g. Daily rooms with tokens). When `/start` arrives, it pops one. After popping, it kicks off a background task to add one more, keeping the pool size stable. 2. **A worker manager.** It spawns the bot as a subprocess with the room/token as arguments, tracks the PID, and cleans up associated resources (delete the room, free any session state) when the process exits. The bot file itself is the standard Pipecat shape — `async def bot(runner_args)`, transport configured from `runner_args`. It doesn't know whether the room it joined was pre-allocated or freshly created. ## Tradeoffs worth being explicit about - **Latency vs. idle cost.** A warm pool of N pre-created rooms means you pay for those rooms even when nobody's using them. Set the pool size to match your typical idle-to-bursty ratio. - **Single-host concurrency ceiling.** The pattern works beautifully up to ~the host's capacity, then degrades sharply. Know what that ceiling is for your pipeline. - **Replenishment timing.** If you only refill after a pop, a sudden burst empties the pool faster than it can refill. A second background task that monitors pool size against a target can help. - **Process lifecycle.** Subprocesses crash; bots disconnect; transport sessions time out. The dispatcher needs to handle all three — typically by `await proc.wait()` in a background task that cleans up the associated transport resources on exit. - **Replicate the pattern, scale horizontally.** Running this dispatcher on multiple hosts is straightforward if your dispatch path doesn't need stickiness — put a load balancer in front and let the pools be per-host. ## See also - [`instant-voice` example](https://github.com/pipecat-ai/pipecat-examples/tree/main/instant-voice) — a working implementation with both `RoomPool` and `BotManager`. - [VM per session](./vm-per-session) — the opposite tradeoff (no warm capacity, full per-session isolation). # Managed Agent Runtime Source: https://docs.pipecat.ai/pipecat/deployment/patterns/managed-runtime.md Hand the bot lifecycle off to a runtime that owns scaling, dispatch, and execution. You write the bot file and a thin proxy server. The proxy receives session-start requests from clients and forwards them to an external agent runtime; the runtime owns the bot's execution environment, scaling, lifecycle, and (typically) the WebRTC offer/answer plumbing. You don't run a pool, you don't spawn subprocesses, you don't manage VM-level resources — the runtime does. This is the shape used in the [`aws-agentcore-webrtc`](https://github.com/pipecat-ai/pipecat-examples/tree/main/deployment/aws-agentcore-webrtc) example: the proxy is a small FastAPI app that calls `bedrock-agentcore` to invoke the agent, and the agent is a containerized bot file deployed to AgentCore Runtime. [Pipecat Cloud](/pipecat-cloud/introduction) is architecturally a managed runtime in this family. The difference from running PCC and rolling your own is mostly about how much of the runtime you want to operate vs. consume. ## When it fits - You'd rather not operate dispatch, fleet, or pool infrastructure at all. - You're already heavily invested in a cloud platform that has an agent-runtime offering (AWS Bedrock AgentCore, GCP Vertex AI agents, etc.). - Your bots fit comfortably within the runtime's constraints (container size, network egress, GPU availability if you need one). - You're willing to trade some flexibility for a lower-ops footprint. ## When it doesn't - The runtime's feature ceiling or pricing model doesn't match your bot. Agent runtimes tend to be opinionated about how the bot communicates with the outside world; if your bot needs unusual networking, custom transport, or specific compute shapes, you can hit walls. - You need portability across cloud providers and don't want a runtime-shaped lock-in. - Cold-start behavior of the runtime is unacceptable for your latency requirements. Most runtimes have warm-instance options but they cost extra. ## How it usually looks There are two pieces: 1. **A proxy server** that exposes whatever session-start API your clients expect (commonly `POST /start` and a per-session proxy under `/sessions/{id}/...`). It authenticates the request and forwards it to the runtime's invocation API. 2. **The bot file**, deployed to the runtime as a container. It receives session inputs (e.g. a WebRTC offer) through whatever channel the runtime supplies and runs the pipeline. In the AgentCore example the proxy forwards WebRTC offers and ICE candidates as JSON payloads via `bedrock.invoke_agent_runtime()`, and the agent container responds with a WebRTC answer over a streaming response. The bot file inside the container is a standard Pipecat bot. ## Tradeoffs worth being explicit about - **Low ops vs. lock-in.** You'll have less infrastructure to operate but you're betting on the runtime's roadmap and pricing. - **Feature ceiling.** Anything the runtime doesn't natively support — custom transports, unusual networking, specific GPU types, persistent volumes — is friction. Worth confirming the runtime supports your bot's actual shape, not just the example shape. - **Cold starts.** Runtimes typically scale from zero and pay cold-start cost on first invocation. Many offer "min-instance" settings that keep capacity warm at a flat cost. - **Observability.** Logs and metrics live in the runtime's surface, not yours. If you have an existing logging stack, plan for getting the bot's output into it (or accept living in the runtime's UI). ## See also - [`aws-agentcore-webrtc` example](https://github.com/pipecat-ai/pipecat-examples/tree/main/deployment/aws-agentcore-webrtc) — worked example of the pattern on Bedrock AgentCore with WebRTC. - [`aws-agentcore-websocket`](https://github.com/pipecat-ai/pipecat-examples/tree/main/deployment/aws-agentcore-websocket) and [`aws-agentcore-webrtc-kvs`](https://github.com/pipecat-ai/pipecat-examples/tree/main/deployment/aws-agentcore-webrtc-kvs) — same pattern, other AgentCore transport modes. - [Pipecat Cloud](/pipecat-cloud/introduction) — Daily's managed runtime, purpose-built for Pipecat. # Telephony in Production Source: https://docs.pipecat.ai/pipecat/deployment/telephony-in-production.md Webhook-driven dispatch, carrier-specific gotchas, and how telephony differs from WebRTC. Telephony bots have a different shape than WebRTC bots. The session doesn't start because a client sent your dispatcher an HTTP request — it starts because the carrier (Twilio, Telnyx, Plivo, Exotel, or your SIP provider) is calling your webhook to tell you there's an inbound call. The dispatch path is _inverted_, and a handful of carrier-specific concerns show up that don't exist in WebRTC. This page is about the operational shape, not the bot code itself. The dispatch _patterns_ described elsewhere — [VM per session](./patterns/vm-per-session), [warm pool with subprocess workers](./patterns/warm-pool-subprocess), [managed agent runtime](./patterns/managed-runtime) — all apply to telephony. What changes is what triggers them, and which patterns are actually viable: telephony adds tight latency constraints (a caller expects to hear something within a second or two of being connected) that often rule out cold-start-heavy patterns like vanilla VM-per-session. Warm capacity, in some form, is usually a requirement. ## What's different from WebRTC In WebRTC, your client app initiates the session. Your dispatcher gets a request from a client it knows, can authenticate, and can return a session ID to. In telephony, the carrier is the client. The dispatcher gets a webhook from a third party with a phone number on it and has to: 1. Respond synchronously with carrier-specific XML (TwiML for Twilio, the equivalent for Telnyx/Plivo) that tells the carrier where to send the bidirectional media stream. 2. Accept a long-lived WebSocket from the carrier carrying audio in both directions. 3. Spawn a bot wired to that WebSocket as its transport. The development runner's telephony mode handles this for you locally: ```bash python bot.py -t twilio -x your-name.ngrok.io python bot.py -t telnyx -x your-name.ngrok.io python bot.py -t plivo -x your-name.ngrok.io python bot.py -t exotel ``` The runner serves all transports at once when you omit `-t`, but for telephony you still pass `-t ` so it registers the carrier-specific XML webhook (`POST /`) built from the `--proxy` hostname. In production, whatever serves the carrier's webhook needs to do the same things: return the right XML, accept the WebSocket, hand it to a bot. For worked examples of telephony bots: - [`twilio-chatbot`](https://github.com/pipecat-ai/pipecat-examples/tree/main/twilio-chatbot) — inbound and outbound Twilio integration. - [`telnyx-chatbot`](https://github.com/pipecat-ai/pipecat-examples/tree/main/telnyx-chatbot), [`plivo-chatbot`](https://github.com/pipecat-ai/pipecat-examples/tree/main/plivo-chatbot), [`exotel-chatbot`](https://github.com/pipecat-ai/pipecat-examples/tree/main/exotel-chatbot) — equivalents for other carriers. - [`phone-chatbot/`](https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot) — Daily PSTN and SIP-based dial-in and dial-out, including transfer scenarios. - [`pipecat-cloud-daily-pstn-server`](https://github.com/pipecat-ai/pipecat-examples/tree/main/deployment/pipecat-cloud-daily-pstn-server) — webhook server for Daily's pinless dial-in flow against Pipecat Cloud. ## Carrier-specific gotchas These are the things teams running telephony in production routinely run into: ### Codecs and audio encoding Telephony providers transmit audio in carrier-specific formats — typically μ-law (Twilio, Telnyx) or a-law, at 8 kHz. Pipecat's [telephony serializers](https://github.com/pipecat-ai/pipecat/tree/main/src/pipecat/serializers) handle these on the wire, but you should be aware that: - The pipeline operates in PCM internally; the serializer converts at the transport boundary. - 8 kHz is _narrow_. Local STT/TTS models with broader bandwidth assumptions can sound worse than they do on WebRTC. - Twilio, Telnyx, Plivo, Exotel, Vonage, and Genesys all have slightly different framing and timing on the wire. Use the matching serializer. ### Session timeouts and reconnects Carriers enforce maximum call durations and may drop calls that go silent for too long. Decide how the bot handles graceful end-of-call (which you probably want to drive from the carrier's call-state events, not just from disconnect) vs. unexpected hang-ups. ### Webhook verification Carriers sign webhooks (Twilio's `X-Twilio-Signature`, Daily's pinless HMAC, etc.). In development you can skip verification; in production you should not, because the webhook URL is necessarily public and unsigned requests are trivial to forge. The [PSTN server example](https://github.com/pipecat-ai/pipecat-examples/tree/main/deployment/pipecat-cloud-daily-pstn-server) shows HMAC verification for Daily's pinless flow. ### Dial-out and call transfers If your product needs to _make_ calls, not just receive them, the dispatch shape changes again — now you're making an authenticated call to the carrier's API to initiate a call, and the bot needs to be running before the carrier connects the human party. Most carriers can do "create call → connect media when answered", but the orchestration is more involved than inbound. The [`phone-chatbot/daily-pstn-dial-out`](https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-pstn-dial-out) and [`daily-twilio-sip-dial-out`](https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-twilio-sip-dial-out) examples cover this. Warm and cold transfers (handing a call from the bot to a human agent without dropping the caller) add another layer; see [`daily-pstn-cold-transfer`](https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-pstn-cold-transfer) and [`daily-pstn-warm-transfer`](https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-pstn-warm-transfer). ## Pairing telephony with a dispatch pattern Telephony adds carrier-specific concerns on top of the same dispatch question you'd answer for WebRTC. You still need to decide who actually runs the bot process — the development runner directly, a VM per session, a warm-pool dispatcher, a managed runtime — and the answer is independent of the carrier choice. A common production shape is: carrier webhook → your dispatcher (validates the webhook signature, decides which bot to run) → your dispatch pattern of choice spawns a bot wired to the carrier's media WebSocket. # Fly.io Source: https://docs.pipecat.ai/pipecat/deployment/platforms/fly.md Deploy Pipecat bots to Fly.io machines: app configuration, scaling, and running voice agents in production. ## Project setup This is a worked example of the [VM per session](../patterns/vm-per-session) pattern: a long-running dispatcher receives session-start requests and calls Fly's machines API to spawn a fresh VM per session. The bot file in this example uses the older standalone `main(room_url, token)` shape rather than the newer `bot(runner_args)` [runner shape](../running-bots-locally) — both work in production; the runner shape is preferred for new bots, but the dispatch logic shown here is independent of which one your bot uses. We mentioned in the [overview](../overview) that you'd ideally containerize the dispatcher web service and the `bot.py` separately. To keep this example simple, we use the same container image for both. ![Fly.io Pipecat deployment](/images/deployment-fly.png) ### Install the Fly CLI You can find instructions for creating and setting up your fly account [here](https://fly.io/docs/getting-started/). ## Creating the Pipecat project We have created a template project [here](https://github.com/pipecat-ai/pipecat-examples/tree/main/deployment/flyio-example) which you can clone. Since we're targeting production use-cases, this example uses Daily (WebRTC) as a transport, but you can configure your bot however you like. ### Adding a fly.toml Add a `fly.toml` to the root of your project directory. Here is a basic example: ```toml fly.toml app = 'some-unique-app-name' primary_region = 'sjc' [build] [env] FLY_APP_NAME = 'some-unique-app-name' [http_service] internal_port = 7860 force_https = true auto_stop_machines = true auto_start_machines = true min_machines_running = 0 processes = ['app'] [[vm]] memory = 512 cpu_kind = 'shared' cpus = 1 ``` For apps with lots of users, consider what resources your HTTP service will require to meet load. We'll define our `bot.py` resources later, so you can set and scale these as you like (`fly scale ...`) ### Environment setup Our bot requires some API keys and configuration, so create a `.env` in your project root: ```shell .env DAILY_API_KEY= OPENAI_API_KEY= ELEVENLABS_API_KEY= ELEVENLABS_VOICE_ID= FLY_API_KEY= FLY_APP_NAME= ``` Of course, the exact keys you need will depend on which services you are using within your `bot.py`. **Important:** your `FLY_APP_NAME` should match the name of your fly instance, such as that declared in your fly.toml. The `.env` will allow us to test in local development, but is not included in the deployment. You'll need to set them as Fly app secrets, which you can do via the Fly dashboard or cli. `fly secrets set ...` ## Containerize our app Our Fly deployment will need a container image; let's create a simple `Dockerfile` in the root of the project: ```shell Dockerfile FROM python:3.11-slim-bookworm # Open port 7860 for http service ENV FAST_API_PORT=7860 EXPOSE 7860 # Install Python dependencies COPY \*.py . COPY ./requirements.txt requirements.txt RUN pip3 install --no-cache-dir --upgrade -r requirements.txt # Install models RUN python3 install_deps.py # Start the FastAPI server CMD python3 bot_runner.py --port ${FAST_API_PORT} ```` ```shell .dockerignore **/.DS_Store .env .env.* fly.toml ```` You can use any base image as long as Python is available Our container does the following: - Opens port `7860` to serve our `bot_runner.py` FastAPI service. - Downloads the necessary python dependencies. - Download / cache the model dependencies the `bot.py` requires. - Runs the `bot_runner.py` and listens for web requests. ### What models are we downloading? To support voice activity detection, we're using Silero VAD. Whilst the filesize is not huge, having each new machine download the Silero model at runtime will impact bootup time. Instead, we include the model as part of the Docker image so it's cached and available. You could, of course, also attach a network volume to each instance if you plan to include larger files as part of your deployment and don't want to bloat the size of your image. ## Launching new machines in `bot_runner.py` When a user starts a session with our Pipecat bot, we want to launch a new machine on fly.io with it's own system resources. Let's grab the bot_runner.py from the example repo [here](https://github.com/pipecat-ai/pipecat-examples/blob/main/deployment/flyio-example/bot_runner.py). This runner differs from others in the Pipecat repo; we've added a new method that sends a REST request to Fly to provision a new machine for the session. This method is invoked as part of the `/start_bot` endpoint: ```python bot_runner.py FLY_API_HOST = os.getenv("FLY_API_HOST", "https://api.machines.dev/v1") FLY_APP_NAME = os.getenv("FLY_APP_NAME", "your-fly-app-name") FLY_API_KEY = os.getenv("FLY_API_KEY", "") FLY_HEADERS = { 'Authorization': f"Bearer {FLY_API_KEY}", 'Content-Type': 'application/json' } def spawn_fly_machine(room_url: str, token: str): # Use the same image as the bot runner res = requests.get(f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines", headers=FLY_HEADERS) if res.status_code != 200: raise Exception(f"Unable to get machine info from Fly: {res.text}") image = res.json()[0]['config']['image'] # Machine configuration cmd = f"python3 bot.py -u {room_url} -t {token}" cmd = cmd.split() worker_props = { "config": { "image": image, "auto_destroy": True, "init": { "cmd": cmd }, "restart": { "policy": "no" }, "guest": { "cpu_kind": "shared", "cpus": 1, "memory_mb": 1024 # Note: 512 is just enough to run VAD, but 1gb is better } }, } # Spawn a new machine instance res = requests.post( f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines", headers=FLY_HEADERS, json=worker_props) if res.status_code != 200: raise Exception(f"Problem starting a bot worker: {res.text}") # Wait for the machine to enter the started state vm_id = res.json()['id'] res = requests.get( f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines/{vm_id}/wait?state=started", headers=FLY_HEADERS) if res.status_code != 200: raise Exception(f"Bot was unable to enter started state: {res.text}") ``` We want to make sure the machine started ok before returning any data to the user. Fly launches machines pretty fast, but will timeout if things take longer than they should. Depending on your transport method, you may want to optimistically return a response to the user, so they can join the room and poll for the status of their bot. ## Launch the Fly project Getting your bot on Fly is as simple as: `fly launch` or `fly launch --org orgname` if you're part of a team. This will step you through some configuration, and build and deploy your Docker image. Be sure to configure your app secrets with the necessary environment variables once the deployment has complete. Assuming all goes well, you can update with any changes with `fly deploy`. ### Test it out Start a new bot instance by sending a `POST` request to `https://your-fly-url.fly.dev/start_bot`. All being well, this will return a room URL and token. A nice feature of Fly is the ability to monitor your machines (with live logs) via their dashboard: https://fly.io/apps/YOUR-APP_NAME/machines This is really helpful for monitoring the status of your spawned machine, and debugging if things do not work as expected. This example is configured to expire after 5 minutes. The bot process is also configured to exit after the user leaves the room. This is a good way to ensure we don't have any hanging VMs, although you'll likely need to configure this behaviour this to meet your own needs. You'll also notice that we set `restart policy` to `no`. This prevents the machine attempting to restart after the session has concluded and the process exits. --- ## Important considerations This example does little in the way of load balancing or app security. Indeed, a user can spawn a new machine on your account simply by sending a `POST` request to the `bot_runner.py`. Be sure to configure a maximum number of instances, or authenticate requests to avoid costs getting out of control. We also deployed our `bot.py` on a machine with the same image as our `bot_runner.py`. To optimize container file sizes and increase security, consider individual images that only deploy resources they require. # Cerebrium Source: https://docs.pipecat.ai/pipecat/deployment/platforms/cerebrium.md Deploy Pipecat voice agents to Cerebrium: configuration, deployment steps, and production setup. [Cerebrium](https://www.cerebrium.ai) is a serverless Infrastructure platform that makes it easy for companies to build, deploy and scale AI applications. Cerebrium offers both CPUs and GPUs (H100s, A100s etc) with extremely low cold start times allowing us to create highly performant applications in the most cost efficient manner. This guide walks through deploying a bot to Cerebrium's serverless platform. Architecturally it sits in the [managed-runtime](../patterns/managed-runtime) family — Cerebrium owns the container lifecycle and scaling — though with closer access to the underlying compute (and GPU support) than typical agent runtimes offer. ### Install the Cerebrium CLI To get started, let us run the following commands: 1. Run `uv tool install cerebrium` to install the Cerebrium CLI globally. 2. Run `cerebrium login` to authenticate yourself. If you don't have a Cerebrium account, you can create one and get started with $30 in free credits. ### Create a Cerebrium project 1. Create a new Cerebrium project: ```bash cerebrium init pipecat-agent ``` 2. This will create two key files: - `main.py` - Your application entrypoint - `cerebrium.toml` - Configuration for build and environment settings Update your `cerebrium.toml` with the necessary configuration: ```toml [cerebrium.hardware] region = "us-east-1" provider = "aws" compute = "CPU" cpu = 4 memory = 18.0 [cerebrium.dependencies.pip] torch = ">=2.0.0" "pipecat-ai[silero, daily, openai, cartesia, deepgram]" = "latest" aiohttp = "latest" torchaudio = "latest" ``` In order for our application to work, we need to copy our API keys from the various platforms. Navigate to the Secrets section in your Cerebrium dashboard to store your API keys: - `OPENAI_API_KEY` - We use OpenAI for the LLM. You can get your API key from [here](https://platform.openai.com/api-keys) - `DAILY_API_KEY` - For WebRTC transport (used to create rooms and tokens). You can get your key from [here](https://dashboard.daily.co/developers) - `DEEPGRAM_API_KEY` - For speech-to-text. You can get your key from [here](https://console.deepgram.com/) - `CARTESIA_API_KEY` - For text-to-speech. You can get your API key from [here](https://play.cartesia.ai/keys) We access these secrets in our code as if they are normal ENV vars. You can swap in any LLM, STT, or TTS service you wish to use. ### Agent setup We create a basic pipeline setup in our `main.py` that combines our LLM, TTS and Daily WebRTC transport layer. ```python import os import sys from loguru import logger from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.frames.frames import EndFrame, LLMRunFrame from pipecat.pipeline.pipeline import Pipeline from pipecat.workers.runner import WorkerRunner from pipecat.pipeline.worker import PipelineParams, PipelineWorker from pipecat.processors.aggregators.llm_context import LLMContext from pipecat.processors.aggregators.llm_response_universal import ( LLMContextAggregatorPair, LLMUserAggregatorParams, ) from pipecat.services.cartesia.tts import CartesiaTTSService from pipecat.services.deepgram.stt import DeepgramSTTService from pipecat.services.openai.llm import OpenAILLMService from pipecat.transports.daily.transport import DailyParams, DailyTransport logger.remove(0) logger.add(sys.stderr, level="DEBUG") async def main(room_url: str, token: str): transport = DailyTransport( room_url, token, "Friendly bot", DailyParams( audio_in_enabled=True, audio_out_enabled=True, ), ) stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY")) llm = OpenAILLMService( api_key=os.environ.get("OPENAI_API_KEY"), model="gpt-4o", settings=OpenAILLMService.Settings( system_instruction=( "You are a helpful AI assistant in a voice conversation. " "Respond naturally and keep your answers conversational." ), ), ) tts = CartesiaTTSService( api_key=os.getenv("CARTESIA_API_KEY"), voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady ) context = LLMContext() user_aggregator, assistant_aggregator = LLMContextAggregatorPair( context, user_params=LLMUserAggregatorParams( vad_analyzer=SileroVADAnalyzer(), ), ) pipeline = Pipeline( [ transport.input(), stt, user_aggregator, llm, tts, transport.output(), assistant_aggregator, ] ) worker = PipelineWorker( pipeline, params=PipelineParams( enable_metrics=True, enable_usage_metrics=True, ), ) @transport.event_handler("on_first_participant_joined") async def on_first_participant_joined(transport, participant): context.add_message({"role": "user", "content": "Introduce yourself."}) await worker.queue_frames([LLMRunFrame()]) @transport.event_handler("on_participant_left") async def on_participant_left(transport, participant, reason): await worker.queue_frame(EndFrame()) @transport.event_handler("on_call_state_updated") async def on_call_state_updated(transport, state): if state == "left": await worker.queue_frame(EndFrame()) runner = WorkerRunner() await runner.run(worker) ``` First, the `main` function initializes the Daily transport to send and receive audio for the Daily room we'll connect to. We pass in the `room_url` we want to join and a `token` that authenticates the bot. Next, we wire up the AI services: Deepgram for speech-to-text, OpenAI for the LLM, and Cartesia for text-to-speech. The user aggregator is configured with a Silero VAD analyzer to detect when the user has finished speaking, which drives turn-taking. The pipeline chains these together: audio comes in from the transport, gets transcribed by Deepgram, accumulates into the LLM context via the user aggregator, gets a response from OpenAI, is spoken by Cartesia, and goes back out through the transport. The assistant aggregator captures the spoken response back into the context so the LLM has memory of the conversation. Finally, three event handlers manage the session lifecycle: when the first participant joins, the bot kicks off the conversation by introducing itself; when the participant leaves or the call ends, the bot terminates cleanly via an `EndFrame`. ### Deploy bot Deploy your application to Cerebrium: ```bash cerebrium deploy ``` You will then see that an endpoint is created for your bot at `POST \/main` that you can call with your room_url and token. Let us test it. ### Test it out ```python import asyncio import os import time import requests from loguru import logger def create_room(): url = "https://api.daily.co/v1/rooms/" headers = { "Content-Type": "application/json", "Authorization": f"Bearer {os.environ.get('DAILY_API_KEY')}", } data = { "properties": { "exp": int(time.time()) + 60 * 5, # 5 minutes "eject_at_room_exp": True, } } response = requests.post(url, headers=headers, json=data) if response.status_code == 200: room_info = response.json() token = create_token(room_info["name"]) if token and "token" in token: room_info["token"] = token["token"] else: logger.error("Failed to create token") return { "message": "There was an error creating your room", "status_code": 500, } return room_info else: logger.error(f"Failed to create room: {response.status_code}") return {"message": "There was an error creating your room", "status_code": 500} def create_token(room_name: str): url = "https://api.daily.co/v1/meeting-tokens" headers = { "Content-Type": "application/json", "Authorization": f"Bearer {os.environ.get('DAILY_API_KEY')}", } data = {"properties": {"room_name": room_name, "is_owner": True}} response = requests.post(url, headers=headers, json=data) if response.status_code == 200: return response.json() else: logger.error(f"Failed to create token: {response.status_code}") return None if __name__ == "__main__": room_info = create_room() room_url = room_info["url"] print(f"Join room: {room_url}") asyncio.run(main(room_url, room_info["token"])) ``` ## Future Considerations Since Cerebrium supports both CPU and GPU workloads if you would like to lower the latency of your application then the best would be to get model weights from various providers and run them locally. You can do this for: - LLM: Run any OpenSource model using a framework such as vLLM - TTS: Deepgram offers TTS models that can be run locally - STT: Deepgram offers a local model that can be run locally If you implement all three models locally, you should have much better performance. We have been able to get ~300ms voice-to-voice responses. ## Examples - [Fastest voice agent](https://docs.cerebrium.ai/v4/examples/realtime-voice-agents): Local only implementation - [RAG voice agent](https://www.cerebrium.ai/blog/creating-a-realtime-rag-voice-agent): Create a voice agent that can do RAG using Cerebrium + OpenAI + Pinecone - [Twilio voice agent](https://docs.cerebrium.ai/v4/examples/twilio-voice-agent): Create a voice agent that can receive phone calls via Twilio - [OpenAI Realtime API implementation](https://www.cerebrium.ai/blog/an-alternative-to-openai-realtime-api-for-voice-capabilities): Create a voice agent that can receive phone calls via OpenAI Realtime API # Modal Source: https://docs.pipecat.ai/pipecat/deployment/platforms/modal.md Deploy Pipecat voice agents to Modal: configuration, deployment steps, and running bots in production. [Modal](https://www.modal.com) is well-suited for Pipecat deployments because it handles container orchestration, scaling, and cold starts efficiently. This makes it a good choice for production Pipecat bots that need reliable performance. This guide walks through the Modal example included in the Pipecat repository. Architecturally it sits between the [VM-per-session](../patterns/vm-per-session) and [managed-runtime](../patterns/managed-runtime) patterns: a Modal Function plays the role of the per-session VM, but Modal manages the container lifecycle and image distribution for you. View the complete Modal deployment example in our GitHub repository ![Diagram of the deployment architecture of Modal and Pipecat example](/images/modal.jpg) ## Install the Modal CLI Follow Modal's official instructions for creating an account and setting up the CLI ## Deploy a self-serve LLM 1. Deploy Modal's OpenAI-compatible LLM service: ```bash git clone https://github.com/modal-labs/modal-examples cd modal-examples modal deploy 06_gpu_and_ml/llm-serving/vllm_inference.py ``` Refer to Modal's guide and example for [Deploying an OpenAI-compatible LLM service with vLLM](https://modal.com/docs/examples/vllm_inference) for more details. 2. Take note of the endpoint URL from the previous step, which will look like: ``` https://{your-workspace}--example-vllm-openai-compatible-serve.modal.run ``` You'll need this for the `bot_vllm.py` file in the next section. The default Modal LLM example uses Llama-3.1 and will shut down after 15 minutes of inactivity. Cold starts take 5-10 minutes. To prepare the service, we recommend visiting the `/docs` endpoint (`https://--example-vllm-openai-compatible-serve.modal.run/docs`) for your deployed LLM and wait for it to fully load before connecting your client. ## Deploy FastAPI App and Pipecat pipeline to Modal 1. Setup environment variables: ```bash cd server cp .env.example .env # Modify .env to provide your service API Keys ``` Alternatively, you can configure your Modal app to use [secrets](https://modal.com/docs/guide/secrets). 2. Update the `modal_url` in `server/src/bot_vllm.py` to point to the URL you received from the self-serve LLM deployment in the previous step. 3. From within the `server` directory, test the app locally: ```bash modal serve app.py ``` 4. Deploy to production: ```bash modal deploy app.py ``` 5. Note the endpoint URL produced from this deployment. It will look like: ```bash https://{your-workspace}--pipecat-modal-fastapi-app.modal.run ``` You'll need this URL for the client's `app.js` configuration mentioned in its README. ## Launch your bots on Modal ### Option 1: Direct Link Simply click on the URL displayed after running the server or deploy step to launch an agent and be redirected to a Daily room to talk with the launched bot. This will use the OpenAI pipeline. ### Option 2: Connect via an RTVI Client Follow the instructions provided in the [client folder's README](https://github.com/pipecat-ai/pipecat-examples/tree/main/deployment/modal-example/client/javascript) for building and running a custom client that connects to your Modal endpoint. The provided client includes a dropdown for choosing which bot pipeline to run. ## Navigating your LLM, server, and Pipecat logs On your [Modal dashboard](https://modal.com/apps), you should have two Apps listed under Live Apps: 1. `example-vllm-openai-compatible`: This App contains the containers and logs used to run your self-hosted LLM. There will be just one App Function listed: `serve`. Click on this function to view logs for your LLM. 2. `pipecat-modal`: This App contains the containers and logs used to run your `connect` endpoints and Pipecat pipelines. It will list two App Functions: 1. `fastapi_app`: This function is running the endpoints that your client will interact with and initiate starting a new pipeline (`/`, `/connect`, `/status`). Click on this function to see logs for each endpoint hit. 2. `bot_runner`: This function handles launching and running a bot pipeline. Click on this function to get a list of all pipeline runs and access each run's logs. ## Modal & Pipecat Tips - In most other Pipecat examples, we use `Popen` to launch the pipeline process from the `/connect` endpoint. In this example, we use a Modal function instead. This allows us to run the pipelines using a separately defined Modal image as well as run each pipeline in an isolated container. - For the FastAPI and most common Pipecat Pipeline containers, a default `debian_slim` CPU-only should be all that's required to run. GPU containers are needed for self-hosted services. - To minimize cold starts of the pipeline and reduce latency for users, set `min_containers=1` on the Modal Function that launches the pipeline to ensure at least one warm instance of your function is always available. ## Next steps For next steps on running a self-hosted LLM and reducing latency, check out all of Modal's LLM examples # Pipecat Examples Source: https://docs.pipecat.ai/pipecat/examples/overview.md Complete Pipecat example applications and quickstart demos: single-agent and multi-agent apps to accelerate development. ## Single-agent examples Quickstart} tags={["Getting Started"]}> Get started with a basic voice bot using `pipecat init quickstart`. [View Quickstart Guide →](/pipecat/get-started/quickstart) Simple Chatbot} tags={["Web & Mobile"]}> A full client-server example, showing how to connect different clients with a Pipecat bot. Client SDK examples include JavaScript, React, React Native, iOS, and Android. [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/simple-chatbot) Twilio Chatbot} tags={["Telephony"]}> Twilio websocket + Media streams. Inbound and outbound calling. [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/twilio-chatbot) Telnyx Chatbot} tags={["Telephony"]}> Telnyx websocket + Media streams. Inbound and outbound calling. [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/telnyx-chatbot) Plivo Chatbot} tags={["Telephony"]}> Plivo websocket + Media streams. Inbound and outbound calling. [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/plivo-chatbot) Exotel Chatbot} tags={["Telephony"]}> Exotel websocket + Media streams. Inbound and outbound calling. [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/exotel-chatbot) Daily PSTN & SIP Calling} tags={["Telephony"]}> Add PSTN & SIP calling to your Pipecat bot. Examples include Daily PSTN and Daily + Twilio SIP. Learn how to place inbound and outbound calls, as well as how to cold transfer a call. [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot) PSTN Warm Transfer} tags={["Telephony"]}> Warm transfer a PSTN call to a human agent using Daily PSTN. [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-pstn-warm-transfer) WhatsApp Chatbot} tags={["Telephony"]}> Add WhatsApp WebRTC calling to your Pipecat bot. [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/whatsapp) OpenTelemetry} tags={["Logging & Analytics"]}> Add OpenTelemetry to your Pipecat bot. Examples include Langfuse and Jaeger. [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/open-telemetry) IVR Navigation} tags={["Telephony"]}> Automatically navigate an IVR call tree by providing Pipecat with a goal. [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/ivr-navigation) Voicemail Detection} tags={["Telephony"]}> Detect voicemail in a phone call and leave a message. [View Example →](https://github.com/pipecat-ai/pipecat/blob/main/examples/features/features-voicemail-detection.py) Tavus} tags={["Video Avatar"]}> Use Tavus to create a video avatar for your Pipecat bot. [View Example →](https://github.com/pipecat-ai/pipecat/blob/main/examples/video-avatar/video-avatar-tavus-video-service.py) HeyGen} tags={["Video Avatar"]}> Use HeyGen to create a video avatar for your Pipecat bot. [View Example →](https://github.com/pipecat-ai/pipecat/blob/main/examples/video-avatar/video-avatar-heygen-video-service.py) Simli} tags={["Video Avatar"]}> Use Simli to create a video avatar for your Pipecat bot. [View Example →](https://github.com/pipecat-ai/pipecat/blob/main/examples/video-avatar/video-avatar-simli-video-service.py) Push to Talk} tags={["Web & Mobile"]}> A push-to-talk client interface, allowing users to press a button to start talking to the bot. [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/push-to-talk) SmallWebRTC + Docker} tags={["Web & Mobile"]}> Learn how to deploy the SmallWebRTCTransport in a Docker container. [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/p2p-webrtc/docker) Structured Conversation} tags={["Flows"]}> Learn how to use Pipecat Flows to create a structured navigation flow. This example shows a simple restaurant reservation flow. [View Example →](https://github.com/pipecat-ai/pipecat/blob/main/examples/flows/restaurant_reservation.py) Sentry Metrics} tags={["Logging & Analytics"]}> Learn how to use Sentry to track metrics and errors in your Pipecat bot. [View Example →](https://github.com/pipecat-ai/pipecat/blob/main/examples/observability/observability-sentry-metrics.py) Vonage Audio Connector} tags={["WebSocket Audio"]}> Stream real-time audio from an active Vonage Video API session into a Pipecat pipeline using the Vonage Audio Connector. [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/vonage-audio-bot) ## Multi-agent examples These build coordinated multi-agent systems. All of them live in the [pipecat repository](https://github.com/pipecat-ai/pipecat/tree/main/examples/multi-worker). Two LLM Agents} tags={["Local"]}> A greeter and a support agent that transfer control between each other. Demonstrates agent handoff with `activate_worker()` and the `@tool` decorator. [View Example →](https://github.com/pipecat-ai/pipecat/tree/main/examples/multi-worker/local-handoff/local-handoff-two-agents.py) Two LLM Agents with TTS} tags={["Local"]}> Same as above, but each agent has its own TTS with a distinct voice. The main agent has no TTS -- audio comes from the active LLM agent through the bus. [View Example →](https://github.com/pipecat-ai/pipecat/tree/main/examples/multi-worker/local-handoff/local-handoff-two-agents-tts.py) Parallel Debate} tags={["Local"]}> A moderator dispatches a topic to three agents in parallel using `job_group()`. Each one argues from a different perspective. Demonstrates job coordination. [View Example →](https://github.com/pipecat-ai/pipecat/tree/main/examples/multi-worker/parallel-debate/parallel-debate.py) Voice Code Assistant} tags={["Local"]}> A voice agent backed by a code agent that uses Claude Agent SDK with tools (Read, Bash, Glob, Grep) to answer coding questions. Demonstrates job-based agent integration. [View Example →](https://github.com/pipecat-ai/pipecat/tree/main/examples/multi-worker/code-assistant) Redis Handoff} tags={["Distributed"]}> The two-agent handoff split across separate processes using `RedisBus`. The main transport agent runs independently from the LLM agents. [View Example →](https://github.com/pipecat-ai/pipecat/tree/main/examples/multi-worker/distributed-handoff/redis-handoff) PGMQ Handoff} tags={["Distributed"]}> The same distributed handoff on a `PgmqBus` (Postgres / Supabase), for ops-friendly infrastructure. [View Example →](https://github.com/pipecat-ai/pipecat/tree/main/examples/multi-worker/distributed-handoff/pgmq-handoff) Remote Proxy Assistant} tags={["Distributed"]}> A main agent connects to a remote LLM server over WebSocket using proxy agents. Demonstrates point-to-point distributed deployment. [View Example →](https://github.com/pipecat-ai/pipecat/tree/main/examples/multi-worker/remote-proxy-assistant) --- ## 💡 Need Help Getting Started? Browse all examples on GitHub Browse code snippets & techniques Get help from the community # Recipes Source: https://docs.pipecat.ai/pipecat/examples/recipes.md Pipecat code recipes: focused snippets and techniques for common voice agent tasks and pipeline patterns. Function Calling} tags={["LLM"]}> Add function calling to your Pipecat bot. Examples exist for each LLM provider supported in Pipecat. [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/getting-started/07-function-calling.py) Record Audio} tags={["Recording & Logging"]}> Collect audio frames from the user and bot for later processing or storage. [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/audio/audio-recording.py) Capture Transcripts} tags={["Recording & Logging"]}> Capture user and bot transcripts for later processing or storage. [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/turn-management/turn-management-user-assistant-turns.py) Play Background Sound} tags={["Audio"]}> Play a background sound in your Pipecat bot. The audio is mixed with the transport audio to create a single integrated audio stream. [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/audio/audio-bot-background-sound.py) Mute User Input} tags={["User Interaction"]}> Specify a strategy for mute to mute user input, allowing the bot to continue without interruption. [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/turn-management/turn-management-user-mute-strategy.py) Wake Phrase} tags={["User Interaction"]}> Use a wake phrase to wake up your Pipecat bot. [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/features/features-wake-phrase.py) Play Sound Effects} tags={["Audio"]}> Play sound effects in your Pipecat bot. [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/audio/audio-sound-effects.py) Switch Languages} tags={["Multilingual"]}> A ParallelPipeline example showing how to dynamically switch languages. [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/features/features-switch-languages.py) Detect an Idle User} tags={["User Interaction"]}> Detect when a user is idle and automatically respond. [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/turn-management/turn-management-detect-user-idle.py) Debug with an Observer} tags={["Debugging"]}> Learn how to debug your Pipecat bot with an Observer by observing frames flowing through the pipeline. [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/observability/observability-observer.py) Live Debugger (Whisker)} tags={["Debugging"]}> A live graphical debugger for the Pipecat voice and multimodal conversational AI framework. It lets you visualize pipelines and debug frames in real time — so you can see exactly what your bot is thinking and doing. [View Recipe →](https://github.com/pipecat-ai/whisker) Collect Emails} tags={["Recording & Logging"]}> Parse user email from the LLM response. [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/features/features-user-email-gathering.py) Smart Turn Detection} tags={["User Interaction"]}> Detect when a user has finished speaking and automatically respond. Learn more about [smart-turn model](https://github.com/pipecat-ai/smart-turn). [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/turn-management/turn-management-smart-turn-local.py) MCP Tools} tags={["Integration"]}> Use MCP tools to interact with external services. [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/mcp/mcp-stdio.py) MCP Tools (Streamable HTTP)} tags={["Integration"]}> Use MCP tools over Streamable HTTP transport for remote MCP servers. [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/mcp/mcp-streamable-http.py) Multiple MCP Servers} tags={["Integration"]}> Combine tools from multiple MCP servers (stdio and HTTP) in a single bot. [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/mcp/mcp-multiple-mcp.py) Interruption Strategies} tags={["User Interaction"]}> Learn how to configure interruption strategies for your Pipecat bot. [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/turn-management/turn-management-interruption-config.py) Describe Video} tags={["Vision"]}> Pass a video frame from a live video stream to a model and get a description. [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/vision/vision-openai.py) User and Bot Turn Events} tags={["Events"]}> Handle user and bot end of turn events to add custom logic after a turn. [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/turn-management/turn-management-turn-tracking-observer.py) --- ## 💡 Need Help Getting Started? Browse all recipes on GitHub Browse complete applications Get help from the community # Introduction to Pipecat Clients Source: https://docs.pipecat.ai/client/introduction.md Build real-time voice and multimodal apps with Pipecat client SDKs: microphone, playback, and transport handling built in. Pipecat's client SDKs connect your users to Pipecat agents running on the server. They handle the real-time media layer — microphone and camera access, audio playback, transport connections, and the event stream from your bot — so you can focus on building your application. Build and run your first Pipecat voice application ## Available SDKs Core SDK for web applications Hooks and components for React apps iOS and Android via React Native Native Swift SDK for iOS Native Kotlin SDK for Android Native SDK for desktop and embedded ## How It Fits Together Pipecat's architecture is split between server and client. The server runs your AI pipeline — speech recognition, LLM, text-to-speech — and the client connects your user to it over a real-time transport. The client SDKs implement the [RTVI standard](/client/rtvi-standard) — an open protocol for real-time AI inference — which means they work with any RTVI-compatible server, not just Pipecat. ## What the SDKs Handle - **Transport management** — establishing, maintaining, and tearing down connections - **Media capture** — microphone and camera device access and streaming - **Audio playback** — rendering bot audio output - **Session state** — tracking connection state and bot readiness - **Event stream** — callbacks and events for transcriptions, bot speaking, errors, and more - **Messaging** — sending custom messages to your bot and handling responses ## Ready to Build? Get a voice conversation running in minutes Understand transports, sessions, events, and media Full reference for all SDK methods, hooks, and callbacks Complete example applications across all platforms # Pipecat Client Quickstart Source: https://docs.pipecat.ai/client/get-started/quickstart.md Get a Pipecat client voice app running in minutes with the CLI, with React, JavaScript, and mobile SDK variants. You are currently viewing the React version of this page. Use the dropdown to the right to customize this page for your client framework. You are currently viewing the JavaScript version of this page. Use the dropdown to the right to customize this page for your client framework. This quickstart guide will help you build your first Pipecat voice AI bot with a React front-end and run it locally. You'll create a simple conversational agent that you connect and talk to in real-time via your browser. This quickstart guide will help you build your first Pipecat voice AI bot with a vanilla JavaScript front-end and run it locally. You'll create a simple conversational agent that you connect and talk to in real-time via your browser. The CLI generates React and vanilla JavaScript projects. For React Native, iOS, Android, or C++, see [Building a Voice UI](/client/guides/building-a-voice-ui) to get started without the CLI. ## Prerequisites - Python 3.11+ and [uv](https://docs.astral.sh/uv/getting-started/installation/) - Node.js 18+ ## Step 1: Scaffold your project ```bash # Install the Pipecat CLI uv tool install "pipecat-ai[cli]" # Start your project pipecat init ``` When `init` asks how you want to build, choose **Scaffold a runnable bot now**. The wizard will guide you through the setup. Choose the following options: This quickstart uses the interactive `pipecat init` wizard so you end up with the same project this guide walks through below. When you're ready to build your own bot, we recommend driving development with a coding agent — see [Build Your Next Bot](/pipecat/get-started/build-your-next-bot). - **Project name**: `my-voice-app` - **Bot type**: `Web/Mobile` - **Client framework**: `React` - **React dev server**: `Vite` - **Transport**: `SmallWebRTC` The rest is up to you! The CLI generates a complete project with two main directories: - `bot/` — a Python Pipecat bot - `client/` — a React front-end built on the [Voice UI Kit](/client/voice-ui-kit) - **Project name**: `my-voice-app` - **Bot type**: `Web/Mobile` - **Client framework**: `Vanilla JS` - **Transport**: `SmallWebRTC` The rest is up to you! The CLI generates a complete project with two main directories: - `bot/` — a Python Pipecat bot - `client/` — a vanilla JavaScript front-end built on the Pipecat JS SDK ## Step 2: Start the bot ```bash cd my-voice-app/bot cp .env.example .env # add your API keys uv sync uv run bot.py ``` The bot starts at `http://localhost:7860`. ## Step 3: Start the client ```bash cd ../client npm install npm run dev ``` Open `http://localhost:5173`, click **Connect**, allow microphone access, and start talking. **First run note**: The initial startup may take ~20 seconds as Pipecat downloads required models and imports. Subsequent runs will be much faster. 🎉 **Success!** Your bot is running locally. Now let's deploy it to production so others can use it. --- ## Understanding the Quickstart Client The React app is built on the [Voice UI Kit](/client/voice-ui-kit) — Pipecat's library of pre-built voice UI components. Here's what each file does: **`src/config.ts`** — transport configuration. By default it uses SmallWebRTC for local development. To switch to Daily for production, add your Daily credentials and update `AVAILABLE_TRANSPORTS`. ```ts const botStartUrl = import.meta.env.VITE_BOT_START_URL || "http://localhost:7860/start"; export const DEFAULT_TRANSPORT: TransportType = "smallwebrtc"; ``` **`src/main.tsx`** — entry point. `PipecatAppBase` handles transport creation, connection lifecycle, loading states, and errors, passing a ready `client` down to your app via a render prop. ```tsx {({ client, handleConnect, handleDisconnect, error }) => !client ? : error ? {error} : } ``` **`src/components/App.tsx`** — the UI. Built entirely from Voice UI Kit components: | Component | What it does | | ------------------- | ---------------------------------------------------------------- | | `ConnectButton` | Connect/disconnect button with built-in loading and error states | | `UserAudioControl` | Mic mute/unmute toggle | | `ConversationPanel` | Live transcript of the conversation | | `EventsPanel` | Real-time stream of RTVI events — useful during development | The vanilla JS app is built directly on the Pipecat JS SDK. Here's what each file does: **`src/config.js`** — transport configuration. By default it uses SmallWebRTC for local development. Also exports a `createTransport()` helper that dynamically imports the correct transport package. To switch to Daily for production, add your Daily credentials and update `AVAILABLE_TRANSPORTS`. ```js const botStartUrl = import.meta.env.VITE_BOT_START_URL || "http://localhost:7860/start"; export const DEFAULT_TRANSPORT = "smallwebrtc"; ``` **`src/app.js`** — the entire client UI, implemented as a `VoiceChatClient` class. On construction it wires up DOM elements and event listeners. Calling `connect()` creates a `PipecatClient`, sets up the audio track, and starts the bot: ```js async connect() { const transport = await createTransport(this.transportType); this.client = new PipecatClient({ transport, enableMic: true, callbacks: { onUserTranscript: (data) => { if (data.final) this.addConversationMessage(data.text, 'user'); }, onBotTranscript: (data) => { this.addConversationMessage(data.text, 'bot'); }, // ... }, }); await this.client.startBotAndConnect(TRANSPORT_CONFIG[this.transportType]); } ``` **`index.html`** — the page structure. Contains the transport selector, connect button, mic toggle, conversation log, and events panel that `app.js` references by ID. | Element | What it does | | -------------------------- | ----------------------------------------------------------- | | `#connect-btn` | Connect/disconnect button | | `#mic-btn` / `#mic-status` | Mic mute/unmute toggle | | `#conversation-log` | Live transcript of the conversation | | `#events-log` | Real-time stream of RTVI events — useful during development | ## Deploying to production The `.env.example` file shows the production configuration. Point `VITE_BOT_START_URL` at your deployed bot's start endpoint: ```bash # Pipecat Cloud VITE_BOT_START_URL="https://api.pipecat.daily.co/v1/public/{agentName}/start" VITE_BOT_START_PUBLIC_API_KEY="your-public-api-key" ``` The client app is a static site — build it with `npm run build` and deploy to any static hosting provider (Vercel, Netlify, S3, etc.). ## Next steps Customize themes, swap components, and extend the generated UI Build a React voice app from scratch — understand the SDK without abstractions Understand transports, sessions, events, and media Deploy to Pipecat Cloud for production Build a voice app from scratch — understand the SDK without abstractions Understand transports, sessions, events, and media Deploy to Pipecat Cloud for production # The RTVI Standard Source: https://docs.pipecat.ai/client/rtvi-standard.md The RTVI standard for real-time voice interaction: the open protocol connecting Pipecat clients and servers. The RTVI (Real-Time Voice and Video Inference) standard defines a set of message types and structures sent between clients and servers. It is designed to facilitate real-time interactions between clients and AI applications that require voice, video, and text communication. It provides a consistent framework for building applications that can communicate with AI models and the backends running those models in real-time. This page documents version 1.0 of the RTVI standard, released in June 2025. ## Key Features RTVI provides a flexible connection model that allows clients to connect to AI services and coordinate state. The standard includes built-in support for real-time transcription of audio streams. The standard defines a messaging protocol for sending and receiving messages between clients and servers, allowing for efficient communication of requests and responses. The standard supports advanced interactions with large language models (LLMs), including context management, function call handline, and search results. RTVI supports events to provide insight into the input/output and state for typical services that exist in speech-to-speech workflows. RTVI provides mechanisms for collecting metrics and monitoring the performance of server-side services. ## Terms - **Client**: The front-end application or user interface that interacts with the RTVI server. - **Server**: The backend-end service that runs the AI framework and processes requests from the client. - **User**: The end user interacting with the client application. - **Bot**: The AI interacting with the user, technically an amalgamation of a large language model (LLM) and a text-to-speech (TTS) service. ## RTVI Message Format The messages defined as part of the RTVI protocol adhere to the following format: ```json { "id": string, "label": "rtvi-ai", "type": string, "data": unknown } ``` A unique identifier for the message, used to correlate requests and responses. A label that identifies this message as an RTVI message. This field is required and should always be set to `'rtvi-ai'`. The type of message being sent. This field is required and should be set to one of the predefined RTVI message types listed below. The payload of the message, which can be any data structure relevant to the message type. ## RTVI Message Types Following the above format, this section describes the various message types defined by the RTVI standard. Each message type has a specific purpose and structure, allowing for clear communication between clients and servers. Each message type below includes either a 🤖 or 🏄 emoji to denote whether the message is sent from the bot (🤖) or client (🏄). ### Connection Management #### client-ready 🏄 Indicates that the client is ready to receive messages and interact with the server. Typically sent after the transport media channels have connected. - **type**: `'client-ready'` - **data**: - **version**: `string` The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. - **about**: `AboutClient Object` An object containing information about the client, such as its rtvi-version, client library, and any other relevant metadata. The `AboutClient` object follows this structure: Any platform-specific details that may be relevant to the server. This could include information about the browser, operating system, or any other environment-specific data needed by the server. This field is optional and open-ended, so please be mindful of the data you include here and any security concerns that may arise from exposing sensitive or personal-identifiable information. #### bot-ready 🤖 Indicates that the bot is ready to receive messages and interact with the client. Typically send after the transport media channels have connected. - **type**: `'bot-ready'` - **data**: - **version**: `string` The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations. - **about**: `any` (Optional) An object containing information about the server or bot. It's structure and value are both undefined by default. This provides flexibility to include any relevant metadata your client may need to know about the server at connection time, without any built-in security concerns. Please be mindful of the data you include here and any security concerns that may arise from exposing sensitive information. #### disconnect-bot 🏄 Indicates that the client wishes to disconnect from the bot. Typically used when the client is shutting down or no longer needs to interact with the bot. Note: Disconnets should happen automatically when either the client or bot disconnects from the transport, so this message is intended for the case where a client may want to remain connected to the transport but no longer wishes to interact with the bot. - **type**: `'disconnect-bot'` - **data**: `undefined` #### error 🤖 Indicates an error occurred during bot initialization or runtime. - **type**: `'error'` - **data**: - **message**: `string` Description of the error. - **fatal**: `boolean` Indicates if the error is fatal to the session. ### Speaking and Transcription #### user-started-speaking 🤖 Emitted when the user begins speaking - **type**: `'user-started-speaking'` - **data**: None #### user-stopped-speaking 🤖 Emitted when the user stops speaking - **type**: `'user-stopped-speaking'` - **data**: None #### vad-user-started-speaking 🤖 Disabled by default; enable with `RTVIObserverParams(vad_user_speaking_enabled=True)`. Emitted when the VAD (Voice Activity Detection) detects the user started speaking. This is the raw VAD signal, emitted independently of turn finalization (unlike `user-started-speaking`, which a turn strategy may gate or defer). - **type**: `'vad-user-started-speaking'` - **data**: None #### vad-user-stopped-speaking 🤖 Disabled by default; enable with `RTVIObserverParams(vad_user_speaking_enabled=True)`. Emitted when the VAD (Voice Activity Detection) detects the user stopped speaking. This is the raw VAD signal, emitted independently of turn finalization (unlike `user-stopped-speaking`, which a turn strategy may gate or defer). - **type**: `'vad-user-stopped-speaking'` - **data**: None #### bot-started-speaking 🤖 Emitted when the bot begins speaking - **type**: `'bot-started-speaking'` - **data**: None #### bot-stopped-speaking 🤖 Emitted when the bot stops speaking - **type**: `'bot-stopped-speaking'` - **data**: None #### user-mute-started 🤖 Introduced in RTVI version 1.2.0 (Pipecat 0.0.102, client-js 1.6.0). Emitted when the server begins ignoring audio from the client (server-side muting). The client should continue sending audio normally but may want to show an indication to the user that their input is not being processed. - **type**: `'user-mute-started'` - **data**: None #### user-mute-stopped 🤖 Introduced in RTVI version 1.2.0 (Pipecat 0.0.102, client-js 1.6.0). Emitted when the server stops ignoring audio from the client (server-side muting ends). The client can update its UI to indicate that the user's input is being processed again. - **type**: `'user-mute-stopped'` - **data**: None #### user-transcription 🤖 Real-time transcription of user speech, including both partial and final results. - **type**: `'user-transcription'` - **data**: - **text**: `string` The transcribed text of the user. - **final**: `boolean` Indicates if this is a final transcription or a partial result. - **timestamp**: `string` The timestamp when the transcription was generated. - **user_id**: `string` Identifier for the user who spoke. #### bot-output 🤖 The best-effort representation of the bot's output text, including both spoken and unspoken text. In addition to transcriptions of spoken text, this message type may also include text that the bot outputs but does not speak (e.g., text sent to the client for display purposes only). Along with the text, this event includes a `spoken` flag to indicate whether the text was spoken by the bot or not and an `aggregated_by` field to indicate what the text represents (e.g. "sentence", "word", "code", "url"). - **type**: `'bot-output'` - **data**: - **text**: `string` The output text from the bot. - **spoken**: `boolean` Indicates if this text was spoken by the bot. - **aggregated_by**: `string` Indicates how the text was aggregated (e.g., "sentence", "word", "code", "url"). "sentence" and "word" are reserved aggregation types defined by the RTVI standard. Other aggregation types may be defined by custom text aggregators used by the server. #### bot-transcription 🤖 DEPRECATED in favor of `bot-output` in Pipecat version 0.0.95 and client-js version 1.5.0 Transcription of the bot's speech. Note: This protocol currently does not match the user transcription format to support real-time timestamping for bot transcriptions. Rather, the event is typically sent for each sentence of the bot's response. This difference is currently due to limitations in TTS services which mostly do not support (or support well), accurate timing information. If/when this changes, this protocol may be updated to include the necessary timing information. For now, if you want to attempt real-time transcription to match your bot's speaking, you can try using the `bot-tts-text` message type. - **type**: `'bot-transcription'` - **data**: - **text**: `string` The transcribed text from the bot, typically aggregated at a per-sentence level. ### Client-Server Messaging #### server-message 🤖 An arbitrary message sent from the server to the client. This can be used for custom interactions or commands. This message may be coupled with the `client-message` message type to handle responses from the client. - **type**: `'server-message'` - **data**: `any` The `data` can be any JSON-serializable object, formatted according to your own specifications. #### client-message 🏄 An arbitrary message sent from the client to the server. This can be used for custom interactions or commands. This message may be coupled with the `server-response` message type to handle responses from the server. - **type**: `'client-message'` - **data**: - **t**: `string` - **d**: `unknown` (optional) The data payload should contain a `t` field indicating the type of message and an optional `d` field containing any custom, corresponding data needed for the message. #### server-response 🤖 An message sent from the server to the client in response to a `client-message`. **IMPORTANT**: The `id` should match the `id` of the original `client-message` to correlate the response with the request. - **type**: `'client-message'` - **data**: - **t**: `string` - **d**: `unknown` (optional) The data payload should contain a `t` field indicating the type of message and an optional `d` field containing any custom, corresponding data needed for the message. #### error-response 🤖 Error response to a specific client message. **IMPORTANT**: The `id` should match the `id` of the original `client-message` to correlate the response with the request. - **type**: `'error-response'` - **data**: - **error**: `string` ### Advanced LLM Interactions #### send-text 🏄 A message sent from the client to the server to send text input to the LLM, appended to the user's context. - **type**: `'send-text'` - **data**: - **content**: `string` The text content to be appended to the user context. - **options**: `object` (optional) - **run_immediately**: `boolean` (optional) If `true`, the pipeline should be interrupted and the LLM should process the input immediately after appending it to the context. Defaults to `true`. - **audio_response**: `boolean` (optional) If `false`, the bot should bypass the TTS so that the bot does not respond to the text in audio. Defaults to `true`. #### llm-function-call-started 🤖 Introduced in RTVI version 1.2.0 (Pipecat 0.0.102, client-js 1.6.0). Indicates that a function call has been initiated by the LLM. The amount of metadata included in this event is controlled by the server's `function_call_report_level` configuration. - **type**: `'llm-function-call-started'` - **data**: - **function_name**: `string` (optional) Name of the function being called. Only included if the server's report level is `NAME` or `FULL`. #### llm-function-call-in-progress 🤖 Introduced in RTVI version 1.2.0 (Pipecat 0.0.102, client-js 1.6.0). Indicates that a function call is in progress. This is the successor to the deprecated `llm-function-call` message and is the event that triggers registered `FunctionCallHandler`s when a `function_name` is present. - **type**: `'llm-function-call-in-progress'` - **data**: - **function_name**: `string` (optional) Name of the function being called. Only included if the server's report level is `NAME` or `FULL`. - **tool_call_id**: `string` Unique identifier for this function call. - **arguments**: `Record` (optional) Arguments passed to the function. Only included if the server's report level is `FULL`. #### llm-function-call-stopped 🤖 Introduced in RTVI version 1.2.0 (Pipecat 0.0.102, client-js 1.6.0). Indicates that a function call has completed or been cancelled. - **type**: `'llm-function-call-stopped'` - **data**: - **function_name**: `string` (optional) Name of the function that was called. Only included if the server's report level is `NAME` or `FULL`. - **tool_call_id**: `string` Identifier matching the original function call. - **cancelled**: `boolean` Indicates whether the function call was cancelled before completing. - **result**: `unknown` (optional) The result of the function call, if available. Only included if the server's report level is `FULL`. #### llm-function-call 🤖 DEPRECATED in favor of `llm-function-call-in-progress` in Pipecat version 0.0.102 and client-js version 1.6.0 A function call request from the LLM, sent from the bot to the client. Note that for most cases, an LLM function call will be handled completely server-side. However, in the event that the call requires input from the client or the client needs to be aware of the function call, this message/response schema is required. - **type**: `'llm-function-call'` - **data**: - **function_name**: `string` Name of the function to be called. - **tool_call_id**: `string` Unique identifier for this function call. - **args**: `Record` Arguments to be passed to the function. #### llm-function-call-result 🏄 The result of the function call requested by the LLM, returned from the client. - **type**: `'llm-function-call-result'` - **data**: - **function_name**: `string` Name of the called function. - **tool_call_id**: `string` Identifier matching the original function call. - **arguments**: `Record` Arguments that were passed to the function. - **result**: `Record | string` The result returned by the function. #### bot-llm-search-response 🤖 Search results from the LLM's knowledge base. Currently, Google Gemini is the only LLM that supports built-in search. However, we expect other LLMs to follow suite, which is why this message type is defined as part of the RTVI standard. As more LLMs add support for this feature, the format of this message type may evolve to accommodate discrepancies. - **type**: `'bot-llm-search-response'` - **data**: - **search_result**: `string` (optional) Raw search result text. - **rendered_content**: `string` (optional) Formatted version of the search results. - **origins**: `Array` Source information and confidence scores for search results. The `Origin Object` follows this structure: ```json { "site_uri": string (optional), "site_title": string (optional), "results": Array<{ "text": string, "confidence": number[] }> } ``` **Example:** ```json "id": undefined "label": "rtvi-ai" "type": "bot-llm-search-response" "data": { "origins": [ { "results": [ { "confidence": [0.9881149530410768], "text": "* Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm." }, { "confidence": [0.9692034721374512], "ext": "* A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm." } ], "site_title": "vanderbilt.edu", "site_uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHwif83VK9KAzrbMSGSBsKwL8vWfSfC9pgEWYKmStHyqiRoV1oe8j1S0nbwRg_iWgqAr9wUkiegu3ATC8Ll-cuE-vpzwElRHiJ2KgRYcqnOQMoOeokVpWqi" }, { "results": [ { "confidence": [0.6554043292999268], "text": "In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields." } ], "site_title": "wikipedia.org", "site_uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQESbF-ijx78QbaglrhflHCUWdPTD4M6tYOQigW5hgsHNctRlAHu9ktfPmJx7DfoP5QicE0y-OQY1cRl9w4Id0btiFgLYSKIm2-SPtOHXeNrAlgA7mBnclaGrD7rgnLIbrjl8DgUEJrrvT0CKzuo" }], "rendered_content": "