Community

GitHub

Examples

Changelog

Pipecat

API Reference

Join the Discord

Dialout: WebRTC (Daily)

Pipecat is a framework for building voice-enabled, real-time, multimodal AI applications

Overview

Explore the different types of applications you can build with Pipecat, from voice assistants to multimodal AI agents

Use Cases

Understanding how Pipecat works: frames, processors, and pipelines.

Core Concepts

Get Pipecat and its required services installed on your machine

Installation & Setup

Build and run your first Pipecat application

Quickstart

Basic deployment pattern for Pipecat bots

Deployment pattern

Example: Fly.io

Example: Cerebrium

Enable LLMs to interact with external services and APIs

Function Calling

Learn how to create structured conversations using Pipecat’s flow system

Pipecat Flows

Learn how to integrate Krisp noise cancellation into your Pipecat application

Krisp noise cancellation

Learn how to monitor performance and LLM/TTS usage with Pipecat.

Metrics

How to call your bots (or have them call you)

Dial-in: WebRTC (Daily)

Call your Pipecat bot using Daily WebRTC and a Twilio phone number

Dial-in: WebRTC (Twilio + Daily)

Call your Pipecat bot over websockets using Twilio

Dial-in: WebSockets (Twilio)

AI services integrated with Pipecat and their setup requirements

Supported Services

Reference documentation for Pipecat’s base service classes

Base Service Classes

WebRTC transport implementation using Daily for real-time audio/video communication

Daily

Speech-to-text service implementation using AssemblyAI’s real-time transcription API

AssemblyAI

Speech-to-text service using Azure Cognitive Services Speech SDK

Azure

Speech-to-text service implementation using Deepgram’s real-time transcription API

Deepgram

Speech-to-text service implementation using Gladia’s API

Gladia

Speech-to-text service implementation using locally-downloaded Whisper models

Whisper

Large Language Model service implementation using Anthropic’s Claude API

Anthropic

Large Language Model service implementation using Azure OpenAI API

LLM service implementation using Fireworks AI’s API with OpenAI-compatible interface

Fireworks AI

Large Language Model service implementation using Google’s Gemini API

Google Gemini

LLM service implementation using Grok’s API with OpenAI-compatible interface

Grok

LLM service implementation using Groq’s API with OpenAI-compatible interface

Groq

LLM service implementation using NVIDIA’s NIM (NVIDIA Inference Microservice) API with OpenAI-compatible interface

NVIDIA NIM

LLM service implementation using Ollama with OpenAI-compatible interface

Ollama

Large Language Model services using OpenAI’s chat completion API

OpenAI

LLM service implementation using OpenPipe for LLM request logging and fine-tuning

OpenPipe

LLM service implementation using Together AI’s API with OpenAI-compatible interface

Together AI

Text-to-speech service implementation using Amazon AWS Polly

AWS Polly

Text-to-speech service using Azure Cognitive Services Speech SDK

Text-to-speech services using Cartesia’s WebSocket and HTTP APIs

Cartesia

Text-to-speech service implementation using Deepgram’s Aura API

Text-to-speech service using ElevenLab’s streaming API with word-level timing

ElevenLabs

Text-to-speech service using Google’s Cloud Text-to-Speech API

Google

Text-to-speech service implementation using LMNT’s streaming API

LMNT

Text-to-speech service using OpenAI’s TTS API

Text-to-speech services using PlayHT’s WebSocket and HTTP APIs

PlayHT

Text-to-speech service implementation using Rime AI

Rime

Text-to-speech service implementation using Coqui’s XTTS streaming server

XTTS

Real-time speech-to-speech service implementation using OpenAI’s Realtime Beta API

OpenAI Realtime Beta

Image generation service implementation using fal’s fast SDXL models

Video service implementation for generating AI avatar responses using Tavus

Tavus

Vision service implementation using Moondream for local image analysis and question answering

Moondream

Analytics service for processing and analyzing conversation recordings using Canonical AI’s Voice API

Canonical AI

Performance monitoring integration with Sentry for Pipecat frame processors

Sentry Metrics

Audio noise reduction filter using Krisp AI technology

KrispFilter

Audio noise reduction filter using the noisereduce library

NoisereduceFilter

Voice Activity Detection analyzer using the Silero VAD ONNX model

SileroVADAnalyzer

Audio mixer for combining real-time audio with sound files

SoundfileMixer

Technical reference for Pipecat’s conversation flow system

Core frame classes that all other Pipecat frames inherit from

Base Frame Types

Frame types for handling audio data in Pipecat

Audio Frames

Frame types for handling image data and visual interactions in Pipecat

Image Frames

Frame types for handling text and transcriptions in Pipecat

Text Frames

Frame types for handling LLM interactions and responses in Pipecat

LLM Frames

Frame types for handling LLM function calls and tool execution in Pipecat

Function Calling Frames

Frame types for managing speech detection and interaction events in Pipecat

Speech Event Frames

Frame types for controlling pipeline execution and flow in Pipecat

Pipeline Control Frames

Frame types for controlling and configuring AI services in Pipecat

Service Control Frames

Frame types for error handling and fatal error conditions in Pipecat

Error Frames

Frame types for managing transport-level messages and image requests in Pipecat

Transport Frames

Frame types for collecting and reporting performance metrics in Pipecat

Metrics Frames

Base class for all frame processors in Pipecat, providing core processing functionality

FrameProcessor

Frame processor for aggregating user speech transcriptions into complete responses

UserResponseAggregator

Frame processors for managing and aggregating LLM conversations and contexts

LLM Response Aggregators

Frame processor that combines text frames into complete sentences

SentenceAggregator

Frame processor that combines text prompts with images for vision model processing

VisionImageFrameAggregator

Voice Activity Detection processor using Silero VAD model for speech detection

SileroVAD

Processor for controlling STT muting and interruption handling during bot speech

STTMuteFilter

Abstract base class for all pipeline implementations in Pipecat

BasePipeline

Standard linear pipeline implementation for connecting frame processors

Pipeline

Pipeline implementation for concurrent processing paths

ParallelPipeline

Pipeline implementation for synchronized parallel processing paths

SyncParallelPipeline

Task management for pipeline execution and control

PipelineTask

Orchestrator for managing multiple pipeline tasks with signal handling

Get Started

Guides

Dialout: WebRTC (Daily)