“Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results.

​
How It Works

The flow of interactions in a Pipecat application is typically straightforward:

  1. The bot says something
  2. The user says something
  3. The bot says something
  4. The user says something

This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing.

​
Real-time Processing

Consider a voice-based interaction where the bot needs to respond:

  1. Transcribe the user’s speech as they’re talking
  2. Process the transcription through an LLM
  3. Convert the response to speech
  4. Play the audio to the user

For multimodal models (like GPT-4V), the flow might be different:

  1. Process multiple input streams (audio, video, images)
  2. Send combined input to the multimodal model
  3. Handle various output types (text, generated images, etc.)
  4. Coordinate the presentation of multiple outputs

In both cases, Pipecat:

  • Processes responses as they stream in
  • Handles multiple input/output modalities concurrently
  • Manages resource allocation and synchronization
  • Coordinates parallel processing tasks

This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved.

​
Key Features

​
Real-time Processing

  • Frame-based pipeline architecture for processing real-time data
  • Choose between immediate processing or non-blocking background operations
  • Built-in frame synchronization for multimodal applications

​
Voice-first Design

  • Native support for speech recognition
  • Real-time text-to-speech conversion
  • Voice activity detection
  • Natural conversation handling

​
Flexible Pipeline Architecture

Pipelines are like assembly lines for your AI application, connecting processors that handle different tasks. Each processor can transform data, generate new content, or pass information through unchanged. This architecture makes it easy to build complex applications from simple, reusable components.

# Example of a voice assistant pipeline
pipeline = Pipeline([
    transport.input(),   # Receives audio input
    transcriber,         # Converts speech to text
    llm_processor,       # Generates responses
    tts_service,         # Converts text to speech
    transport.output()   # Plays audio output
])

​
Service Integration

  • Native support for popular AI services (OpenAI, ElevenLabs, etc.)
  • WebRTC integration through Daily
  • Extensible service architecture for custom integrations

​
What You Can Build

Pipecat is ideal for building:

  • Voice-based AI assistants
  • Interactive AI agents
  • Multimodal chatbots
  • Real-time AI processing systems
  • Streaming media applications

​
Getting Started

Ready to build your first Pipecat application? Start with:

​
Join Our Community

Need help or want to share your project? Join our Discord community where you can connect with other developers and get support from the Pipecat team.

Use Cases