# agent
Source: https://docs.pipecat.ai/cli/cloud/agent

Manage agent deployments

The `agent` command provides sub-commands for managing your deployed agents. These commands allow you to view status, start agents, see logs, and manage deployments.

## start

Start a deployed agent instance, creating an active session.

**Usage:**

```shell theme={null}
pipecat cloud agent start [ARGS] [OPTIONS]
```

**Arguments:**

<ParamField type="string">
  Unique string identifier for the agent deployment. Must not contain spaces.
</ParamField>

**Options:**

<ParamField type="string">
  Public API key to authenticate the agent deployment. Will default to any key
  set in your config.

  For more information, see [API keys](/deployment/pipecat-cloud/fundamentals/accounts-and-organizations#api-keys).
</ParamField>

<ParamField type="string">
  Stringified JSON object to pass to the agent deployment. This data will be
  available to the agent as a `data` parameter in your `bot()` method.

  More information [here](/deployment/pipecat-cloud/fundamentals/active-sessions#running-an-agent).
</ParamField>

<ParamField type="boolean">
  Skip summary confirmation before issuing start request.
</ParamField>

<ParamField type="boolean">
  Create a Daily WebRTC session for the agent.
</ParamField>

<ParamField type="string">
  Stringified JSON object with Daily room properties to customize the WebRTC
  session. Only used when `--use-daily` is set to true.

  See [Daily API
  documentation](https://docs.daily.co/reference/rest-api/rooms/config) for
  available properties.
</ParamField>

<ParamField type="str">
  Organization to start the agent for. If not provided, uses the current
  organization from your configuration.
</ParamField>

## stop

Stop an active agent session and clean up its resources.

**Usage:**

```shell theme={null}
pcc agent stop [ARGS] [OPTIONS]
```

**Arguments:**

<ParamField type="string">
  Name of the agent. Must not contain spaces.
</ParamField>

**Options:**

<ParamField type="string">
  ID of the session to stop.
</ParamField>

<ParamField type="string">
  Organization which the agent belongs to. If not provided, uses the current
  organization from your configuration.
</ParamField>

<ParamField type="boolean">
  Bypass prompt for confirmation before stopping the session.
</ParamField>

## status

Shows the current status of an agent deployment, including health and conditions.

**Usage:**

```shell theme={null}
pipecat cloud agent status [ARGS]
```

**Arguments:**

<ParamField type="string">
  Unique string identifier for the agent deployment. Must not contain spaces.
</ParamField>

## deployments

Lists deployment history for an agent, including image versions and timestamps.

**Usage:**

```shell theme={null}
pipecat cloud agent deployments [ARGS]
```

**Arguments:**

<ParamField type="string">
  Unique string identifier for the agent deployment. Must not contain spaces.
</ParamField>

## logs

Displays combined logs from all agent instances, useful for debugging issues.

**Usage:**

```shell theme={null}
pipecat cloud agent logs [ARGS] [OPTIONS]
```

**Arguments:**

<ParamField type="string">
  Unique string identifier for the agent deployment. Must not contain spaces.
</ParamField>

**Options:**

<ParamField type="string">
  Filter logs by severity: `ALL`, `DEBUG`, `INFO`, `WARNING`, `ERROR`,
  `CRITICAL`.
</ParamField>

<ParamField type="int">
  Limit the number of log lines to display.
</ParamField>

<ParamField type="str">
  Filter results for specific agent deployment ID (obtainable from `pipecat
      cloud agent deployments [agent-name]`).
</ParamField>

<ParamField type="str">
  Filter results for specific session ID (obtainable from `pipecat cloud agent
      sessions [agent-name]`).
</ParamField>

## list

Lists all agents in an organization with their details.

**Usage:**

```shell theme={null}
pipecat cloud agent list [OPTIONS]
```

**Options:**

<ParamField type="string">
  Organization to list agents for. If not provided, uses the current
  organization from your configuration.
</ParamField>

<ParamField type="string">
  Filter agents by region. Only agents deployed in the specified region will be
  shown. If not provided, agents from all regions are listed.
</ParamField>

## sessions

Lists active sessions for a specified agent. When there are no active sessions, it suggests how to start a new session.

When used with the `--id` option, displays detailed information about a specific session including CPU and memory usage with sparkline visualizations and percentile summaries.

**Usage:**

```shell theme={null}
pipecat cloud agent sessions [ARGS] [OPTIONS]
```

**Arguments:**

<ParamField type="string">
  Name of the agent to list active sessions for.
</ParamField>

**Options:**

<ParamField type="string">
  Session ID to view detailed metrics for. When provided, displays CPU and memory usage statistics including sparkline visualizations and percentile summaries (p50, p90, p99).
</ParamField>

<ParamField type="string">
  Organization to list sessions for. If not provided, uses the current
  organization from your configuration.
</ParamField>

## delete

Deletes an agent deployment. This will prevent starting new agents and remove all associated data.

<Warning>This action is irreversible. All data will be lost.</Warning>

**Usage:**

```shell theme={null}
pipecat cloud agent delete [ARGS] [OPTIONS]
```

**Arguments:**

<ParamField type="string">
  Unique string identifier for the agent deployment. Must not contain spaces.
</ParamField>

**Options:**

<ParamField type="string">
  Do not prompt for confirmation before deleting the agent.
</ParamField>


# auth
Source: https://docs.pipecat.ai/cli/cloud/auth

Authentication and authorization commands

The `auth` command group manages authentication with Pipecat Cloud, allowing you to login, logout, and check your current account identity.

## login

Begins an authorization flow to authenticate with Pipecat Cloud.

**Usage:**

```shell theme={null}
pipecat cloud auth login [OPTIONS]
```

**Options:**

<ParamField type="boolean">
  Skip opening a browser window for authentication and print the URL instead.
  Useful for remote or containerized environments.
</ParamField>

This command initiates the authentication process by:

1. Generating a unique authentication URL
2. Opening your default web browser (if available)
3. Waiting for you to complete the sign-in process in the browser
4. Retrieving and storing your access token locally

If the browser doesn't open automatically, or if you use the `--headless` option, you can copy and paste the displayed URL into your browser manually.

On successful login, the CLI will store the access token in the local configuration file (defaults to `~/.config/pipecatcloud/pipecatcloud.toml`). This token will be used for all subsequent requests to the Pipecat Cloud API.

You can override the default location by setting the `PIPECAT_CONFIG_PATH` environment variable.

## logout

Signs out of the current Pipecat Cloud account by removing the access token from your local configuration file.

**Usage:**

```shell theme={null}
pipecat cloud auth logout
```

## whoami

Displays information about the currently authenticated user, including user ID, active organization, and Daily API key.

**Usage:**

```shell theme={null}
pipecat cloud auth whoami
```

## Configuration

CLI configuration is stored in `~/.config/pipecatcloud/pipecatcloud.toml`. You can override this location by setting the `PIPECAT_CONFIG_PATH` environment variable.

The configuration stores your access token, active organization, and default API keys.

**View current configuration:**

```shell theme={null}
pipecat cloud --config
```

<Accordion title="Example configuration file">
  <Warning>We do not recommend manually editing the configuration file. Should the file become malformed, please run `pipecat cloud auth logout` to reset.</Warning>

  ```toml theme={null}
    token = "..."
    org = "user-namespace"

    [another-user-org]
    default_public_key = "pk_..."
    default_public_key_name = "Test Key"

    [futher-user-org]
    default_public_key = "pk_..."
    default_public_key_name = "Pipecat Cloud Public Key"
  ```
</Accordion>

***

<Card title="More about user accounts" icon="people-group" href="/deployment/pipecat-cloud/fundamentals/accounts-and-organizations">
  Managing your account and collaborating on agents as part of a team
</Card>


# deploy
Source: https://docs.pipecat.ai/cli/cloud/deploy

Create or modify an agent deployment

The `deploy` command creates a new agent deployment or updates an existing one. It builds a deployment manifest with the provided parameters and monitors the deployment status until the agent is ready.

If the agent name already exists, you'll be prompted to confirm the update unless the `--force` flag is used.

This command will wait for the active deployment / revision to enter a ready state before returning. If the deployment fails, the command will exit with an [error](/deployment/pipecat-cloud/fundamentals/error-codes) with more information.

## Usage

```shell theme={null}
pipecat cloud deploy [ARGS] [OPTIONS]
```

**Arguments:**

<ParamField type="string">
  Unique string identifier for the agent deployment. Must not contain spaces.
</ParamField>

<ParamField type="string">
  URL of the Docker image to deploy. Must be a valid Docker image URL. For
  example: `docker.io/my-repo/my-image:latest`.
</ParamField>

**Options:**

<ParamField type="string">
  Name of the image pull secret to use for accessing private repositories. The
  secret must be previously created using the `pipecat cloud secrets
      image-pull-secret` command.
</ParamField>

<ParamField type="string">
  Organization to deploy the agent to. If not provided, uses the current
  organization from your configuration.
</ParamField>

<ParamField type="string">
  Name of the secret set to use for the deployment. The secret set must exist in
  the specified organization.
</ParamField>

<ParamField type="number">
  Minimum number of agent instances to keep warm at all times. Default is 0,
  which means the agent will scale down to zero when not in use. Setting this to
  1 or higher avoids cold starts.
</ParamField>

<ParamField type="number">
  Maximum number of allowed agent instances. Must be between 1 and 50. If you
  need more agents, please contact us at [help@daily.co](mailto:help@daily.co) or via
  [Discord](https://discord.gg/dailyco).
</ParamField>

<ParamField type="string">
  Enable Krisp VIVA noise cancellation with the specified audio filter model.
  Valid values are:

  * `tel`: Telephony model (up to 16kHz)
  * `pro`: WebRTC model (up to 32kHz)

  In addition to this flag, you also need to enable the `KrispVivaFilter()` for your transport. See the [Krisp VIVA](/server/utilities/audio/krisp-viva-filter) documentation for more information.
</ParamField>

<ParamField type="boolean">
  Enable [managed API keys](/deployment/pipecat-cloud/guides/managed-api-keys)
  for this deployed agent.
</ParamField>

<ParamField type="string">
  The agent profile to use for resource allocation. Valid values are:
  `agent-1x`, `agent-2x`, `agent-3x`.

  See [Agent Profiles](/deployment/pipecat-cloud/fundamentals/deploy#agent-profiles) for more information.
</ParamField>

<ParamField type="string">
  Region where the agent will be deployed. If not specified, uses your
  organization's default region (typically `us-west`). Choose a region close to
  your users for optimal latency.
</ParamField>

<ParamField type="boolean">
  Force deployment and skip confirmation prompts. Use with caution.
</ParamField>

## Examples

**Deploy a new agent:**

```shell theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.1
```

**Update an existing agent with a new image:**

```shell theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.2
```

**Deploy with a specific secret set:**

```shell theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.1 --secrets my-secret-set
```

**Deploy a private image using image pull credentials:**

```shell theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.1 --credentials dockerhub-creds
```

**Keep one instance always warm to avoid cold starts:**

```shell theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.1 --min-agents 1
```

**Limit the maximum number of agent instances:**

```shell theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.1 --max-agents 5
```

**Deploy to a specific region:**

```shell theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.1 --region eu-central
```

**Deploy with Krisp VIVA noise cancellation:**

```shell theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.1 --krisp-viva-audio-filter tel
```

## Configuration File (pcc-deploy.toml)

The `deploy` command supports a configuration file for repeatable deployments. Create a `pcc-deploy.toml` file in your project root to define deployment settings that can be shared across your team and version controlled.

### File Location

Place `pcc-deploy.toml` in the same directory where you run the `pipecat cloud deploy` command. The CLI will automatically detect and use this file.

### Precedence

Values are applied with the following order of precedence:

1. CLI arguments (highest priority)
2. `pcc-deploy.toml` values
3. Default values (lowest priority)

This allows you to define defaults in the config file while still overriding specific values via CLI flags when needed.

### Configuration Options

#### Required Fields

<ParamField type="string">
  Name of the agent to deploy. Must start with a lowercase letter or number, can
  include hyphens, and must end with a lowercase letter or number.

  ```toml theme={null}
  agent_name = "my-voice-agent"
  ```
</ParamField>

<ParamField type="string">
  Docker image URL with tag.

  ```toml theme={null}
  image = "your-dockername/my-agent:0.1"
  ```
</ParamField>

#### Optional Fields

<ParamField type="string">
  Region where the agent will be deployed. If not specified, uses your
  organization's default region (typically `us-west`).

  ```toml theme={null}
  region = "us-east"
  ```
</ParamField>

<ParamField type="string">
  Name of the secret set to use for environment variables. The secret set must
  exist in the same region as the agent.

  ```toml theme={null}
  secret_set = "my-agent-secrets"
  ```
</ParamField>

<ParamField type="string">
  Name of the image pull secret for private registries. The image pull secret must exist in the same region as the agent.

  ```toml theme={null}
  image_credentials = "dockerhub-credentials"
  ```
</ParamField>

<ParamField type="string">
  Agent profile for resource allocation. Valid values: `agent-1x`, `agent-2x`, `agent-3x`.

  ```toml theme={null}
  agent_profile = "agent-2x"
  ```
</ParamField>

<ParamField type="boolean">
  **Deprecated:** Enable legacy Krisp noise cancellation. Use `krisp_viva` instead.

  ```toml theme={null}
  enable_krisp = false
  ```
</ParamField>

<ParamField type="boolean">
  Enable managed API keys for this deployment.

  ```toml theme={null}
  enable_managed_keys = false
  ```
</ParamField>

#### Scaling Configuration

Define auto-scaling behavior in a `[scaling]` section:

<ParamField type="number">
  Minimum number of agent instances to keep warm. Setting to 0 allows scaling to zero but may result in cold starts.

  ```toml theme={null}
  [scaling]
  min_agents = 1
  ```
</ParamField>

<ParamField type="number">
  Maximum number of agent instances allowed.

  ```toml theme={null}
  [scaling]
  max_agents = 20
  ```
</ParamField>

#### Krisp VIVA Configuration

Configure Krisp VIVA noise cancellation in a `[krisp_viva]` section:

<ParamField type="string">
  Krisp VIVA audio filter model. Valid values: `tel` (telephony, up to 16kHz) or `pro` (WebRTC, up to 32kHz). Omit or set to `null` to disable.

  ```toml theme={null}
  [krisp_viva]
  audio_filter = "tel"
  ```
</ParamField>

### Complete Example

```toml theme={null}
# Basic agent configuration
agent_name = "my-voice-agent"
image = "your-dockername/my-voice-agent:0.1"
region = "us-west"

# Secrets and credentials
secret_set = "my-agent-secrets"
image_credentials = "dockerhub-credentials"

# Resource allocation
agent_profile = "agent-1x"

# Features
enable_managed_keys = false

# Auto-scaling configuration
[scaling]
min_agents = 1
max_agents = 20

# Krisp VIVA noise cancellation
[krisp_viva]
audio_filter = "tel"
```

### Using the Configuration File

Once you have a `pcc-deploy.toml` file, simply run:

```shell theme={null}
pipecat cloud deploy
```

The CLI will automatically load your configuration. You can still override any value using CLI flags:

```shell theme={null}
# Use config file but override the region
pipecat cloud deploy --region eu-central

# Use config file but force update without confirmation
pipecat cloud deploy --force
```


# docker
Source: https://docs.pipecat.ai/cli/cloud/docker

Build and push Docker images for agent deployments

The `docker` command provides utilities for building, tagging, and pushing Docker images to container registries. This command automatically parses registry information from your deployment configuration and supports both Docker Hub and custom registries.

## build-push

Build, tag, and push a Docker image for your agent deployment. This command reads configuration from your `pcc-deploy.toml` file to automatically determine registry settings, image names, and versions.

**Usage:**

```shell theme={null}
pipecat cloud docker build-push [ARGS] [OPTIONS]
```

**Arguments:**

<ParamField type="string">
  Name of the agent to build image for. If not provided, uses the `agent_name`
  from your `pcc-deploy.toml` file.
</ParamField>

**Registry Configuration:**

<ParamField type="string">
  Registry type to push to. Supported values: `dockerhub`, `custom`. When not
  specified, automatically detected from the `image` field in your
  `pcc-deploy.toml` file.
</ParamField>

<ParamField type="string">
  Registry username for authentication. When not specified, automatically parsed
  from the `image` field in your `pcc-deploy.toml` file (e.g., `myusername` from
  `myusername/app:1.0`).
</ParamField>

<ParamField type="string">
  Custom registry URL (required for custom registries). When not specified,
  automatically parsed from the `image` field for custom registries (e.g.,
  `gcr.io` from `gcr.io/project/app:1.0`).
</ParamField>

**Build Configuration:**

<ParamField type="string">
  Version tag for the image. When not specified, automatically extracted from
  the `image` field in your `pcc-deploy.toml` file (e.g., `1.0` from
  `myusername/app:1.0`).
</ParamField>

<ParamField type="boolean">
  Build and tag only, do not push to registry. Useful for local testing or when
  you want to push manually later.
</ParamField>

<ParamField type="boolean">
  Do not tag the image as `latest`. By default, images are tagged with both the
  specified version and `latest`.
</ParamField>

## Configuration

The `docker build-push` command reads configuration from your `pcc-deploy.toml` file to minimize required command-line arguments. Here's how different registry setups work:

### Docker Hub (Default)

For Docker Hub repositories, the minimal configuration is:

```toml theme={null}
agent_name = "my-agent"
image = "myusername/my-agent:1.0"
```

This automatically configures:

* Registry: `dockerhub`
* Username: `myusername`
* Agent name: `my-agent`
* Version: `1.0`

### Custom Registry

For custom registries like Google Container Registry, AWS ECR, or private registries:

```toml theme={null}
agent_name = "my-agent"
image = "gcr.io/my-project/my-agent:1.0"
```

This automatically configures:

* Registry: `custom`
* Registry URL: `gcr.io`
* Username/Project: `my-project`
* Agent name: `my-agent`
* Version: `1.0`

### Docker Configuration Section

For advanced configuration, add a `[docker]` section:

```toml theme={null}
agent_name = "my-agent"
image = "myusername/my-agent:1.0"

[docker]
auto_latest = false  # Don't tag as 'latest'
```

Available `[docker]` options:

<ParamField type="boolean">
  Whether to automatically tag the image as `latest` in addition to the
  specified version.
</ParamField>

## Examples

### Basic Usage (Recommended)

With a properly configured `pcc-deploy.toml`:

```shell theme={null}
# Build and push using all configuration from pcc-deploy.toml
pipecat cloud docker build-push
```

### Override Version

```shell theme={null}
# Use a different version than what's in pcc-deploy.toml
pipecat cloud docker build-push --version 2.0
```

### Build Only

```shell theme={null}
# Build and tag locally without pushing
pipecat cloud docker build-push --no-push
```

### Different Registry

```shell theme={null}
# Override registry settings for one-time builds
pipecat cloud docker build-push --registry custom --registry-url my-registry.com --username myuser
```

### Skip Latest Tag

```shell theme={null}
# Only tag with the specific version, not 'latest'
pipecat cloud docker build-push --no-latest
```

## Platform Support

All images are built for the `linux/arm64` platform, which is required for Pipecat Cloud deployments. This is automatically configured and cannot be changed.

## Error Handling

The command provides helpful error messages for common issues:

* **Authentication errors**: Suggests the appropriate `docker login` command
* **Missing Dockerfile**: Indicates that a Dockerfile must be present in the current directory
* **Registry access issues**: Provides guidance on checking permissions and authentication


# organizations
Source: https://docs.pipecat.ai/cli/cloud/organizations

Organization and API key management commands

The `organizations` command group helps you manage your Pipecat Cloud organizations and API keys. You can list and select organizations, as well as create, list, and manage API keys for use with the platform.

<Note>
  Organization and user management is not available via the CLI. Please use the
  [Pipecat Cloud Dashboard](https://pipecat.daily.co) to manage organizations
  and users.
</Note>

## list

Lists all organizations that your account has access to, highlighting the currently active one used for CLI operations.

**Usage:**

```shell theme={null}
pipecat cloud organizations list
```

## select

Changes your active organization for CLI operations.

This command either presents an interactive selection menu or directly sets a specified organization as your default. The selection is stored in your local configuration file (defaults to `~/.config/pipecatcloud/pipecatcloud.toml`) and used for all subsequent CLI commands.

**Usage:**

```shell theme={null}
pipecat cloud organizations select [OPTIONS]
```

**Options:**

<ParamField type="string">
  Bypass prompt by directly specifying a namespace / organization string.
</ParamField>

## keys

The `keys` sub-commands manage API keys for authenticating with Pipecat Cloud services.

### keys list

Lists all API keys for the current organization.

**Usage:**

```shell theme={null}
pipecat cloud organizations keys list [OPTIONS]
```

**Options:**

<ParamField type="string">
  Organization ID to list keys for. If not provided, the default organization
  will be used.
</ParamField>

### keys create

Create a new public API key for account / organization. Command will prompt the user to enter
a human-readable name for the key.

**Usage:**

```shell theme={null}
pipecat cloud organizations keys create [OPTIONS]
```

**Options:**

<ParamField type="string">
  Organization ID to create key for. If not provided, the default organization
  will be used.
</ParamField>

### keys delete

Delete an API key from your organization. Command will prompt the user to select which key they wish to delete.

**Usage:**

```shell theme={null}
pipecat cloud organizations keys delete [OPTIONS]
```

**Options:**

<ParamField type="string">
  Organization ID to delete key for. If not provided, the default organization
  will be used.
</ParamField>

### keys use

Sets a specific API key as your default for CLI operations.

The selected key is stored in the local configuration file (defaults to `~/.config/pipecatcloud/pipecatcloud.toml`) and will be used for all subsequent requests to the Pipecat Cloud API.

Please note that the key must be associated with the same user account or organization as you are making requests to.

If the public key is revoked or deleted via the dashboard, the user will need to re-run this command to select a new key.

**Usage:**

```shell theme={null}
pipecat cloud organizations keys use [OPTIONS]
```

**Options:**

<ParamField type="string">
  Organization ID to select default key from. If not provided, the default
  organization will be used.
</ParamField>

## properties

The `properties` sub-commands manage organization properties such as default region settings.

### properties list

Lists all current property values for your organization.

**Usage:**

```shell theme={null}
pipecat cloud organizations properties list [OPTIONS]
```

**Options:**

<ParamField type="string">
  Organization ID to list properties for. If not provided, the default
  organization will be used.
</ParamField>

### properties schema

Shows available properties with detailed metadata including type information, current values, default values, and available values.

**Usage:**

```shell theme={null}
pipecat cloud organizations properties schema [OPTIONS]
```

**Options:**

<ParamField type="string">
  Organization ID to show properties schema for. If not provided, the default
  organization will be used.
</ParamField>

### properties set

Updates a specific organization property.

**Usage:**

```shell theme={null}
pipecat cloud organizations properties set PROPERTY_NAME VALUE [OPTIONS]
```

**Arguments:**

<ParamField type="string">
  Name of the property to set (e.g., `defaultRegion`)
</ParamField>

<ParamField type="string">
  Value to set for the property
</ParamField>

**Options:**

<ParamField type="string">
  Organization ID to update property for. If not provided, the default
  organization will be used.
</ParamField>

**Example:**

```shell theme={null}
pipecat cloud organizations properties set defaultRegion eu-central
```

## default-region

Convenience command to get or set the default region for your organization.

**Usage:**

```shell theme={null}
pipecat cloud organizations default-region [REGION] [OPTIONS]
```

**Arguments:**

<ParamField type="string">
  Region to set as default. If omitted, displays the current default region and
  available regions.
</ParamField>

**Options:**

<ParamField type="string">
  Organization ID to configure. If not provided, the default organization will
  be used.
</ParamField>

**Examples:**

View current default region:

```shell theme={null}
pipecat cloud organizations default-region
```

Set default region:

```shell theme={null}
pipecat cloud organizations default-region eu-central
```

***

<Card title="More about organizations" icon="people-group" href="/deployment/pipecat-cloud/fundamentals/accounts-and-organizations">
  Managing your account and collaborating on agents as part of a team
</Card>


# regions
Source: https://docs.pipecat.ai/cli/cloud/regions

View available deployment regions

The `regions` command helps you discover which regions are available for deploying agents and storing secrets in Pipecat Cloud.

## list

List all available regions with their codes and display names.

**Usage:**

```shell theme={null}
pipecat cloud regions list
```

This command displays a table of all regions where you can deploy agents and store secrets. Use the region codes shown in this list when specifying the `--region` flag in other commands.

**Example output:**

```
Code         Name
us-west      US West (Oregon)
us-east      US East (Virginia)
eu-central   Europe (Frankfurt)
ap-south     Asia Pacific (Mumbai)
```

## Using regions

Once you know the available region codes, you can use them with other commands:

**Deploy an agent to a specific region:**

```shell theme={null}
pipecat cloud deploy my-agent my-image:latest --region eu-central
```

**Create secrets in a specific region:**

```shell theme={null}
pipecat cloud secrets set my-secrets API_KEY=abc123 --region eu-central
```

**List agents in a specific region:**

```shell theme={null}
pipecat cloud agent list --region us-east
```

<Info>
  Secrets and image pull secrets must be in the same region as the agents that
  use them. When deploying multi-region applications, create separate secret
  sets for each region.
</Info>

<Tip>
  Choose a region close to your users for optimal latency and performance.
</Tip>


# secrets
Source: https://docs.pipecat.ai/cli/cloud/secrets

Secret sets and secret management commands

The `secrets` command group helps you manage sensitive information for your agent deployments. You can create and manage secret sets (key-value pairs) and image pull secrets (for private Docker registries).

## list

List secret sets and image pull secrets for active namespace / organization.

If provided with a valid secret set name, will show the keys of that set (values are hidden.)

**Usage:**

```shell theme={null}
pipecat cloud secrets list [ARGS] [OPTIONS]
```

**Arguments:**

<ParamField type="string">
  Name of the secret set list keys for. Must be a valid string identifier.
</ParamField>

**Options:**

<ParamField type="boolean">
  Show secret sets only. Filter out image pull secrets from the results.
</ParamField>

<ParamField type="string">
  Organization to list secrets for. If not provided, uses the current
  organization from your configuration.
</ParamField>

<ParamField type="string">
  Filter secrets by region. Only secrets in the specified region will be shown.
  If not provided, secrets from all regions are listed.
</ParamField>

## set

Create or update a secret set with the given name and key-value pairs. Secrets can be passed directly as key value pairs or loaded from a file.

**Usage:**

```shell theme={null}
pipecat cloud secrets set [ARGS] [OPTIONS]
```

**Arguments:**

<ParamField type="string">
  Name of the secret set to create or modify. Must be a valid string identifier
  containing only characters, numbers, and hyphens.
</ParamField>

<ParamField type="string">
  List of secret key-value pairs e.g. `KEY1=value1 KEY2="value with spaces"`.

  See [this note](/deployment/pipecat-cloud/fundamentals/secrets#special-characters) on using special
  characters in secret values.

  Example:

  ```shell theme={null}
  pipecat cloud secrets set my-secrets 'API_KEY=123 API_KEY_2="value with spaces"'
  ```
</ParamField>

**Options:**

<ParamField type="string">
  Relative path to a file with a list of secret key-value pairs. Each line in
  the file should be in the format `KEY=value`.

  Example:

  ```shell theme={null}
  pipecat cloud secrets set my-secrets --file .env
  ```
</ParamField>

<ParamField type="boolean">
  Skip confirmations and proceed with the operation.
</ParamField>

<ParamField type="string">
  Organization to create/update the secret set in. If not provided, uses the
  current organization from your configuration.
</ParamField>

<ParamField type="string">
  Region where the secret set will be stored. If not specified, uses your
  organization's default region (typically `us-west`). Secrets must be in the
  same region as the agents that use them.
</ParamField>

**Example:**

Create a secret set in a specific region:

```shell theme={null}
pipecat cloud secrets set my-secrets API_KEY=abc123 --region eu-central
```

## unset

Removes a specific secret key from a secret set.

**Usage:**

```shell theme={null}
pipecat cloud secrets unset [ARGS] [OPTIONS]
```

**Arguments:**

<ParamField type="string">
  Name of the secret set to remove the secret from.
</ParamField>

<ParamField type="string">
  The key of the secret to remove from the set.
</ParamField>

Example:

```shell theme={null}
pipecat cloud secrets unset my-secret-set SOME_KEY
```

**Options:**

<ParamField type="boolean">
  Skip confirmations and proceed with the operation.
</ParamField>

<ParamField type="string">
  Organization containing the secret set. If not provided, uses the current
  organization from your configuration.
</ParamField>

## delete

Deletes an entire secret set.

**Usage:**

```shell theme={null}
pipecat cloud secrets delete [ARGS] [OPTIONS]
```

**Arguments:**

<ParamField type="string">
  Name of the secret set to delete. This action is irreversible.
</ParamField>

**Options:**

<ParamField type="boolean">
  Skip confirmations and proceed with the operation.
</ParamField>

<ParamField type="string">
  Organization containing the secret set. If not provided, uses the current
  organization from your configuration.
</ParamField>

## image-pull-secret

Creates credentials for pulling images from private Docker registries.

This command encodes and securely stores your image repository credentials. These credentials are used with the [deploy](./deploy) command when pulling images from private repositories. If you don't provide credentials directly, the command will prompt you for input and offer the option to encode them in base64 format for additional security.

**Usage:**

```shell theme={null}
pipecat cloud secrets image-pull-secret [ARGS] [OPTIONS]
```

**Arguments:**

<ParamField type="string">
  Name of the credentials set to create or modify. Must be a valid string
  identifier.
</ParamField>

<ParamField type="string">
  Host address of the image repository e.g. `https://index.docker.io/v1/`.
</ParamField>

<ParamField type="string">
  Credentials for the image repository in the form of `username:password`.

  Will prompt you for the value if not provided.

  Example:

  ```shell theme={null}
  pipecat cloud secrets image-pull-secret my-registry-creds https://index.docker.io/v1/ my-username:my-password
  ```
</ParamField>

**Options:**

<ParamField type="boolean">
  Encode the credentials in base64 format.
</ParamField>

<ParamField type="string">
  Organization to create the image pull secret in. If not provided, uses the
  current organization from your configuration.
</ParamField>

<ParamField type="string">
  Region where the image pull secret will be stored. If not specified, uses your
  organization's default region (typically `us-west`). Image pull secrets must
  be in the same region as the agents that use them.
</ParamField>

**Examples:**

Create an image pull secret in a specific region:

```shell theme={null}
pipecat cloud secrets image-pull-secret my-registry-creds https://index.docker.io/v1/ --region eu-central
```

***

<Card title="More about secrets" icon="key" href="/deployment/pipecat-cloud/fundamentals/secrets">
  Learn more about managing application secrets
</Card>


# init
Source: https://docs.pipecat.ai/cli/init

Scaffold a new Pipecat project with an interactive wizard or CLI flags

Create a new Pipecat project with guided setup for bot type, transport, AI services, and deployment options. Supports both an interactive wizard and a fully non-interactive mode for automation.

**Usage:**

```shell theme={null}
pipecat init [OPTIONS]
```

**Options:**

<ParamField type="string">
  Output directory where files will be saved. Defaults to current directory.
</ParamField>

<ParamField type="string">
  Project name. Providing this flag triggers non-interactive mode.
</ParamField>

<ParamField type="string">
  Bot type: `web` or `telephony`.
</ParamField>

<ParamField type="string">
  Transport provider. Repeatable for multiple transports (e.g. `-t daily -t smallwebrtc`). Valid values: `daily`, `smallwebrtc`, `twilio`, `telnyx`, `plivo`, `exotel`, `daily_pstn`, `twilio_daily_sip`.
</ParamField>

<ParamField type="string">
  Pipeline mode: `cascade` or `realtime`.
</ParamField>

<ParamField type="string">
  Speech-to-Text service (cascade mode). e.g. `deepgram_stt`, `openai_stt`.
</ParamField>

<ParamField type="string">
  Language model service (cascade mode). e.g. `openai_llm`, `anthropic_llm`.
</ParamField>

<ParamField type="string">
  Text-to-Speech service (cascade mode). e.g. `cartesia_tts`, `elevenlabs_tts`.
</ParamField>

<ParamField type="string">
  Realtime service (realtime mode). e.g. `openai_realtime`, `gemini_live_realtime`.
</ParamField>

<ParamField type="string">
  Video avatar service (web bots only). e.g. `heygen_video`, `tavus_video`, `simli_video`.
</ParamField>

<ParamField type="string">
  Client framework (web bots only): `react`, `vanilla`, or `none`.
</ParamField>

<ParamField type="string">
  Client dev server (when using `--client-framework react`): `vite` or `nextjs`.
</ParamField>

<ParamField type="string">
  Daily PSTN mode (required when transport is `daily_pstn`): `dial-in` or `dial-out`.
</ParamField>

<ParamField type="string">
  Twilio + Daily SIP mode (required when transport is `twilio_daily_sip`): `dial-in` or `dial-out`.
</ParamField>

<ParamField type="boolean">
  Enable audio recording.
</ParamField>

<ParamField type="boolean">
  Enable transcription logging.
</ParamField>

<ParamField type="boolean">
  Enable smart turn-taking. Defaults to `true` for cascade mode, `false` for realtime.
</ParamField>

<ParamField type="boolean">
  Enable video input (web bots only).
</ParamField>

<ParamField type="boolean">
  Enable video output (web bots only).
</ParamField>

<ParamField type="boolean">
  Generate Pipecat Cloud deployment files (Dockerfile, pcc-deploy.toml).
</ParamField>

<ParamField type="boolean">
  Enable Krisp noise cancellation (requires cloud deployment).
</ParamField>

<ParamField type="boolean">
  Enable observability.
</ParamField>

<ParamField type="string">
  Path to a JSON config file. Triggers non-interactive mode. CLI flags override file values.
</ParamField>

<ParamField type="boolean">
  Print the resolved configuration as JSON without generating any files.
</ParamField>

## Interactive Setup

When run without `--name` or `--config`, the CLI guides you through selecting:

* **Bot type and client framework** - Phone, web (Next.js, Vite, Vanilla JS), or mobile
* **Transport provider** - Daily, Twilio, Telnyx, Plivo, Exotel
* **Pipeline mode** - Cascade or Realtime
* **AI services** - STT, LLM, and TTS providers
* **Optional features** - Additional capabilities for your bot
* **Deployment target** - Local development or Pipecat Cloud

## Non-Interactive Mode

When `--name` or `--config` is provided, all configuration is taken from CLI flags or a JSON config file with no interactive prompts. This is useful for automation, scripting, and coding agents.

All required fields must be specified or the command exits with a list of all missing/invalid fields.

## Examples

### Interactive Wizard

```shell theme={null}
pipecat init
```

### Non-Interactive (Cascade)

```shell theme={null}
pipecat init --name my-bot --bot-type web --transport daily \
  --mode cascade --stt deepgram_stt --llm openai_llm --tts cartesia_tts
```

### Non-Interactive (Realtime)

```shell theme={null}
pipecat init --name rt-bot --bot-type web --transport smallwebrtc \
  --mode realtime --realtime openai_realtime
```

### Multiple Transports

```shell theme={null}
pipecat init --name my-bot --bot-type web \
  --transport daily --transport smallwebrtc \
  --mode cascade --stt deepgram_stt --llm openai_llm --tts cartesia_tts
```

### With React Client

```shell theme={null}
pipecat init --name my-bot --bot-type web --transport daily \
  --mode cascade --stt deepgram_stt --llm openai_llm --tts cartesia_tts \
  --client-framework react --client-server vite
```

### Telephony

```shell theme={null}
pipecat init --name call-bot --bot-type telephony --transport twilio \
  --mode cascade --stt deepgram_stt --llm openai_llm --tts cartesia_tts
```

### Dry Run

```shell theme={null}
# Preview resolved config as JSON without generating files
pipecat init --name my-bot --bot-type web --transport daily \
  --mode cascade --stt deepgram_stt --llm openai_llm --tts cartesia_tts \
  --dry-run
```

### From Config File

```shell theme={null}
pipecat init --config project-config.json
```

Sample `project-config.json`:

```json theme={null}
{
  "project_name": "my-bot",
  "bot_type": "web",
  "transports": ["daily"],
  "mode": "cascade",
  "stt_service": "deepgram_stt",
  "llm_service": "openai_llm",
  "tts_service": "cartesia_tts",
  "recording": false,
  "transcription": false,
  "smart_turn": true,
  "deploy_to_cloud": true,
  "enable_krisp": false,
  "enable_observability": false
}
```

CLI flags override any values in the file, so you can use a base config and customize per-run:

```shell theme={null}
pipecat init --config base-config.json --name custom-name --no-deploy-to-cloud
```

### Specify Output Directory

```shell theme={null}
pipecat init --output my-bot
```

## Generated Project Structure

```
mybot/
├── server/                 # Python bot server
│   ├── bot.py              # Main bot implementation
│   ├── pyproject.toml      # Python dependencies
│   ├── .env.example        # Environment variables template
│   ├── Dockerfile          # Container image (if cloud enabled)
│   └── pcc-deploy.toml     # Deployment config (if cloud enabled)
├── client/                 # Web client (if generated)
│   ├── src/
│   ├── package.json
│   └── ...
├── .gitignore
└── README.md               # Project setup instructions
```


# CLI Overview
Source: https://docs.pipecat.ai/cli/overview

Command-line tool for scaffolding, deploying, and monitoring Pipecat bots

<CardGroup>
  <Card title="Scaffold Projects" icon="wand-magic-sparkles" href="/cli/init">
    Create new phone or web/mobile bots with interactive setup
  </Card>

  <Card title="Deploy to Cloud" icon="rocket" href="/cli/cloud/auth">
    Push your bots to production with one command
  </Card>

  <Card title="Monitor Live Bots" icon="chart-line" href="/cli/tail">
    Watch real-time logs, conversations, and metrics
  </Card>
</CardGroup>

## Requirements

* Python 3.10 or later

## Installation

Install the CLI globally using [uv](https://docs.astral.sh/uv/) (recommended) or [pipx](https://pipx.pypa.io/):

```bash theme={null}
uv tool install pipecat-ai-cli
# or
pipx install pipecat-ai-cli
```

Verify installation:

```bash theme={null}
pipecat --version
```

<Tip>All commands can use either `pipecat` or the shorter `pc` alias.</Tip>

## Commands

**[`pipecat init`](/cli/init)** - Scaffold new projects with interactive setup

**[`pipecat tail`](/cli/tail)** - Monitor sessions in real-time with a terminal dashboard

**[`pipecat cloud`](/cli/cloud/auth)** - Deploy and manage bots on Pipecat Cloud

## Getting Help

View help for any command:

```bash theme={null}
pipecat --help
pipecat init --help
pipecat tail --help
pipecat cloud --help
```

## Next Steps

<Card title="Create Your First Bot" icon="rocket" href="/cli/init">
  Scaffold a new project with pipecat init
</Card>


# tail
Source: https://docs.pipecat.ai/cli/tail

A terminal dashboard for monitoring Pipecat sessions in real-time

**Tail** is a terminal dashboard for monitoring your Pipecat sessions in real-time with logs, conversations, metrics, and audio levels all in one place.

With Tail you can:

* 📜 Follow system logs in real time
* 💬 Track conversations as they happen
* 🔊 Monitor user and agent audio levels
* 📈 Keep an eye on service metrics and usage
* 🖥️ Run locally as a pipeline runner or connect to a remote session

**Usage:**

```shell theme={null}
pipecat tail [OPTIONS]
```

**Options:**

<ParamField type="string">
  WebSocket URL to connect to. Defaults to `ws://localhost:9292`.
</ParamField>

## How to Use Tail

* Add `pipecat-ai-cli` to your project's dependencies.

* Update your Pipecat code to include the `TailObserver`:

  ```python theme={null}
  from pipecat_cli.tail import TailObserver

  task = PipelineTask(
      pipeline,
      observers=[TailObserver()]
  )
  ```

* Start the Tail app separately:

  ```bash theme={null}
  # Connect to local session (default)
  pipecat tail

  # Connect to remote session
  pipecat tail --url wss://my-bot.example.com
  ```


# API Reference
Source: https://docs.pipecat.ai/client/android/api-reference





# SDK Introduction
Source: https://docs.pipecat.ai/client/android/introduction

Build Android applications with Pipecat's Kotlin client library

The Pipecat Android SDK provides a Kotlin implementation for building voice and multimodal AI applications on Android. It handles:

* Real-time audio and video streaming
* Bot communication and state management
* Media device handling
* Event handling

## Installation

Add the dependency for your chosen transport to your `build.gradle` file. For example, to use the Daily transport:

```gradle theme={null}
implementation "ai.pipecat:daily-transport:1.0.3"
```

## Example

Here's a simple example using Daily as the transport layer.

```kotlin theme={null}
val callbacks = object : PipecatEventCallbacks() {

    override fun onBackendError(message: String) {
        Log.e(TAG, "Error from backend: $message")
    }

    // ...
}

val options = PipecatClientOptions(callbacks = callbacks)

val client: PipecatClientDaily = PipecatClient(DailyTransport(context), options)

// Kotlin coroutines:
client.startBotAndConnect(startBotParams).await()

// Or, callbacks:
client.startBotAndConnect(startBotParams).withCallback {
    // ...
}
```

## Documentation

<CardGroup>
  <Card title="API Reference" icon="book" href="https://docs-android.rtvi.ai/">
    Complete SDK API documentation
  </Card>

  <Card title="Daily Transport" icon="network-wired" href="./transports/daily">
    WebRTC implementation using Daily
  </Card>
</CardGroup>


# RTVIClient Migration Guide for Android
Source: https://docs.pipecat.ai/client/android/migration-guide

A Guide to migrating from RTVIClient to PipecatClient for Android

This guide covers migrating from the `RTVIClient` to the new `PipecatClient` in an Android application. The new client introduces simplified configuration, modern messaging patterns, and improved function call handling. For an overview of the changes, see the top-level [RTVIClient Migration Guide](/client/migration-guide).

## Key Changes

### 1. Package and Class Names

**Old**

```kotlin theme={null}
import ai.pipecat.client.RTVIClient
import ai.pipecat.client.RTVIClientOptions
import ai.pipecat.client.RTVIClientParams
import ai.pipecat.client.RTVIEventCallbacks
```

**New**

```kotlin theme={null}
import ai.pipecat.client.PipecatClient
import ai.pipecat.client.PipecatClientOptions
// RTVIClientParams has been removed
import ai.pipecat.client.PipecatEventCallbacks

// Type alias for a Pipecat client using the Daily transport
import ai.pipecat.client.daily.PipecatClientDaily
```

### 2. Client and Transport Configuration

Previously, the `RTVIClient` constructor accepted a transport factory parameter, whereas in v1.0.0 the transport is passed in directly.

Parameters such as the connection URL are no longer passed in at construction time. These should be given to `startBot()`, `startBotAndConnect()`, or `connect()`.

**Old**

```kotlin theme={null}
val options = RTVIClientOptions(
    params = RTVIClientParams(
        baseUrl = baseUrl,
        endpoints = RTVIURLEndpoints(),
    )
)

val client = RTVIClient(DailyTransport.Factory(context), callbacks, options)
```

**New**

```kotlin theme={null}
val options = PipecatClientOptions(callbacks = callbacks)
val client = PipecatClient(DailyTransport(context), options)
```

### 3. Connection

Previously, the `connect()` method did a POST request to your bot's HTTP endpoint, which returned the connection details (such as room URL and token), before making the actual connection.

This functionality has been split out into `startBot()`, which provides more flexibility in how the request is performed.

A helper method `startBotAndConnect()` unifies the two steps and is the equivalent of the original `connect()` method. You can call `connect()` directly if you wish to skip the POST step.

**Old**

```kotlin theme={null}
// Uses parameters passed into the constructor
client.connect().await()
```

**New**

```kotlin theme={null}
// Parameters now passed in at connection time
client.startBotAndConnect(
    APIRequest(
        endpoint = url,
        requestData = Value.Object()
    )
).await()
```

### 4. Function Call Handling

Helpers like `LLMHelper` have been removed. You now register function call handlers directly on the `PipecatClient`.

```kotlin theme={null}
client.registerFunctionCallHandler("get_my_current_location") { data, onResult ->
    thread {
        val location = getCurrentLocation(data.args)
        onResult(location)
    }
}
```

### 5. Pipeline Configuration Initialization

Previously, you could provide a pipeline configuration as part of the `RTVIClient` constructor, and it was expected to be in a specific format. Now, if you would like to pass any initial pipeline configurations, you do so as `requestData` added to the endpoint you provide to `startBot()` or `startBotAndConnect()`. In both cases, you would need server-side code to parse and apply these settings, but now you can define the structure and what pieces of configuration you want to send.

Check out [this section of docs](../js/api-reference/messages#connection-time-configuration) for an example, complete with server-side code showing how to initialize the pipeline configuration at connection time.

### 6. LLM Context Updates

Previously, context updates were done via helpers. Now, use `appendToContext()`:

```kotlin theme={null}
client.appendToContext(
    message = LLMContextMessage(
        role = LLMContextMessage.Role.User,
        content = "Tell me a joke.",
        runImmediately = true
    )
)
```

### 7. Pipeline Configuration Updates

Previously, the client supported updating the pipeline configuration using a specific method that took a configuration object in a generic format. Dynamic and predefined configuration updates, however, are a security concern, allowing clients to override settings and potentially abuse API keys. For this reason, it has been removed and most configuration updates need to be handled custom by your application. To do so, you should take advantage of the client-server messaging system, which allows you to send messages to the server and handle responses. This way, you can implement your own logic for updating configurations securely.

New messaging types replace old action/config helpers:

```kotlin theme={null}
client.sendClientMessage(msgType = "set-voice", data = Value.Str("Janice")).await()

val llmVendor = client.sendClientRequest(msgType = "get-llm-vendor").await().data?.content
Log.i("Example", "LLM Vendor: $llmVendor")
```

Check out [this section of docs](../js/api-reference/messages#sending-custom-messages-to-the-server) for a more complete example, along with an example on making a request (`sendClientRequest()`) to wait for a response.

### 8. Disconnecting

```kotlin theme={null}
client.unregisterFunctionCallHandler()
client.disconnect().await()
```

## Breaking Changes

1. **Constructor Params Removed**: No pipeline or endpoint config in constructor.
2. **Helpers Removed**: RTVIClientHelper, LLMHelper are gone.
3. **Configs Removed**: All configuration-related methods, events, and types have been removed: `getConfig()`, `updateConfig()`, `describeConfig()`, `onConfigUpdated`, `onConfigDescribed`, etc.
4. **Actions Removed**: All actions-related methods, events, and types have been removed: `action()`, `describeActions()`, `onActionsAvailable`, etc.
5. **New Messaging Methods**:

   * `appendToContext()`
   * `sendClientRequest()`
   * `sendClientMessage()`
   * `disconnectBot()`
   * `registerFunctionCallHandler()` / `unregisterFunctionCallHandler()` / `unregisterAllFunctionCallHandlers()`


# Daily WebRTC Transport
Source: https://docs.pipecat.ai/client/android/transports/daily

WebRTC implementation for Android using Daily

The Daily transport implementation enables real-time audio and video communication in your Pipecat Android applications using [Daily's](https://www.daily.co/) WebRTC infrastructure.

## Installation

Add the Daily transport dependency to your `build.gradle`:

```gradle theme={null}
implementation "ai.pipecat:daily-transport:1.0.3"
```

## Usage

Create a client using the Daily transport:

```kotlin theme={null}
val callbacks = object : PipecatEventCallbacks() {

    override fun onBackendError(message: String) {
        Log.e(TAG, "Error from backend: $message")
    }

    // ...
}

val options = PipecatClientOptions(callbacks = callbacks)

val client: PipecatClientDaily = PipecatClient(DailyTransport(context), options)

// Kotlin coroutines
client.startBotAndConnect(startBotParams).await()

// Callbacks
client.startBotAndConnect(startBotParams).withCallback {
    // ...
}
```

## Configuration

Your server endpoint should return Daily-specific configuration:

```json theme={null}
{
  "dailyRoom": "https://your-domain.daily.co/room-name",
  "dailyToken": "your-daily-token"
}
```

## Resources

<CardGroup>
  <Card title="Demo" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/simple-chatbot/client/android">
    Simple Chatbot Demo
  </Card>

  <Card title="Source" icon="github" href="https://github.com/pipecat-ai/pipecat-client-android-transports/">
    Client Transports
  </Card>
</CardGroup>

<Card title="Daily Transport Reference" icon="book-open" href="https://docs-android.rtvi.ai/">
  Complete API documentation for the Daily transport implementation
</Card>


# Gemini Live Websocket Transport
Source: https://docs.pipecat.ai/client/android/transports/gemini-websocket

Websocket implementation for Android using Gemini

The Gemini Live Websocket transport implementation enables real-time audio communication with the Gemini Multimodal Live service, using a direct websocket connection.

<Note>
  Transports of this type are designed primarily for development and testing
  purposes. For production applications, you will need to build a server
  component with a server-friendly transport, like the
  [DailyTransport](./daily), to securely handle API keys.
</Note>

## Installation

Add the transport dependency to your `build.gradle`:

```gradle theme={null}
implementation "ai.pipecat:gemini-live-websocket-transport:0.3.7"
```

## Usage

Create a client:

```kotlin theme={null}
val transport = GeminiLiveWebsocketTransport.Factory(context)

val options = RTVIClientOptions(
    params = RTVIClientParams(
        baseUrl = null,
        config = GeminiLiveWebsocketTransport.buildConfig(
            apiKey = "<your Gemini api key>",
            generationConfig = Value.Object(
                "speech_config" to Value.Object(
                    "voice_config" to Value.Object(
                        "prebuilt_voice_config" to Value.Object(
                            "voice_name" to Value.Str("Puck")
                        )
                    )
                )
            ),
            initialUserMessage = "How tall is the Eiffel Tower?"
        )
    )
)

val client = RTVIClient(transport, callbacks, options)

client.start().withCallback {
    // ...
}
```

## Resources

<CardGroup>
  <Card title="Demo" icon="play" href="https://github.com/pipecat-ai/pipecat-client-android-gemini-live-websocket-demo">
    Simple Chatbot Demo
  </Card>

  <Card title="Source" icon="github" href="https://github.com/pipecat-ai/pipecat-client-android-transports/">
    Client Transports
  </Card>
</CardGroup>

<Card title="Pipecat Android Client Reference" icon="book-open" href="https://docs-android.rtvi.ai/">
  Complete API documentation for the Pipecat Android client.
</Card>


# OpenAI Realtime WebRTC Transport
Source: https://docs.pipecat.ai/client/android/transports/openai-webrtc

WebRTC implementation for Android using OpenAI

The OpenAI Realtime WebRTC transport implementation enables real-time audio communication with the OpenAI Realtime service, using a direct WebRTC connection.

## Installation

Add the transport dependency to your `build.gradle`:

```gradle theme={null}
implementation "ai.pipecat:openai-realtime-webrtc-transport:0.3.7"
```

## Usage

Create a client:

```kotlin theme={null}
val transport = OpenAIRealtimeWebRTCTransport.Factory(context)

val options = RTVIClientOptions(
    params = RTVIClientParams(
        baseUrl = null,
        config = OpenAIRealtimeWebRTCTransport.buildConfig(
            apiKey = apiKey,
            initialMessages = listOf(
                LLMContextMessage(role = "user", content = "How tall is the Eiffel Tower?")
            ),
            initialConfig = OpenAIRealtimeSessionConfig(
                voice = "ballad",
                turnDetection = Value.Object("type" to Value.Str("semantic_vad")),
                inputAudioNoiseReduction = Value.Object("type" to Value.Str("near_field")),
                inputAudioTranscription = Value.Object("model" to Value.Str("gpt-4o-transcribe"))
            )
        )
    )
)

val client = RTVIClient(transport, callbacks, options)

client.start().withCallback {
    // ...
}
```

## Resources

<CardGroup>
  <Card title="Demo" icon="play" href="https://github.com/pipecat-ai/pipecat-client-android-openai-realtime-webrtc-demo">
    Simple Chatbot Demo
  </Card>

  <Card title="Source" icon="github" href="https://github.com/pipecat-ai/pipecat-client-android-transports/">
    Client Transports
  </Card>
</CardGroup>

<Card title="Pipecat Android Client Reference" icon="book-open" href="https://docs-android.rtvi.ai/">
  Complete API documentation for the Pipecat Android client.
</Card>


# Small WebRTC Transport
Source: https://docs.pipecat.ai/client/android/transports/small-webrtc

WebRTC implementation for Android

The Small WebRTC transport implementation enables real-time audio communication with the Small WebRTC Pipecat transport, using a direct WebRTC connection.

## Installation

Add the transport dependency to your `build.gradle`:

```gradle theme={null}
implementation "ai.pipecat:small-webrtc-transport:0.3.7"
```

## Usage

Create a client:

```kotlin theme={null}
val transport = SmallWebRTCTransport.Factory(context, baseUrl)

val options = RTVIClientOptions(
    params = RTVIClientParams(baseUrl = null),
    enableMic = true,
    enableCam = true
)

val client = RTVIClient(transport, callbacks, options)

client.start().withCallback {
    // ...
}
```

## Resources

<CardGroup>
  <Card title="Demo" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/p2p-webrtc/video-transform/client/android">
    Demo App
  </Card>

  <Card title="Source" icon="github" href="https://github.com/pipecat-ai/pipecat-client-android-transports/">
    Client Transports
  </Card>
</CardGroup>

<Card title="Pipecat Android Client Reference" icon="book-open" href="https://docs-android.rtvi.ai/">
  Complete API documentation for the Pipecat Android client.
</Card>


# SDK Introduction
Source: https://docs.pipecat.ai/client/c++/introduction

Build native applications with Pipecat’s C++ client library

The Pipecat C++ SDK provides a native implementation for building voice and multimodal AI applications. It supports:

* Linux (`x86_64` and `aarch64`)
* macOS (`aarch64`)
* Windows (`x86_64`)

## Dependencies

### libcurl

The SDK uses [libcurl](https://curl.se/libcurl/) for HTTP requests.

<Tabs>
  <Tab title="Linux">
    ```bash theme={null}
    sudo apt-get install libcurl4-openssl-dev
    ```
  </Tab>

  <Tab title="macOS">
    On macOS `libcurl` is already included so there is nothing to install.
  </Tab>

  <Tab title="Windows">
    On Windows we use [vcpkg](https://vcpkg.io/en/) to install dependencies. You
    need to set it up following one of the
    [tutorials](https://learn.microsoft.com/en-us/vcpkg/get_started/get-started).

    The `libcurl` dependency will be automatically downloaded when building.
  </Tab>
</Tabs>

## Installation

Clone the SDK:

<Tabs>
  <Tab title="Linux/macOS/Windows">
    ```bash theme={null}
    git clone https://github.com/pipecat-ai/pipecat-client-cxx
    cd pipecat-client-cxx
    ```
  </Tab>
</Tabs>

Build the SDK using CMake:

<Tabs>
  <Tab title="Linux/macOS">
    ```bash theme={null}
    cmake . -G Ninja -Bbuild -DCMAKE_BUILD_TYPE=Release
    ninja -C build
    ```
  </Tab>

  <Tab title="Windows">
    ```bash theme={null}
    # Initialize Visual Studio environment
    "C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Auxiliary\Build\vcvarsall.bat" amd64
    # Configure and build
    cmake . -Bbuild --preset vcpkg
    cmake --buildbuild --config Release
    ```
  </Tab>
</Tabs>

### Cross-compilation

For Linux aarch64:

```bash theme={null}
cmake . -G Ninja -Bbuild -DCMAKE_TOOLCHAIN_FILE=aarch64-linux-toolchain.cmake -DCMAKE_BUILD_TYPE=Release
ninja -C build
```

## Documentation

<CardGroup>
  <Card title="API Reference" icon="book" href="https://github.com/pipecat-ai/pipecat-client-cxx">
    Complete SDK API documentation
  </Card>

  <Card title="Daily Transport" icon="network-wired" href="./transport">
    WebRTC implementation using Daily
  </Card>
</CardGroup>


# Daily WebRTC Transport
Source: https://docs.pipecat.ai/client/c++/transport

WebRTC implementation for C++ using Daily

The Daily transport implementation enables real-time audio and video communication in your Pipecat C++ applications using [Daily's](https://www.daily.co/) WebRTC infrastructure.

## Dependencies

### Daily Core C++ SDK

Download the [Daily Core C++ SDK](https://github.com/daily-co/daily-core-sdk) from the [available releases](https://github.com/daily-co/daily-core-sdk/releases) for your platform and set:

```bash theme={null}
export DAILY_CORE_PATH=/path/to/daily-core-sdk
```

### Pipecat C++ SDK

Build the base [Pipecat C++ SDK](https://github.com/pipecat-ai/pipecat-client-cxx-daily) first and set:

```bash theme={null}
export PIPECAT_SDK_PATH=/path/to/pipecat-client-cxx
```

## Building

First, set a few environment variables:

```bash theme={null}
PIPECAT_SDK_PATH=/path/to/pipecat-client-cxx
DAILY_CORE_PATH=/path/to/daily-core-sdk
```

Then, build the project:

<Tabs>
  <Tab title="Linux/macOS">
    ```bash theme={null}
    cmake . -G Ninja -Bbuild -DCMAKE_BUILD_TYPE=Release
    ninja -C build
    ```
  </Tab>

  <Tab title="Windows">
    ```bash theme={null}
    # Initialize Visual Studio environment
    "C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Auxiliary\Build\vcvarsall.bat" amd64

    # Configure and build
    cmake . -Bbuild --preset vcpkg
    cmake --build build --config Release
    ```
  </Tab>
</Tabs>

## Examples

<CardGroup>
  <Card title="Basic Client" icon="code" href="https://github.com/pipecat-ai/pipecat-client-cxx-daily/blob/main/examples/c++">
    Simple C++ implementation example
  </Card>

  <Card title="Audio Client" icon="waveform" href="https://github.com/pipecat-ai/pipecat-client-cxx-daily/blob/main/examples/c++-portaudio">
    C++ client with PortAudio support
  </Card>

  <Card title="Node.js Server" icon="server" href="https://github.com/pipecat-ai/pipecat-client-cxx-daily/blob/main/examples/server">
    Example Node.js proxy implementation
  </Card>
</CardGroup>


# Client SDKs
Source: https://docs.pipecat.ai/client/introduction

Client libraries for building real-time AI applications with Pipecat

<Note>
  All Client SDKs have transitioned to v1.0, which uses a new, simpler API
  design. For guidance in transitioning to the new API, please refer to the
  migration guide for each platform. If you have any questions or need
  assistance, please reach out to us on [Discord](https://discord.gg/pipecat).
</Note>

Pipecat provides client SDKs for multiple platforms, all implementing the RTVI (Real-Time Voice and Video Inference) standard. These SDKs make it easy to build real-time AI applications that can handle voice, video, and text interactions.

<CardGroup>
  <Card title="Javascript" icon="JS" href="/client/js/introduction">
    Pipecat JS SDK
  </Card>

  <Card title="React" icon="react" href="/client/react/introduction">
    Pipecat React SDK
  </Card>

  <Card title="React Native" icon="react" href="/client/react-native/introduction">
    Pipecat React Native SDK
  </Card>

  <Card title="Swift" icon="swift" href="/client/ios/introduction">
    Pipecat iOS SDK
  </Card>

  <Card title="Kotlin" icon="android" href="/client/android/introduction">
    Pipecat Android SDK
  </Card>

  <Card title="C++" icon="C" href="/client/c++/introduction">
    Pipecat C++ SDK
  </Card>
</CardGroup>

## Core Functionality

All Pipecat client SDKs provide:

<CardGroup>
  <Card title="Media Management" icon="video">
    Handle device inputs and media streams for audio and video
  </Card>

  <Card title="Bot Integration" icon="robot">
    Configure and communicate with your Pipecat bot
  </Card>

  <Card title="Session Management" icon="arrows-rotate">
    Manage connection state and error handling
  </Card>
</CardGroup>

## Core Types

### PipecatClient

The main class for interacting with Pipecat bots. It is the primary type you will interact with.

### Transport

The `PipecatClient` wraps a Transport, which defines and provides the underlying connection mechanism (e.g., WebSocket, WebRTC). Your Pipecat pipeline will contain a corresponding transport.

### RTVIMessage

Represents a message sent to or received from a Pipecat bot.

## Simple Usage Examples

<Tabs>
  <Tab title="Connecting to a Bot">
    Establish ongoing connections via WebSocket or WebRTC for:

    * Live voice conversations
    * Real-time video processing
    * Continuous interactions

    <CodeGroup>
      ```javascript javascript theme={null}
      // Example: Establishing a real-time connection
      import { RTVIEvent, RTVIMessage, PipecatClient } from "@pipecat-ai/client-js";
      import { DailyTransport } from "@pipecat-ai/daily-transport";

      const pcClient = new PipecatClient({
        transport: new DailyTransport(),
        enableMic: true,
        enableCam: false,
        enableScreenShare: false,
        callbacks: {
          onBotConnected: () => {
            console.log("[CALLBACK] Bot connected");
          },
          onBotDisconnected: () => {
            console.log("[CALLBACK] Bot disconnected");
          },
          onBotReady: () => {
            console.log("[CALLBACK] Bot ready to chat!");
          },
        },
      });

      try {
        // Below, we use a REST endpoint to fetch connection credentials for our
        // Daily Transport. Alternatively, you could provide those credentials
        // directly to `connect()`.
        await pcClient.startBotAndConnect({
          endpoint: "https://your-connect-end-point-here/connect",
        });
      } catch (e) {
        console.error(e.message);
      }

      // Events (alternative approach to constructor-provided callbacks)
      pcClient.on(RTVIEvent.Connected, () => {
        console.log("[EVENT] User connected");
      });
      pcClient.on(RTVIEvent.Disconnected, () => {
        console.log("[EVENT] User disconnected");
      });
      ```

      ```jsx react theme={null}
      // Example: Using PipecatClient in a React component
      import { PipecatClient } from "@pipecat-ai/client-js";
      import {
        PipecatClientProvider,
        PipecatClientAudio,
        usePipecatClient,
        useRTVIClientEvent,
      } from "@pipecat-ai/client-react";
      import { DailyTransport } from "@pipecat-ai/daily-transport";

      // Create the client instance
      const client = new PipecatClient({
        transport: new DailyTransport(),
        enableMic: true,
      });

      // Root component wraps the app with the provider
      function App() {
        return (
          <PipecatClientProvider client={client}>
            <VoiceBot />
            <PipecatClientAudio />
          </PipecatClientProvider>
        );
      }

      // Component using the client
      function VoiceBot() {
        const client = usePipecatClient();

        const handleClick = async () => {
          await client.startBotAndConnect({
            endpoint: `${process.env.PIPECAT_API_URL || "/api"}/connect`
          });
        };

        return (
          <button onClick={handleClick}>Start Conversation</button>;
        );
      }

      function EventListener() {
        useRTVIClientEvent(
          RTVIEvent.Connected,
          useCallback(() => {
            console.log("[EVENT] User connected");
          }, [])
        );
      }
      ```
    </CodeGroup>
  </Tab>

  <Tab title="Custom Messaging">
    Send custom messages and handle responses from your bot. This is useful for:

    * Running server-side functionality
    * Triggering specific bot actions
    * Querying the server
    * Responding to server requests

    <CodeGroup>
      ```javascript javascript theme={null}
      import { PipecatClient } from "@pipecat-ai/client-js";

      const pcClient = new PipecatClient({
        transport: new DailyTransport(),
        callbacks: {
          onBotConnected: () => {
            pcClient
              .sendClientRequest("get-language")
              .then((response) => {
                console.log("[CALLBACK] Bot using language:", response);
                if (response !== preferredLanguage) {
                  pcClient.sendClientMessage("set-language", {
                    language: preferredLanguage,
                  });
                }
              })
              .catch((error) => {
                console.error("[CALLBACK] Error getting language:", error);
              });
          },
          onServerMessage: (message) => {
            console.log("[CALLBACK] Received message from server:", message);
          },
        },
      });
      // Here we have obtained the connection details separately and pass them
      // directly to connect().
      // Alternatively, you can use a connection endpoint to fetch these details
      // using `startBotAndConnect()`.
      await pcClient.connect({
        url: "https://your-daily-room-url",
        token: "your-daily-token",
      });
      ```

      ```jsx react theme={null}
      // Example: Messaging in a React application
      import { useCallback } from "react";
      import { RTVIEvent, TransportState } from "@pipecat-ai/client-js";
      import { usePipecatClient, useRTVIClientEvent } from "@pipecat-ai/client-react";

      function EventListener() {
        const pcClient = usePipecatClient();
        useRTVIClientEvent(
          RTVIEvent.BotConnected,
          useCallback(() => {
            pcClient
              .sendClientRequest("get-language")
              .then((response) => {
                console.log("[CALLBACK] Bot using language:", response);
                if (response !== preferredLanguage) {
                  pcClient.sendClientMessage("set-language", {
                    language: preferredLanguage,
                  });
                }
              })
              .catch((error) => {
                console.error("[CALLBACK] Error getting language:", error);
              });
          }, [])
        );
        useRTVIClientEvent(
          RTVIEvent.ServerMessage,
          useCallback((data) => {
            console.log("[CALLBACK] Received message from server:", data);
          }, [])
        );
      }
      ```
    </CodeGroup>
  </Tab>
</Tabs>

## About RTVI

Pipecat's client SDKs implement the RTVI (Real-Time Voice and Video Inference) standard, an open specification for real-time AI inference. This means:

* Your code can work with any RTVI-compatible inference service
* You get battle-tested tooling for real-time multimedia handling
* You can easily set up development and testing environments

## Next Steps

Get started by trying out examples:

<CardGroup>
  <Card title="Simple Chatbot Example" icon="robot" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/simple-chatbot">
    Complete client-server example with both bot backend (Python) and frontend
    implementation (JS, React, React Native, iOS, and Android).
  </Card>

  <Card title="More Examples" icon="code" href="https://github.com/pipecat-ai/pipecat-examples">
    Explore our full collection of example applications and implementations
    across different platforms and use cases.
  </Card>
</CardGroup>


# API Reference
Source: https://docs.pipecat.ai/client/ios/api-reference





# SDK Introduction
Source: https://docs.pipecat.ai/client/ios/introduction

Build iOS applications with Pipecat’s Swift client library

The Pipecat iOS SDK provides a Swift implementation for building voice and multimodal AI applications on iOS. It handles:

* Real-time audio streaming
* Bot communication and state management
* Media device handling
* Configuration management
* Event handling

## Installation

Add the SDK to your project using Swift Package Manager:

```swift theme={null}
// Core SDK
.package(url: "https://github.com/pipecat-ai/pipecat-client-ios.git", from: "1.0.0"),

// Daily transport implementation
.package(url: "https://github.com/pipecat-ai/pipecat-client-ios-daily.git", from: "1.0.0"),
```

Then add the dependencies to your target:

```swift theme={null}
.target(name: "YourApp", dependencies: [
    .product(name: "PipecatClientIOS", package: "pipecat-client-ios")
    .product(name: "PipecatClientIOSDaily", package: "pipecat-client-ios-daily")
]),
```

## Example

Here's a simple example using Daily as the transport layer:

```swift theme={null}
import PipecatClientIOS
import PipecatClientIOSDaily

let pipecatClientOptions = PipecatClientOptions.init(
    transport: DailyTransport.init(),
    enableMic: currentSettings.enableMic,
    enableCam: false,
)
self.pipecatClientIOS = PipecatClient.init(
    options: pipecatClientOptions
)
let startBotParams = APIRequest.init(endpoint: URL(string: $PIPECAT_API_URL + "/connect")!)
self.pipecatClientIOS?.startBotAndConnect(startBotParams: startBotParams) { (result: Result<DailyTransportConnectionParams, AsyncExecutionError>) in
    switch result {
    case .failure(let error):
        // handle error
    case .success(_):
        // handle success
    }
}
```

## Documentation

<CardGroup>
  <Card title="API Reference" icon="book" href="https://docs-ios.pipecat.ai/">
    Complete SDK API documentation
  </Card>

  <Card title="Source" icon="github" href="https://github.com/pipecat-ai/pipecat-client-ios/">
    Pipecat Client iOS
  </Card>
</CardGroup>

<CardGroup>
  <Card title="Demo" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/simple-chatbot/client/ios">
    Simple Chatbot Demo
  </Card>

  <Card title="Daily Transport" icon="network-wired" href="./transports/daily">
    WebRTC implementation using Daily
  </Card>
</CardGroup>


# RTVIClient Migration Guide for IOS
Source: https://docs.pipecat.ai/client/ios/migration-guide

A Guide to migrating from RTVIClient to PipecatClient for iOS

This guide covers migrating from the `RTVIClient` to the new `PipecatClient` in an iOS (Swift) application. The new client introduces simplified configuration, modern messaging patterns, and improved function call handling. For an overview of the changes, see the top-level [RTVIClient Migration Guide](/client/migration-guide).

## Key Changes

### 1. Package and Class Names

**Old**

```swift theme={null}
import RTVIClient
import RTVIClientIOSDaily
```

**New**

```swift theme={null}
import PipecatClientIOS
import PipecatClientIOSDaily
```

### 2. Client and Transport Configuration

Previously, the client constructor accepted parameters like pipeline or endpoint configuration. Now, the configuration is passed explicitly to the `startBotAndConnect()` or `connect()` methods, and the constructor takes only transport options.

**Old**

```swift theme={null}
let client = RTVIClient(params: ...)
```

**New**

```swift theme={null}
let pipecatClientOptions = PipecatClientOptions(
    transport: DailyTransport(),
    enableMic: true,
    enableCam: false
)

let client = PipecatClient(options: pipecatClientOptions)
```

### 3. Connection Method

The connection process is now explicit and supports modern async handling.

**Old**

```swift theme={null}
client.connect()
```

**New**

```swift theme={null}
let startBotParams = APIRequest(endpoint: URL(string: "https://your-server/connect")!)
client.startBotAndConnect(startBotParams: startBotParams) { result in
    switch result {
    case .success(let connectionParams):
        print("Connected successfully: \(connectionParams)")
    case .failure(let error):
        print("Connection failed: \(error)")
    }
}
```

### 4. Function Call Handling

Helpers like `LLMHelper` have been removed. You now register function call handlers directly on the `PipecatClient`.

**Old**

```swift theme={null}
let llmHelper = LLMHelper()
client.registerHelper("llm", llmHelper)
llmHelper?.delegate = self

extension CallContainerModel: LLMHelperDelegate {
    enum ToolsFunctions: String {
        case getMyCurrentLocation = "get_my_current_location"
        case setRestaurantLocation = "set_restaurant_location"
    }
    func onLLMFunctionCall(functionCallData: LLMFunctionCallData, onResult: ((Value) async -> Void)) async {
        var result = Value.object([:])
        if let selectedFunction = ToolsFunctions(rawValue: functionCallData.functionName) {
            // Use a switch to handle the different enum cases
            switch selectedFunction {
            case .getMyCurrentLocation:
                result = await self.getCurrentLocation()
            case .setRestaurantLocation:
                self.handleRestaurantLocation(restaurantInfo: functionCallData.args)
                result = .string("success")
            }
        } else {
            print("Invalid function received \(functionCallData.functionName)")
        }
        await onResult(result)
    }
}

```

**New**

```swift theme={null}
client.registerFunctionCallHandler(functionName: "get_my_current_location") { functionCallData, onResult in
    let location = await getCurrentLocation()
    await onResult(location)
}

client.registerFunctionCallHandler(functionName: "set_restaurant_location") { functionCallData, onResult in
    handleRestaurantLocation(functionCallData.args)
    await onResult(.string("success"))
}
```

### 5. Pipeline Configuration Initialization

Previously, you could provide a pipeline configuration as part of the `RTVIClient` constructor and it was expected to be in a specific format. Now, if you would like to pass any initial pipeline configurations, you do so as `requestData` added to the endpoint you provide to `startBot()` or `startBotAndConnect()`. In both cases, you would need server-side code to parse and apply these settings, but now you can define the structure and what pieces of configuration you want to send.

Check out [this section of docs](../js/api-reference/messages#connection-time-configuration) for an example, complete with server-side code showing how to initialize the pipeline configuration at connection time.

### 6. LLM Context Updates

Previously, context updates were done via helpers. Now, use `appendToContext()`:

```swift theme={null}
client.appendToContext(
    role: "user",
    content: "Tell me a joke.",
    run_immediately: true
)
```

### 7. Pipeline Configuration Updates

Previously, the client supported updating the pipeline configuration using a specific method that took a configuration object in a generic format. Dynamic and predefined configuration updates, however, are a security concern, allowing clients to override settings and potentially abuse API keys. For this reason, it has been removed and most configuration updates need to be handled custom by your application. To do so, you should take advantage of the client-server messaging system, which allows you to send messages to the server and handle responses. This way, you can implement your own logic for updating configurations securely.

New messaging types replace old action/config helpers:

```swift theme={null}
try client.sendClientMessage(msgType: "set-voice", data: .string("Janice"))

let llmVendor = try await client.sendClientRequest(msgType: "get-llm-vendor").d?.asString
print("LLM Vendor: \(llmVendor ?? "")")
```

Check out [this section of docs](../js/api-reference/messages#sending-custom-messages-to-the-server) for a more complete example, along with an example on making a request (`sendClientRequest()`) to wait for a response.

### 8. Disconnecting

```swift theme={null}
client.unregisterAllFunctionCallHandlers()
client.disconnect(completion: nil)
```

## Breaking Changes

1. **Constructor Params Removed**: No pipeline or endpoint config in constructor.
2. **Helpers Removed**: RTVIClientHelper, LLMHelper are gone.
3. **Configs Removed**: All configuration-related methods, events, and types have been removed: `getConfig()`, `updateConfig()`, `describeConfig()`, `onConfigUpdated`, `onConfigDescribed`, etc.
4. **Actions Removed**: All actions-related methods, events, and types have been removed: `action()`, `describeActions()`, `onActionsAvailable`, etc.
5. **New Messaging Methods**:

   * `appendToContext()`
   * `sendClientRequest()`
   * `sendClientMessage()`
   * `disconnectBot()`
   * `registerFunctionCallHandler()` / `unregisterFunctionCallHandler()` / `unregisterAllFunctionCallHandlers()`

## Migration Steps

1. Update imports to `PipecatClientIOS` and `PipecatClientIOSDaily`.
2. Move connection configuration from constructor to `startBotAndConnect()` or `connect()` methods.
3. Replace helper-based function calls with `registerFunctionCallHandler()`.
4. Replace context updates with `appendToContext()`.
5. Replace all action and config related methods, events, and types with `sendClientMessage()` or `sendClientRequest()` for custom messages.
6. Ensure proper disconnection via `unregisterAllFunctionCallHandlers()` and `disconnect()`.

***

For detailed examples, see [Travel Companion iOS Example](https://github.com/pipecat-ai/pipecat-examples/blob/main/travel-companion/client/ios/TravelCompanion/model/CallContainerModel.swift).


# Daily WebRTC Transport
Source: https://docs.pipecat.ai/client/ios/transports/daily

WebRTC implementation for iOS using Daily

The Daily transport implementation enables real-time audio and video communication in your Pipecat iOS applications using [Daily's](https://www.daily.co/) WebRTC infrastructure.

## Installation

Add the Daily transport package to your project:

```swift theme={null}
.package(url: "https://github.com/pipecat-ai/pipecat-client-ios-daily.git", from: "1.0.0")

// Add to your target dependencies
.target(name: "YourApp", dependencies: [
    .product(name: "PipecatClientIOSDaily", package: "pipecat-client-ios-daily")
])
```

## Usage

Create a client using the Daily transport:

```swift theme={null}
import PipecatClientIOS
import PipecatClientIOSDaily

let pipecatClientOptions = PipecatClientOptions.init(
    transport: DailyTransport.init(),
    enableMic: currentSettings.enableMic,
    enableCam: false,
)
self.pipecatClientIOS = PipecatClient.init(
    options: pipecatClientOptions
)
let startBotParams = APIRequest.init(endpoint: URL(string: $PIPECAT_API_URL + "/connect")!)
self.pipecatClientIOS?.startBotAndConnect(startBotParams: startBotParams) { (result: Result<DailyTransportConnectionParams, AsyncExecutionError>) in
    switch result {
    case .failure(let error):
        // handle error
    case .success(_):
        // handle success
    }
}
```

## Configuration

Your server endpoint should return Daily-specific configuration:

```swift theme={null}
// Example server response
{
    "url": "https://your-domain.daily.co/room-name",
    "token": "your-daily-token"
}
```

## API Reference

<CardGroup>
  <Card title="Demo" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/simple-chatbot/client/ios">
    Simple Chatbot Demo
  </Card>

  <Card title="Source" icon="github" href="https://github.com/pipecat-ai/pipecat-client-ios-daily/">
    Daily Transport
  </Card>
</CardGroup>

<Card title="Daily Transport Reference" icon="book-open" href="https://docs-ios.pipecat.ai/PipecatClientIOSDaily/documentation/pipecatclientiosdaily">
  Complete API documentation for the Daily transport implementation
</Card>


# Gemini Live Websocket Transport
Source: https://docs.pipecat.ai/client/ios/transports/gemini-websocket

Websocket implementation for iOS using Gemini

The Gemini Live Websocket transport implementation enables real-time audio communication with the Gemini Multimodal Live service, using a direct websocket connection.

<Note>
  Transports of this type are designed primarily for development and testing
  purposes. For production applications, you will need to build a server
  component with a server-friendly transport, like the
  [DailyTransport](./daily), to securely handle API keys.
</Note>

## Installation

Add the Gemini transport package to your project:

```swift theme={null}
.package(url: "https://github.com/pipecat-ai/pipecat-client-ios-gemini-live-websocket.git", from: "0.3.1"),

// Add to your target dependencies
.target(name: "YourApp", dependencies: [
    .product(name: "PipecatClientIOSGeminiLiveWebSocket", package: "pipecat-client-ios-gemini-live-websocket")
],
```

## Usage

Create a client:

```swift theme={null}
let options: RTVIClientOptions = .init(
    params: .init(config: [
        .init(
            service: "llm",
            options: [
                .init(name: "api_key", value: .string("<your Gemini api key>")),
                .init(name: "initial_messages", value: .array([
                    .object([
                        "role": .string("user"), // "user" | "system"
                        "content": .string("I need your help planning my next vacation.")
                    ])
                ])),
                .init(name: "generation_config", value: .object([
                    "speech_config": .object([
                        "voice_config": .object([
                            "prebuilt_voice_config": .object([
                                "voice_name": .string("Puck") // "Puck" | "Charon" | "Kore" | "Fenrir" | "Aoede"
                            ])
                        ])
                    ])
                ]))
            ]
        )
    ])
)

let client = GeminiLiveWebSocketVoiceClient(options: options)

try await client.start()
```

## API Reference

<CardGroup>
  <Card title="Demo" icon="play" href="https://github.com/pipecat-ai/pipecat-client-ios-gemini-live-websocket-demo">
    Simple Chatbot Gemini Demo
  </Card>

  <Card title="Source" icon="github" href="https://github.com/pipecat-ai/pipecat-client-ios-gemini-live-websocket">
    iOS Gemini Live WebSocket
  </Card>
</CardGroup>

<Card title="Pipecat iOS Client Reference" icon="book-open" href="https://docs-ios.pipecat.ai/PipecatClientIOSGeminiLiveWebSocket/documentation/pipecatclientiosgeminilivewebsocket">
  Complete API documentation for the Gemini transport implementation
</Card>


# OpenAIRealTimeWebRTCTransport
Source: https://docs.pipecat.ai/client/ios/transports/openai-webrtc



## Overview

The OpenAI Realtime WebRTC transport implementation enables real-time audio communication directly with the [OpenAI Realtime API using WebRTC](https://platform.openai.com/docs/guides/realtime-webrtc) voice-to-voice service.
It handles media device management, audio/video streams, and state management for the connection.

## Installation

Add the OpenAI transport package to your project:

```swift theme={null}
.package(url: "https://github.com/pipecat-ai/pipecat-client-ios-openai-realtime.git", from: "0.0.1"),

// Add to your target dependencies
.target(name: "YourApp", dependencies: [
    .product(name: "PipecatClientIOSOpenAIRealtimeWebrtc", package: "pipecat-client-ios-openai-realtime")
],
```

## Usage

Create a client:

```swift theme={null}
let rtviClientOptions = RTVIClientOptions.init(
    enableMic: currentSettings.enableMic,
    enableCam: false,
    params: .init(config: [
        .init(
            service: "llm",
            options: [
                .init(name: "api_key", value: .string(openaiAPIKey)),
                .init(name: "initial_messages", value: .array([
                    .object([
                        "role": .string("user"), // "user" | "system"
                        "content": .string("Start by introducing yourself.")
                    ])
                ])),
                .init(name: "session_config", value: .object([
                    "instructions": .string("You are Chatbot, a friendly and helpful assistant who provides useful information, including weather updates."),
                    "voice": .string("echo"),
                    "input_audio_noise_reduction": .object([
                        "type": .string("near_field")
                    ]),
                    "turn_detection": .object([
                        "type": .string("semantic_vad")
                    ])
                ])),
            ]
        )
    ])
)
self.rtviClientIOS = RTVIClient.init(
    transport: OpenAIRealtimeTransport.init(options: rtviClientOptions),
    options: rtviClientOptions
)
try await rtviClientIOS.start()
```

<Note type="warning">
  Currently, invalid session configurations will result in the OpenAI connection
  being failed.
</Note>

## API Reference

<CardGroup>
  <Card title="Demo" icon="play" href="https://github.com/pipecat-ai/pipecat-client-ios-openai-realtime-webrtc-demo">
    Simple Chatbot OpenAI Demo
  </Card>

  <Card title="Source" icon="github" href="https://github.com/pipecat-ai/pipecat-client-ios-openai-realtime-webrtc">
    iOS OpenAI Realtime WebRTC
  </Card>
</CardGroup>

<Card title="Pipecat iOS Client Reference" icon="book-open" href="https://docs-ios.pipecat.ai/PipecatClientIOSOpenAIRealtimeWebrtc/documentation/pipecatclientiosopenairealtimewebrtc">
  Complete API documentation for the OpenAI transport implementation
</Card>


# SmallWebRTCTransport
Source: https://docs.pipecat.ai/client/ios/transports/small-webrtc

A lightweight WebRTC transport for peer-to-peer connections with Pipecat for iOS

`SmallWebRTCTransport` enables peer-to-peer WebRTC connections between clients and your Pipecat application. It implements bidirectional audio and video streaming using WebRTC for real-time communication.

This transport is intended for lightweight implementations. It expects your Pipecat server to include the corresponding [`SmallWebRTCTransport` server-side](/server/services/transport/small-webrtc) implementation.

## Installation

Add the `SmallWebRTCTransport` package to your project:

```swift theme={null}
.package(url: "https://github.com/pipecat-ai/pipecat-client-ios-small-webrtc.git", from: "0.0.1")

// Add to your target dependencies
.target(name: "YourApp", dependencies: [
    .product(name: "PipecatClientIOSSmallWebrtc", package: "pipecat-client-ios-small-webrtc")
])
```

## Usage

Create a client using the `SmallWebRTCTransport`:

```swift theme={null}
import PipecatClientIOS
import PipecatClientIOSSmallWebrtc

let rtviClientOptions = RTVIClientOptions.init(
    enableMic: currentSettings.enableMic,
    enableCam: currentSettings.enableCam,
    params: RTVIClientParams(
        config: [
            .init(
                service: SmallWebRTCTransport.SERVICE_NAME,
                options: [
                    .init(name: "server_url", value: .string($PIPECAT_SERVER_URL))
                ]
            )
        ]
    )
)
self.rtviClientIOS = RTVIClient.init(
    transport: SmallWebRTCTransport.init(options: rtviClientOptions),
    options: rtviClientOptions
)
self.rtviClientIOS?.start() { result in
    switch result {
    case .failure(let error):
        // handle error
    case .success(_):
        // handle success
    }
}
```

## API Reference

<CardGroup>
  <Card title="Demo" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/p2p-webrtc/video-transform/client/ios">
    Video transform Demo
  </Card>

  <Card title="Source" icon="github" href="https://github.com/pipecat-ai/pipecat-client-ios-small-webrtc/">
    Small WebRTC transport
  </Card>
</CardGroup>

<Card title="SmallWebRTCTransport Reference" icon="book-open" href="https://docs-ios.pipecat.ai/PipecatClientIOSSmallWebrtc/documentation/pipecatclientiossmallwebrtc">
  Complete API documentation for the Small WebRTC transport implementation
</Card>


# Callbacks and events
Source: https://docs.pipecat.ai/client/js/api-reference/callbacks



The Pipecat JavaScript client listens for messages and events from the bot via the transport layer. This allows you to respond to changes in state, errors, and other events. The client implements the RTVI standard for these communications.

## Event Handling Options

You can handle events in two ways:

### 1. Callbacks

Define handlers in the client constructor:

```typescript theme={null}
const pcClient = new PipecatClient({
  callbacks: {
    onBotReady: () => console.log("Bot ready via callback"),
    // ... other callbacks
  },
});
```

### 2. Event Listeners

Add handlers using the event emitter pattern:

```typescript theme={null}
pcClient.on(RTVIEvent.BotReady, () => console.log("Bot ready via event"));
```

<Note>
  Events and callbacks provide the same functionality. Choose the pattern that
  best fits your application's architecture.
</Note>

## Callbacks

### State and connectivity

<ParamField>
  Local user successfully established a connection to the transport.
</ParamField>

<ParamField>
  Local user disconnected from the transport, either intentionally by calling
  `pcClient.disconnect()` or due to an error.
</ParamField>

<ParamField type="state:TransportState">
  Provides a `TransportState` string representing the connectivity state of the
  local client. See [transports](../transports/transport) for state explanation.
</ParamField>

<ParamField type="botResponse: unknown">
  A call to [`startBot()`](./client-methods#startbot) (i.e. a pre-connection
  REST endpoint) was successful and the bot should now be started or in the
  process of starting. The callback receives any data returned from your
  endpoint.
</ParamField>

<ParamField type="botReadyData:BotReadyData">
  The bot has been instantiated, its pipeline is configured, and it is receiving
  user media and interactions. This method is passed a `BotReadyData` object,
  which contains the RTVI `version` number. Since the bot is remote and may be
  using a different version of RTVI than the client, you can use the passed
  `version` string to check for compatibility.
</ParamField>

<ParamField>
  Bot connected to the transport and is configuring. Note: bot connectivity does
  not infer that its pipeline is yet ready to run. Please use `onBotReady`
  instead.
</ParamField>

<ParamField type="participant: Participant">
  Bot disconnected from the transport. This may occur due to session expiry, a
  pipeline error or for any reason the server deems the session over.
</ParamField>

<ParamField type="participant: Participant">
  A non-bot participant joined the session.
</ParamField>

<ParamField type="participant: Participant">
  A non-bot participant left the session. Note: excluded local participant.
</ParamField>

### Messages and errors

<ParamField type="data:any">
  Receives custom messages sent from the server to the client. This provides a
  generic channel for server-to-client communication. The data structure is
  flexible and defined by the server implementation.
</ParamField>

<ParamField type="message:RTVIMessage">
  Response error when an action fails or an unknown message type is sent from
  the client.
</ParamField>

<ParamField type="message:RTVIMessage">
  Error signalled by the bot. This could be due to a malformed config update or
  an unknown action dispatch or the inability to complete a client request. The
  message parameter is of type `error` and matches [the RTVI
  standard](/client/rtvi-standard#error-%F0%9F%A4%96). Its `data` field includes
  a `message` string that describes the error and a `fatal` boolean indicating
  if the error is unrecoverable and resulted in a bot disconnection. If `fatal`
  is true, the client will automatically disconnect.
</ParamField>

### Media and devices

<ParamField type="mics:MediaDeviceInfo[]">
  Lists available local media microphone devices. Triggered when a new device
  becomes available, a device is removed, or in response to
  `pcClient.initDevices()`.
</ParamField>

<ParamField type="cams:MediaDeviceInfo[]">
  Lists available local media camera devices. Triggered when a new device
  becomes available, a device is removed, or in response to
  `pcClient.initDevices()`.
</ParamField>

<ParamField type="speakers:MediaDeviceInfo[]">
  Lists available local speaker devices. Triggered when a new device becomes
  available, a device is removed, or in response to `pcClient.initDevices()`.
</ParamField>

<ParamField type="mic:MediaDeviceInfo">
  User selected a new microphone as their selected/active device.
</ParamField>

<ParamField type="cam:MediaDeviceInfo">
  User selected a new camera as their selected/active device.
</ParamField>

<ParamField type="speaker:MediaDeviceInfo">
  User selected a new speaker as their selected/active device.
</ParamField>

<ParamField type="error:DeviceError">
  Error related to media devices, such as camera or microphone issues. This
  could be due to permissions, device unavailability, or other related problems.
  See the [DeviceError](./errors#deviceerror) section for more details about the
  return type.
</ParamField>

<ParamField type="track: MediaStreamTrack, participant:Participant">
  Media track from a local or remote participant/bot was started and playable.
  Can be either an audio or video track.
</ParamField>

<ParamField type="track: MediaStreamTrack, participant:Participant">
  Media track from a local or remote participant/bot was stopped and no longer
  playable.
</ParamField>

<ParamField type="track: MediaStreamTrack, participant:Participant">
  Media track from a local or remote participant's screenshare was started and
  playable. Can be either an audio or video track.
</ParamField>

<ParamField type="track: MediaStreamTrack, participant:Participant">
  Media track from a local or remote participant's screenshare was stopped and
  no longer playable.
</ParamField>

### Audio and Voice Activity

<ParamField type="level:number">
  Local audio gain level (0 to 1).
</ParamField>

<ParamField type="level: number, participant: Participant">
  Remote audio gain level (0 to 1). Note: if more than one participant is
  connected to the transport, the `participant` property details the associated
  peer/bot.
</ParamField>

<ParamField>
  The bot started speaking/sending speech audio.
</ParamField>

<ParamField>
  The bot stopped speaking/sending speech audio.
</ParamField>

<ParamField>
  The local user started speaking. This method is more reliable than using audio
  gain and is the result of the bot's VAD (voice activity detection) model. This
  provides a more accurate result in noisy environments.
</ParamField>

<ParamField>
  The local user stopped speaking, indicated by the VAD model.
</ParamField>

### Transcription

<ParamField type="TranscriptData">
  Transcribed local user input (both partial and final).

  Callback receives a `TranscriptData` object:

  <Expandable title="TranscriptData">
    <ParamField type="string">
      The transcribed text.
    </ParamField>

    <ParamField type="boolean">
      Indicates if the text is final (true) or partial (false).
    </ParamField>

    <ParamField type="string">
      The timestamp of the transcription.
    </ParamField>

    <ParamField type="string">
      The ID of the user the transcription is for.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField type="BotOutputData">
  A best-effort stream of the bot's output text, including both spoken and unspoken content. This callback is triggered as the bot aggregates the LLM's response into sentences or other logical text blocks as well as word-by-word during TTS synthesis. The callback receives a `BotOutputData` object:

  <Expandable title="BotOutputData">
    <ParamField type="string">
      The aggregated output text from the bot.
    </ParamField>

    <ParamField type="boolean">
      Indicates if the text has been spoken by the bot.
    </ParamField>

    <ParamField type="'sentence' | 'word' | string">
      The aggregation type used for this output (e.g., "sentence", "code"). "sentence" and "word" are reserved aggregation types defined by the RTVI standard. Other aggregation types may be defined by custom text aggregators used by the server. "word" aggregated outputs are sent at the time of TTS synthesis for real-time word-level streaming and can be used in lieu of `onBotTtsText` if desired.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField type="text:BotLLMTextData">
  <Warning>DEPRECATED in favor of `onBotOutput` in Pipecat version 0.0.95 and client-js version 1.5.0</Warning>
  Finalized bot output text generated by the LLM. Sentence aggregated.
</ParamField>

### Service-specific Events

<ParamField type="BotLLMSearchResponseData">
  Bot LLM search response text generated by the LLM service. This is typically
  used for search or retrieval tasks.

  <Warning>
    Search capabilities are currently only supported by Google Gemini. To take
    advantage of this event, your pipeline must include a
    [`GoogleLLMService`](/server/services/llm/gemini) and your pipeline task
    should include the
    [`GoogleRTVIObserver`](/server/frameworks/rtvi/google-rtvi-observer) in lieu
    of the typical `RTVIObserver`.
  </Warning>

  <Expandable title="BotLLMSearchResponseData">
    <ParamField type="string">
      The search result text.
    </ParamField>

    <ParamField type="string">
      The rendered content of the search result.
    </ParamField>

    <ParamField type="LLMSearchOrigin[]">
      The origins of the search result.

      <Expandable title="LLMSearchOrigin">
        <ParamField type="string">
          The URI of the site where the search result was found.
        </ParamField>

        <ParamField type="string">
          The title of the site where the search result was found.
        </ParamField>

        <ParamField type="LLMSearchResult[]">
          The individual search results.

          <Expandable title="LLMSearchResult">
            <ParamField type="string">
              The text of the search result.
            </ParamField>

            <ParamField type="number[]">
              The confidence scores for the search result.
            </ParamField>
          </Expandable>
        </ParamField>
      </Expandable>
    </ParamField>
  </Expandable>
</ParamField>

<ParamField type="BotLLMTextData">
  Streamed LLM token response text generated by the LLM service.

  <Expandable title="BotLLMTextData">
    <ParamField type="string">
      The text of the LLM response.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField>LLM service inference started.</ParamField>

<ParamField>LLM service inference concluded.</ParamField>

<ParamField type="BotTTSTextData">
  If your TTS service supports streamed responses over sockets, the text
  parameter contains the words from TTS service as they are spoken. If you are
  using a HTTP based TTS service, the text parameter will contain the full text
  of the TTS response.

  <Expandable title="BotTTSTextData">
    <ParamField type="string">
      The text of the LLM response.
    </ParamField>
  </Expandable>
</ParamField>

<ParamField>TTS service started inference.</ParamField>

<ParamField>TTS service inference concluded.</ParamField>

### Other

<ParamField>
  Pipeline mterics data provided by Pipecat. [Learn
  more](/guides/fundamentals/metrics).
</ParamField>

## Events

Each callback described above has a corresponding event that can be listened for using the `.on()` method. This allows you to handle the same functionality using either callbacks or event listeners, depending on your preferred architecture.

Here's the complete reference mapping events to their corresponding callbacks:

### State and connectivity Events

| Event Name              | Callback Name             | Data Type        |
| ----------------------- | ------------------------- | ---------------- |
| `Connected`             | `onConnected`             | -                |
| `Disconnected`          | `onDisconnected`          | -                |
| `TransportStateChanged` | `onTransportStateChanged` | `TransportState` |
| `BotReady`              | `onBotReady`              | `BotReadyData`   |
| `BotConnected`          | `onBotConnected`          | -                |
| `BotDisconnected`       | `onBotDisconnected`       | `Participant`    |
| `ParticipantConnected`  | `onParticipantJoined`     | `Participant`    |
| `ParticipantLeft`       | `onParticipantLeft`       | `Participant`    |

### Message and Error Events

| Event Name      | Callback Name     | Data Type     |
| --------------- | ----------------- | ------------- |
| `ServerMessage` | `onServerMessage` | `any`         |
| `MessageError`  | `onMessageError`  | `RTVIMessage` |
| `Error`         | `onError`         | `RTVIMessage` |

### Media Events

| Event Name             | Callback Name            | Data Type                       |
| ---------------------- | ------------------------ | ------------------------------- |
| `TrackStarted`         | `onTrackStarted`         | `MediaStreamTrack, Participant` |
| `TrackStopped`         | `onTrackStopped`         | `MediaStreamTrack, Participant` |
| `AvailableMicsUpdated` | `onAvailableMicsUpdated` | `MediaDeviceInfo[]`             |
| `AvailableCamsUpdated` | `onAvailableCamsUpdated` | `MediaDeviceInfo[]`             |
| `MicUpdated`           | `onMicUpdated`           | `MediaDeviceInfo`               |
| `CamUpdated`           | `onCamUpdated`           | `MediaDeviceInfo`               |
| `SpeakerUpdated`       | `onSpeakerUpdated`       | `MediaDeviceInfo`               |
| `DeviceError`          | `onDeviceError`          | `DeviceError`                   |

### Audio Activity Events

| Event Name            | Callback Name           | Data Type             |
| --------------------- | ----------------------- | --------------------- |
| `LocalAudioLevel`     | `onLocalAudioLevel`     | `number`              |
| `RemoteAudioLevel`    | `onRemoteAudioLevel`    | `number, Participant` |
| `BotStartedSpeaking`  | `onBotStartedSpeaking`  | -                     |
| `BotStoppedSpeaking`  | `onBotStoppedSpeaking`  | -                     |
| `UserStartedSpeaking` | `onUserStartedSpeaking` | -                     |
| `UserStoppedSpeaking` | `onUserStoppedSpeaking` | -                     |

### Text and Transcription Events

| Event Name          | Callback Name         | Data Type        |
| ------------------- | --------------------- | ---------------- |
| `UserTranscript`    | `onUserTranscript`    | `TranscriptData` |
| `BotOutput`         | `onBotOutput`         | `BotOutputData`  |
| ~~`BotTranscript`~~ | ~~`onBotTranscript`~~ | `BotLLMTextData` |
| `BotLlmText`        | `onBotLlmText`        | `BotLLMTextData` |
| `BotTtsText`        | `onBotTtsText`        | `BotTTSTextData` |

### Service State Events

| Event Name             | Callback Name            | Data Type                  |
| ---------------------- | ------------------------ | -------------------------- |
| `BotLlmSearchResponse` | `onBotLlmSearchResponse` | `BotLLMSearchResponseData` |
| `BotLlmStarted`        | `onBotLlmStarted`        | -                          |
| `BotLlmStopped`        | `onBotLlmStopped`        | -                          |
| `BotTtsStarted`        | `onBotTtsStarted`        | -                          |
| `BotTtsStopped`        | `onBotTtsStopped`        | -                          |

### Other Events

| Event Name | Callback Name | Data Type            |
| ---------- | ------------- | -------------------- |
| `Metrics`  | `onMetrics`   | `PipecatMetricsData` |

## Usage Example

```typescript theme={null}
import { PipecatClient, RTVIEvent } from "@pipecat-ai/client-js";

// Using callbacks
const pcClient = new PipecatClient({
  callbacks: {
    onBotReady: () => console.log("Bot ready via callback"),
    onUserTranscript: (data) => console.log("Transcript:", data.text),
  },
});

// Alternate approach: Using event listeners
pcClient.on(RTVIEvent.BotReady, () => {
  console.log("Bot ready via event");
});
```

## Transport Compatibility

<TransportTable />


# PipecatClient Constructor
Source: https://docs.pipecat.ai/client/js/api-reference/client-constructor

Setting up the PipecatClient

```javascript theme={null}
import { PipecatClient } from "@pipecat-ai/client-js";
```

`PipecatClient` is the primary component for building the client-side portion of a client-bot interaction. It is designed to work with various transport layers, such as WebRTC, WebSockets, or HTTP, allowing you to pick and choose the communication layer that best suits your application while maintaining a consistent API.

<Note>
  When initializing the `PipecatClient`, you must provide a transport instance to the constructor for your chosen protocol or provider. See [Transport](/client/js/transports) for more information. For the purpose of this guide, we'll demonstrate using the [Daily WebRTC transport](/client/js/transports/daily).
</Note>

## Example

```typescript theme={null}
import { RTVIEvent, RTVIMessage, PipecatClient } from "@pipecat-ai/client-js";
import { DailyTransport } from "@pipecat-ai/daily-transport";

const PipecatClient = new PipecatClient({
  transport: new DailyTransport(),
  enableMic: true,
  enableCam: false,
  enableScreenShare: false,
  timeout: 15 * 1000,
  callbacks: {
    onConnected: () => {
      console.log("[CALLBACK] User connected");
    },
    onDisconnected: () => {
      console.log("[CALLBACK] User disconnected");
    },
    onTransportStateChanged: (state: string) => {
      console.log("[CALLBACK] State change:", state);
    },
    onBotConnected: () => {
      console.log("[CALLBACK] Bot connected");
    },
    onBotDisconnected: () => {
      console.log("[CALLBACK] Bot disconnected");
    },
    onBotReady: () => {
      console.log("[CALLBACK] Bot ready to chat!");
    },
  },
});
```

***

## API reference

### transport

<ParamField type="Class<Transport>">
  An instance of the `Transport` type you will use to connect to your bot service (`PipecatClient.connect()`). Transports implement the underlying device management, connectivity, media transmission, and state logic that manage the lifecycle of your session.

  <Note>
    As a best practice, we recommend you construct the transport inline in the client constructor, as opposed to holding a reference to it. Access to the transport is typically unnecessary. For advanced use cases that do require access to the transport, we recommend doing so via the `PipecatClient.transport` property, which provides some additional safeguards.
  </Note>

  ```typescript theme={null}
  import { PipecatClient } from "@pipecat-ai/client-js";
  import { DailyTransport } from "@pipecat-ai/daily-transport";

  const pcClient = new PipecatClient({
    transport: new DailyTransport(),
  });
  ```
</ParamField>

### callbacks

<ParamField type="RTVIEventCallbacks">
  Map of callback functions. See [callbacks](./callbacks).
</ParamField>

### Media Initialization

<ParamField type="boolean">
  Enable user's local microphone device.
</ParamField>

<ParamField type="boolean">
  Enable user's local webcam device. Note: Not all transports support video. Setting this value in that case will have no effect.
</ParamField>

<ParamField type="boolean">
  Enable user's local screen share. Note: Not all transports support screen sharing. Setting this value in that case will have no effect.
</ParamField>


# Client Methods
Source: https://docs.pipecat.ai/client/js/api-reference/client-methods



The Pipecat JavaScript client provides a comprehensive set of methods for managing bot interactions and media handling. These core methods are documented below.

## Session connectivity

### startBot()

`async startBot(startBotParams: APIEndpoint): Promise<TransportConnectionParams | unknown>`

This method hits your server endpoint to start the bot and optionally obtain the connection parameters needed for `connect()` to connect the `Transport`. It returns a Promise that resolves with the response from the server.

<ParamField type="APIEndpoint">
  The `APIEndpoint` object should have the following shape:

  <Expandable title="APIEndpoint">
    <ParamField type="string">
      The URL of the endpoint to connect to. This should be a valid REST endpoint.
    </ParamField>

    <ParamField type="Headers">
      Optional headers to include in the request to the endpoint. This can be used to pass authentication tokens or other necessary headers.
    </ParamField>

    <ParamField type="object">
      Optional request data to include in the request to the endpoint. This can be used to pass additional data to your server-side endpoint. Oftentimes, this is used to pass the initial prompt or other configuration data to initialize the bot.
    </ParamField>

    <ParamField type="number">
      Optional timeout in milliseconds for the request to the endpoint.
    </ParamField>
  </Expandable>
</ParamField>

During the `startBot()` process, the transport state will transition through the states: "authenticating" and "authenticated".

```javascript theme={null}
try {
  await pcClient.startBot({
    endpoint: "/api/start", // Your server endpoint to start the bot
    requestData: {
      initial_prompt: "You are a pirate captain",
      llm_provider: "openai"
    }
  });
} catch (error) {
  console.error("Error starting the bot:", error);
}
```

### connect()

`async connect(connectParams): Promise<BotReadyData>`

This method initiates the connection process, optionally passing parameters that your transport class requires to establish a connection or an endpoint to your server for obtaining those parameters.

<ParamField type="TransportConnectionParams">
  An object containing the `TransportConnectionParams` your Transport expects.

  Check your transport class documentation for the expected shape of `TransportConnectionParams`. For example, the DailyTransport expects a `url` and `token`.

  <Warning>
    In 1.2.0 we deprecated support for passing a `ConnectionEndpoint` object directly to `connect()`. Instead, you should use the `startBot()` or `startBotAndConnect()` methods to fetch connection parameters from your server endpoint and then pass those parameters directly to `connect()`.
  </Warning>
</ParamField>

This method can be try / catched to handle errors at startup:

```typescript theme={null}
try {
  await pcClient.connect({
    webrtcUrl: "http://my-server/api/offer"
  });
} catch (error) {
  console.error("Error connecting to the bot:", error);
}
```

During the connection process, the transport state will transition through the following states: "connecting", "connected", "ready".

<Note>
  Calling `connect()` asynchronously will resolve when the bot and client signal that they are ready. See [messages and events](./messages). If you want to call `connect()` without `await`, you can use the `onBotReady` callback or `BotReady` event to know when you can interact with the bot.
</Note>

<Warning>
  Attempting to call `connect()` when the transport is already in a 'connected' or 'ready' state will throw an error. You should [disconnect](#disconnect) from a session first before attempting to connect again.
</Warning>

### startBotAndConnect()

`async startBotAndConnect(startBotParams: APIEndpoint): Promise<BotReadyData>`

This method combines the functionality of `startBot()` and `connect()`. It first starts the bot by hitting your server endpoint and then connects the transport passing the response from the endpoint to the transport as connection parameters.

```javascript theme={null}
try {
  await pcClient.startBotAndConnect({
    endpoint: "/api/start", // Your server endpoint to start the bot
    requestData: {
      initial_prompt: "You are a pirate captain",
      llm_provider: "openai"
    }
  });
} catch (error) {
  console.error("Error starting up:", error);
}
```

It's equivalent to: `pcClient.startBot(...).then((resp) => pcClient.connect(resp))`.

### disconnect()

`async disconnect(): Promise<void>`

Disconnects from the active session. The transport state will transition to "disconnecting" and then "disconnected".

It is common practice for bots to exit and cleanup when the client disconnects.

```typescript theme={null}
await pcClient.disconnect();
```

### disconnectBot()

`disconnectBot(): void`

Triggers the bot to disconnect from the session, leaving the client connected.

```typescript theme={null}
await pcClient.disconnectBot();
```

## Messages

Custom messaging between the client and the bot. This is useful for sending data to the bot, triggering specific actions, reacting to server events, or querying the server.

For more, see: [messages and events](./messages).

### sendClientMessage()

`sendClientMessage(msgType: string, data?: unknown): void`

Sends a custom message to the bot and does not expect a response. This is useful for sending data to the bot or triggering specific actions.

<ParamField type="string">
  A string identifying the message.
</ParamField>

<ParamField type="unknown">
  Optional data to send with the message. This can be any JSON-serializable object.
</ParamField>

### sendClientRequest()

`async sendClientRequest(msgType: string, data: unknown, timeout?: number): Promise<unknown>`

Sends a custom request to the bot and expects a response. This is useful for querying the server or triggering specific actions that require a response. The method returns a Promise that resolves with the data from response.

<ParamField type="string">
  A string identifying the message.
</ParamField>

<ParamField type="unknown">
  Optional data to send with the message. This can be any JSON-serializable object.
</ParamField>

<ParamField type="number">
  Optional timeout in milliseconds for the request. If the request does not receive a response within this time, it will reject with an RTVIMessage of type `'error-response'`.
</ParamField>

## Devices

### initDevices()

`async initDevices(): Promise<void>`

Initializes the media device selection machinery, based on `enableCam`/`enableMic` selections and defaults (i.e. turns on the local cam/mic). This method can be called before `connect()` to test and switch between camera and microphone sources.

```typescript theme={null}
await pcClient.initDevices();
```

### getAllMics()

`async getAllMics(): Promise<MediaDeviceInfo[]>`

Returns a list of available microphones in the form of [`MediaDeviceInfo[]`](https://developer.mozilla.org/en-US/docs/Web/API/MediaDeviceInfo).

```typescript theme={null}
mic_device_list = pcClient.getAllMics();
```

### getAllCams()

`async getAllCams(): Promise<MediaDeviceInfo[]>`

Returns a list of available cameras in the form of [`MediaDeviceInfo[]`](https://developer.mozilla.org/en-US/docs/Web/API/MediaDeviceInfo).

```typescript theme={null}
cam_device_list = pcClient.getAllCams();
```

### getAllSpeakers()

`async getAllSpeakers(): Promise<MediaDeviceInfo[]>`

Returns a list of available speakers in the form of [`MediaDeviceInfo[]`](https://developer.mozilla.org/en-US/docs/Web/API/MediaDeviceInfo).

```typescript theme={null}
speaker_device_list = pcClient.getAllSpeakers();
```

### selectedMic

`selectedMic: MediaDeviceInfo | {}`

The currently selected microphone, represented as a `MediaDeviceInfo` object. If no microphone is selected, it returns an empty object.

```typescript theme={null}
current_mic = pcClient.selectedMic;
```

### selectedCam

`selectedCam: MediaDeviceInfo | {}`

The currently selected camera, represented as a `MediaDeviceInfo` object. If no camera is selected, it returns an empty object.

```typescript theme={null}
current_cam = pcClient.selectedCam;
```

### selectedSpeaker

`selectedSpeaker: MediaDeviceInfo | {}`

The currently selected speaker, represented as a `MediaDeviceInfo` object. If no speaker is selected, it returns an empty object.

```typescript theme={null}
current_speaker = pcClient.selectedSpeaker;
```

### updateMic()

`updateMic(micId: string): void`

Switches to the microphone identified by the provided `micId`, which should match a `deviceId` in the list returned from [`getAllMics()`](#getAllMics).

<ParamField type="string">
  deviceId
</ParamField>

```typescript theme={null}
pcClient.updateMic(deviceId);
```

### updateCam()

`updateCam(camId: string): void`

Switches to the camera identified by the provided `camId`, which should match a `deviceId` in the list returned from [`getAllCams()`](#getAllCams).

<ParamField type="string">
  deviceId
</ParamField>

```typescript theme={null}
pcClient.updateCam(deviceId);
```

### updateSpeaker()

`updateSpeaker(speakerId: string): void`

Switches to the speaker identified by the provided `speakerId`, which should match a `deviceId` in the list returned from [`getAllSpeakers()`](#getAllSpeakers).

<ParamField type="string">
  deviceId
</ParamField>

```typescript theme={null}
pcClient.updateSpeaker(deviceId);
```

### enableMic(enable: boolean)

`enableMic(enable: boolean): void`

Turn on or off (unmute or mute) the client mic input.

<ParamField type="boolean">
  A boolean indicating whether to enable (`true`) or disable (`false`) the microphone.
</ParamField>

```typescript theme={null}
pcClient.enableMic(true);
```

### enableCam(enable: boolean)

`enableCam(enable: boolean): void`

Turn on or off the client cam input.

<ParamField type="boolean">
  A boolean indicating whether to enable (`true`) or disable (`false`) the camera.
</ParamField>

```typescript theme={null}
pcClient.enableCam(true);
```

### enableScreenShare(enable: boolean)

`enableScreenShare(enable: boolean): void`

Start a screen share from the client's device.

<ParamField type="boolean">
  A boolean indicating whether to enable (`true`) or disable (`false`) screen sharing.
</ParamField>

```typescript theme={null}
pcClient.enableScreenShare(true);
```

### isMicEnabled

`isMicEnabled: boolean`

An accessor to determine if the client's microphone is enabled.

```typescript theme={null}
mic_enabled = pcClient.isMicEnabled;
```

### isCamEnabled

`isCamEnabled: boolean`

An accessor to determine if the client's camera is enabled.

```typescript theme={null}
cam_enabled = pcClient.isCamEnabled;
```

### isSharingScreen

An accessor to determine if the client is sharing their screen.

```typescript theme={null}
screen_sharing = pcClient.isSharingScreen;
```

## Tracks (audio and video)

### tracks()

`tracks(): Tracks`

Returns a `Tracks` object with available `MediaStreamTrack` objects for both the client and the bot.

```typescript theme={null}
live_tracks_list = pcClient.tracks()
```

**Tracks Type**

```typescript theme={null}
{
  local: {
    audio?: MediaStreamTrack;
    video?: MediaStreamTrack;
  },
  bot?: {
    audio?: MediaStreamTrack;
    video?: MediaStreamTrack;
  }
}
```

## Advanced LLM Interactions

### sendText()

`async sendText(content: string, options?: SendTextOptions = {}): void`

A method to append text to the user's context. This is useful for providing text input as an alternative to audio input for the user.

<ParamField type="string">
  The text content to send to the bot.
</ParamField>

<ParamField type="SendTextOptions">
  An optional set of options for how the bot should handle the text input.

  <Expandable title="SendTextOptions">
    <ParamField type="boolean">
      Whether to immediately run the bot with the updated context. If `false`, the context will be updated but the bot will not be run until the next message or action that triggers the bot to run (like the user speaking).
    </ParamField>

    <ParamField type="boolean">
      Whether the bot should respond with audio. If `true`, the bot's response will be processed by TTS and be spoken. If `false`, the bot will bypass the TTS and respond with text only.
    </ParamField>
  </Expandable>
</ParamField>

### appendToContext()

`async appendToContext(context: LLMContextMessage): boolean`

<Warning>
  In 0.0.4, we deprecated `appendToContext` in favor of `send-text` to close a security vulnerability as well as lay the groundwork for proper support of passing other types of context, like images and files.
</Warning>

A method to append data to the bot's context. This is useful for providing additional information or context to the bot during the conversation.

<ParamField type="LLMContextMessage">
  The context to append. This should be an object with the following shape:

  <Expandable title="LLMContextMessage">
    <ParamField type="string">
      The role to append the context to. Currently only "user" or "assistant" are supported.
    </ParamField>

    <ParamField type="unknown">
      The content to append to the context. This can be any JSON-serializable object.
    </ParamField>

    <ParamField type="boolean">
      Whether to immediately run the bot with the updated context. If `false`, the context will be updated but the bot will not be run until the next message or action that triggers the bot to run (like the user speaking).
    </ParamField>
  </Expandable>
</ParamField>

### registerFunctionCallHandler()

`registerFunctionCallHandler(functionName: string, callback: FunctionCallCallback): void`

Registers a function call handler that will be called when the bot requests a function call. This is useful for when the server-side function handler needs information from the client to execute the function call or when the client needs to perform some action based on the running of function call.

<ParamField type="string">
  The name of the function to handle. This should match the function name in the bot's context.
</ParamField>

<ParamField type="FunctionCallCallback">
  `type FunctionCallCallback = (fn: FunctionCallParams) => Promise<LLMFunctionCallResult | void>`

  The callback function to call when the bot sends a function call request. This function should accept the following parameters:

  <Expandable title="FunctionCallParams">
    <ParamField type="string">
      The name of the function being called. It should always match the name you registered the handler under.
    </ParamField>

    <ParamField type="Record<string, unknown>">
      The arguments passed to the function call. This is a key-value object where the keys are the argument names and the values are the argument values.
    </ParamField>
  </Expandable>

  The callback should return a Promise that resolves with the result of the function call or void if no result is needed. If returning a result, it should be a `string` or `Record<string, unknown>`.
</ParamField>

## Other

### transport

`transport: Transport`

A safe accessor for the transport instance used by the client. This is useful for accessing transport-specific methods or properties that are not exposed directly on the client.

```typescript theme={null}
const transport = pcClient.transport as DailyTransport;
transport.getSessionInfo();
```

### setLogLevel()

`setLogLevel(level: LogLevel): void`

Sets the log level for the client. This is useful for debugging and controlling the verbosity of logs. The log levels are defined in the `LogLevel` enum:

```typescript theme={null}
export enum LogLevel {
  NONE = 0,
  ERROR = 1,
  WARN = 2,
  INFO = 3,
  DEBUG = 4,
}
```

By default, the log level is set to `LogLevel.DEBUG`.

```typescript theme={null}
pcClient.setLogLevel(LogLevel.INFO);
```

## Transport Compatibility

<TransportTable />


# Errors
Source: https://docs.pipecat.ai/client/js/api-reference/errors



## RTVIError Type

Base `PipecatClient` error type, extends [`Error`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error) and primarily introduces the `status` field. Most methods will try to throw an error of this type when something goes wrong. This is different from the RTVI error event and its corresponding [`onError` callback](./callbacks#param-on-error), which are used for communicating errors that are sent by the bot.

<ParamField type="number">
  A unique identifier (or HTTP code if applicable) for the error.
</ParamField>

<ParamField type="string">
  A human-readable message describing the error.
</ParamField>

## Pre-defined RTVIErrors

### ConnectionTimeoutError

Emitted when the bot does not enter a ready state within the specified timeout period during the `connect()` method call.

### StartBotError

Emitted from `startBot()` or `startBotAndConnect()` when the endpoint responds with an error or the `fetch` itself fails. This may be due to the endpoint being unavailable, or the server failing to parse the provided data.

<ParamField type="string">
  All `StartBotError` instances will have an `error` field set to `invalid-request-error`.
</ParamField>

<ParamField type="number">
  HTTP status code returned by the endpoint, if applicable.
</ParamField>

<ParamField type="string">
  Verbose error message returned by the endpoint, if provided. To take advantage of this, the endpoint should return an error response with a JSON object with an `info` field containing the error message.
</ParamField>

### TransportStartError

Emitted when the Transport is not able to connect. You may need to check the connection parameters provided or returned from you endpoint.

### BotNotReadyError

Emitted when the client attempts to perform an action or method that requires the bot to be in a ready state, but the bot is not ready. You must call `connect()` first and wait for the bot to be ready before performing such actions.

### DeviceError

Emitted when there is an issue with acquiring or using a media device, such as a camera or microphone. This could be due to permissions issues, device unavailability, or other related problems.

<ParamField type="array<'cam' | 'mic' | 'speaker'>">
  List of media devices, `'cam'`, `'mic'`, and/or `'speaker'`, that are unavailable or could not be accessed.
</ParamField>

<ParamField type="string">
  The `type` field will indicate what type of device error occurred. Options include:

  * `"in-use"`: A device is currently in use by another application and cannot be accessed. *windows only*
  * `"permissions"`: The user has not granted permission to access the media device.
  * `"undefined-mediadevices"`: `getUserMedia()` is not an available API on the current platform or browser.
  * `"not-found"`: The specified media device could not be found.
  * `"constraints"`: The media device could not be configured with the specified constraints.
  * `"unknown"`: An unknown error occurred while accessing the media device.
</ParamField>

<ParamField type="object">
  Additional details about the device error, if available.
</ParamField>

### UnsupportedFeatureError

Not all Transports are created equal, and some may not support certain features. This error is thrown when a feature is requested that the current Transport does not support.

<ParamField type="string">
  This custom field will contain the name of the unsupported feature.
</ParamField>


# Custom Messaging
Source: https://docs.pipecat.ai/client/js/api-reference/messages



The Pipecat JavaScript client can send and receive arbitrary messages to/from the server running the bot. This page outlines and demonstrates both client and server code for passing and responding to custom messages as well as providing arbitrary data at connection time.

## Connection-Time Configuration

Oftentimes clients need to provide configuration data to the server when starting the bot. This can include things like preferred language, user preferences, initial messages, or any other data that the server needs to know about the client. This must occur before the bot is started and therefore is not part of the RTVI standard, but rather a custom implementation. That said, the `PipecatClient` makes it easy to send this data as part of the `startBot()` and `startBotAndConnect()` methods, by passing an API endpoint object with the `requestData` property. Your server endpoint can then handle this data as needed. In the example below, we demonstrate sending an initial prompt and preferred language to the server when connecting.

<CodeGroup>
  ```javascript client theme={null}
  try {
      pcClient.startBotAndConnect({
          endpoint: '/api/start', // Your server endpoint to start the bot
          requestData: {
              initial_prompt: "You are a pirate captain",
              preferred_language: 'en-US'
          }
      });
  } catch (error) {
      console.error("Error starting the bot:", error);
  }
  ```

  ```python FastAPI endpoint theme={null}

  def validate_request_data(body: Dict[str, Any]) -> Tuple[str, str]:
      """Validate and extract prompt and language from request data."""
      if not isinstance(body, dict):
          raise ValueError("Request body must be a dictionary")

      prompt = body.get("initial_prompt", "You are a pirate captain")
      lang = body.get("preferred_language", "en-US")

      if not isinstance(prompt, str) or not isinstance(lang, str):
          raise ValueError("Both initial_prompt and preferred_language must be strings")

      return prompt, lang

  @app.post("/api/start")
  async def start(request: Request) -> Dict[Any, Any]:
      """Startup endpoint that creates a room, starts the bot, and returns
         connection credentials.

      This endpoint is called by clients to kick off a session and establish
      a connection.

      Returns:
          Dict[Any, Any]: Authentication bundle containing room_url and token

      Raises:
          HTTPException: If room creation, token generation, or bot startup fails
      """
      body = await request.json()
      try:
          prompt, lang = validate_request_data(body)
      except ValueError as e:
          raise HTTPException(status_code=400, info=f"Invalid request data: {e}")

      print("Creating room for RTVI connection", body)
      room_url, token = await create_room_and_token()
      print(f"Room URL: {room_url}")

      # Start the bot process
      try:
          proc = subprocess.Popen(
              [f"python3 -m bot -u {room_url} -t {token} -p {prompt} -l {lang}"],
              shell=True,
              bufsize=1,
              cwd=os.path.dirname(os.path.abspath(__file__)),
          )
          bot_procs[proc.pid] = (proc, room_url)
      except Exception as e:
          raise HTTPException(status_code=500, detail=f"Failed to start subprocess: {e}")

      # Return the authentication bundle in format expected by DailyTransport
      return {"url": room_url, "token": token}
  ```

  ```python bot theme={null}
  import argparse
  import asyncio

  def extract_arguments():
      parser = argparse.ArgumentParser(description="Example")
      parser.add_argument(
          "-u", "--room-url", type=str, default=os.getenv("DAILY_SAMPLE_ROOM_URL", "")
      )
      parser.add_argument(
          "-t", "--token", type=str, default=os.getenv("DAILY_SAMPLE_ROOM_TOKEN", None)
      )
      parser.add_argument(
          "-p", "--prompt", type=str, default="You are a pirate captain"
      )
      parser.add_argument("-l", "--language", type=str, default="en-US")
      return parser.parse_args()

  async def main():
      args = extract_arguments()
      print(f"room_url: {args.room_url}")

      daily_transport = DailyTransport(
          args.room_url,
          args.token,
          "Chatbot",
          DailyParams(
              audio_in_enabled=True,
              audio_out_enabled=True,
          ),
      )

      llm = GeminiLiveLLMService(
          api_key=os.getenv("GOOGLE_API_KEY"),
          voice_id="Puck",
          system_instruction=SYSTEM_INSTRUCTION,
          params=InputParams(
              language=args.language,  # Pass preferred language to LLM
          ),
      )

      messages = [ { role: "system", content: args.prompt } ]
      context = LLMContext(messages)
      user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
          context,
          user_params=LLMUserAggregatorParams(
              vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
          ),
      )

      pipeline = Pipeline(
          [
              daily_transport.input(),
              user_aggregator,
              llm,
              daily_transport.output(),
              assistant_aggregator,
          ]
      )

      task = PipelineTask(pipeline)

      @rtvi.event_handler("on_client_ready")
      async def on_client_ready(rtvi):
          logger.debug("Client ready")
          # Kick off the conversation
          await task.queue_frames([LLMRunFrame()])

      @transport.event_handler("on_client_disconnected")
      async def on_client_disconnected(transport, client):
          logger.debug(f"Client disconnected")
          await task.cancel()

      runner = PipelineRunner(handle_sigint=False)

      await runner.run(task)

  if __name__ == "__main__":
      asyncio.run(main())
  ```
</CodeGroup>

## Sending Custom Messages to the Server

Once connected, you can send custom messages to the server using the `sendClientMessage` method. This is useful for triggering specific actions or sending data that the server needs to process.

<CodeGroup>
  ```javascript client theme={null}
  try {
      pcClient.sendClientMessage('set-language', { language: 'en-US' });
  } catch (error) {
      console.error("Error sending message to server:", error);
  }
  ```

  ```python bot theme={null}
      task = PipelineTask(
          pipeline,
          params,
          observers=[RTVIObserver(rtvi)],
      )

      @rtvi.event_handler("on_client_message")
      async def on_client_message(rtvi, msg):
          print("RTVI client message:", msg.type, msg.data)
          if msg.type == "set-language":
              language = msg.data.get("language", "en-US")
              await task.queue_frames([STTUpdateSettingsFrame(language=language)])

  ### Alternatively, if your message requires asynchronous processing or storing of
  ### state, you may want to handle it from inside a FrameProcessor, listen for a
  ### RTVIClientMessageFrame and push a RTVIServerResponseFrame
  class CustomFrameProcessor(FrameProcessor):
      async def process_frame(self, frame: Frame, direction: FrameDirection):
          await super().process_frame(frame, direction)
          if isinstance(frame, RTVIClientMessageFrame):
              print("RTVI client message:", frame.msg_id, frame.type, frame.data)
              if frame.type == "set-language":
                  language = frame.data.get("language", "en-US")
                  await self.push_frame(STTUpdateSettingsFrame(language=language))
                  return
          await self.push_frame(frame, direction)
  ```
</CodeGroup>

## Requesting Information from the Server

You can also request information from the server using the `sendClientRequest` method. This is useful for querying the server for specific data or triggering and action and getting a success/failure response.

<CodeGroup>
  ```javascript client theme={null}
  try {
      const response = await pcClient.sendClientRequest('get-language');
      console.log("Current language:", response.language);
  } catch (error) {
      console.error("Error requesting data from server:", error);
  }
  ```

  ```python bot theme={null}
  @rtvi.event_handler("on_client_message")
  async def on_client_message(rtvi, msg):
      print("RTVI client message:", msg.type, msg.data)
      if msg.type == "get-language":
          await rtvi.send_server_response(msg, {"language": get_current_language()})
      else:
          await rtvi.send_error_response(msg, "Unknown request type")

  ### Alternatively, if your message requires asynchronous processing or storing of
  ### state, you may want to handle it from inside a FrameProcessor, listen for a
  ### RTVIClientMessageFrame and push a RTVIServerResponseFrame
  class CustomFrameProcessor(FrameProcessor):
      async def process_frame(self, frame: Frame, direction: FrameDirection):
          await super().process_frame(frame, direction)
          if isinstance(frame, RTVIClientMessageFrame):
              print("RTVI client message:", frame.msg_id, frame.type, frame.data)
              if frame.type == "get-language":
                  data = {"language": get_current_language()}
                  await self.push_frame(
                      RTVIServerResponseFrame(
                          client_msg=frame,
                          data=data,
                      ),
                  )
                  return
              else:
                  await self.push_frame(
                      RTVIServerResponseFrame(
                          client_msg=frame,
                          error="Unknown request type"
                      )
                  )
          await self.push_frame(frame, direction)
  ```
</CodeGroup>

## Handling Custom Messages from the Server

You can handle custom messages sent from the server using the `onServerMessage` callback. This allows you to process messages that the server sends back to the client, such as notifications or updates.

<CodeGroup>
  ```javascript client theme={null}
  pcClient.onServerMessage((message) => {
      console.log("Received message from server:", message);
      if (message.data.msg === 'language-updated') {
          console.log("Language updated to:", message.data.language);
      }
  });
  ```

  ```python bot theme={null}
  ## From inside an Observer, call `send_server_message` directly on your rtvi instance
  class CustomObserver(BaseObserver):
      async def on_push_frame(self, data: FramePushed):
          if isinstance(frame, STTUpdateSettingsFrame):
              for key, value in settings.items():
                  if key == "language":
                      await rtvi.send_server_message({
                          "msg": "language-updated",
                          "language": value
                      })

  ### Alternatively, from inside a FrameProcessor, push a RTVIServerMessageFrame
  class CustomFrameProcessor(FrameProcessor):
      async def process_frame(self, frame: Frame, direction: FrameDirection):
          await super().process_frame(frame, direction)
          if isinstance(frame, STTUpdateSettingsFrame):
              for key, value in settings.items():
                  if key == "language":
                      await self.push_frame(
                          RTVIServerMessageFrame(
                              data={
                                  "msg": "language-updated",
                                  "language": value
                              }
                          )
                      )
          await self.push_frame(frame, direction)
  ```
</CodeGroup>


# SDK Introduction
Source: https://docs.pipecat.ai/client/js/introduction

Build web applications with Pipecat’s JavaScript client library

The Pipecat JavaScript SDK provides a lightweight client implementation that handles:

* Device and media stream management
* Connecting to Pipecat bots
* Messaging with Pipecat bots and handling responses using the RTVI standard
* Managing session state and errors

## Installation

Install the SDK and a transport implementation (e.g. Daily for WebRTC):

```bash theme={null}
npm install @pipecat-ai/client-js
npm install @pipecat-ai/[daily-transport, small-webrtc-transport, etc.]
```

## Example

Here's a simple example using Daily as the transport layer:

```javascript theme={null}
import { PipecatClient } from "@pipecat-ai/client-js";
import { DailyTransport } from "@pipecat-ai/daily-transport";

// Handle incoming audio from the bot
function handleBotAudio(track, participant) {
  if (participant.local || track.kind !== "audio") return;

  const audioElement = document.createElement("audio");
  audioElement.srcObject = new MediaStream([track]);
  document.body.appendChild(audioElement);
  audioElement.play();
}

// Create and configure the client
const pcClient = new PipecatClient({
  transport: new DailyTransport(),
  enableMic: true,
  callbacks: {
    onTrackStarted: handleBotAudio,
  },
});

// Connect to your bot
pcClient.connect({
  url: "https://your-daily-room-url",
  token: "your-daily-token",
});
```

## Explore the SDK

<CardGroup>
  <Card title="Client Constructor" icon="cube" href="/client/js/api-reference/client-constructor">
    Configure your client instance with transport and callbacks
  </Card>

  <Card title="Client Methods" icon="code" href="/client/js/api-reference/client-methods">
    Core methods for interacting with your bot
  </Card>

  <Card title="API Reference" icon="book" href="/client/js/api-reference">
    Detailed documentation of all available APIs
  </Card>
</CardGroup>

The Pipecat JavaScript SDK implements the [RTVI standard](/client/rtvi-standard) for real-time AI inference, ensuring compatibility with any RTVI-compatible server and transport layer.


# RTVIClient Migration Guide for JavaScript
Source: https://docs.pipecat.ai/client/js/migration-guide

A Guide to migrating from a JavaScript RTVIClient to PipecatClient

This guide covers migrating from RTVIClient to the new `PipecatClient` in a JavaScript application. The new client introduces simplified configuration and improved client-server messaging. For an overview of the changes, see the top-level [RTVIClient Migration Guide](/client/migration-guide).

## Key Changes

### 1. Package and Class Names

**Old**

```javascript theme={null}
import { RTVIClient } from '@pipecat-ai/client-js';
```

**New**

```javascript theme={null}
// New
import { PipecatClient } from '@pipecat-ai/client-js';
```

### 2. Client and Transport Configuration

The core difference here is that the client no longer accepts a `params` field. The configuration that used to be provided here is no longer supported and hitting a `/connect` endpoint is now an explicit step and provided directly to either the `startBot()` or `startBotAndConnect()` method.

**Old**

```javascript theme={null}
const transport = new DailyTransport();
const client = new RTVIClient({
  transport,
  params: {
    baseUrl: 'http://localhost:7860',
    endpoints: {
      connect: '/connect'
    }
  }
});
```

**New**

```javascript theme={null}
const client = new PipecatClient({
  transport: new DailyTransport(),
  // Connection params moved to startBotAndConnect() call
});
```

### 3. Connection Method

Previously, `connect()` was called on the client instance without parameters. Now, you provide connection parameters directly to the `connect()` method. This allows for more flexibility and customization of the connection process. For ease of use, we've also introduced a `startBot()` and `startBotAndConnect()` method where you can provide an API endpoint that returns the connection parameters.

**Old**

```javascript theme={null}
await client.connect();
```

**New**

There are three options for connecting now. For migration purposes, you are likely to use option 3 (`startBotAndConnect()`):

1. **Direct Connection**: Provide the connection details directly to `connect()`.

```javascript theme={null}
const cxnDetails = {
  url: 'https://your-daily-room-url',
  token: 'your-daily-token'
};
await client.connect(cxnDetails);
```

2. **Using startBot()**: Fetch connection details from an API endpoint and pass them to `connect()`.

```javascript theme={null}
let cxnDetails = await client.startBot({
  endpoint: 'http://localhost:7860/connect',
  requestData: {
    // Any custom data your /connect endpoint requires
    llm_provider: 'openai',
    initial_prompt: "You are a pirate captain",
    // Any additional data
  }
});
cxnDetails = modifyCxnDetails(cxnDetails); // Modify if needed
await client.connect(cxnDetails);
```

3. **Using startBotAndConnect()**: Fetch connection details and connect in one step.

```javascript theme={null}
await client.startBotAndConnect({
  endpoint: 'http://localhost:7860/connect',
  requestData: {
    // Any custom data your /connect endpoint requires
    llm_provider: 'openai',
    initial_prompt: "You are a pirate captain",
    // Any additional data
  }
});
```

### 4. Function Call Handling

Previously, you would use a helper class to handle function calls and provide a single callback to handle any/all function calls. Now, you can register a callback for each function call by name directly on the `PipecatClient` instance.

**Old**

```javascript theme={null}
let llmHelper = new LLMHelper({});
llmHelper.handleFunctionCall(async (data) => {
  return await this.handleFunctionCall(data.functionName, data.arguments);
});
client.registerHelper('openai', llmHelper);
```

**New**

```javascript theme={null}
client.registerFunctionCallHandler('functionName', async (data) => {
  // Handle function call
  return result;
});
```

### 5. Pipeline Configuration Initialization

Previously, you could provide a pipeline configuration as part of the `RTVIClient` constructor and it was expected to be in a specific format. Now, if you would like to pass any initial pipeline configurations, you do so as `requestData` added to the endpoint you provide to `connect()`. In both cases, you would need server-side code to parse and apply these settings, but now you can define the structure and what pieces of configuration you want to send.

**Old**

```javascript theme={null}
const pipelineConfig = [{
  "service": "llm",
  "options": [
    {
      "name": "initial_messages",
      "value": [
        {
          "role": "system",
          "content": `You are a pirate captain.`
        }
      ]
    },
  ],
},
{
  "service": "tts",
  "options": [
    {
      "name": "language",
      "value": "en-US"
    }
  ]
}];

const rtviClient = new RTVIClient({
  ...
  params: {
    ...
    config: pipelineConfig
  },
});
rtviClient.connect();
```

**New**

Check out [this section of docs](./api-reference/messages#connection-time-configuration) for an example, complete with server-side code showing how to initialize the pipeline configuration at connection time.

```javascript theme={null}
try {
  pcClient.startBotAndConnect({
    endpoint: 'https://your-server/connect',
    requestData: {
      initial_prompt: "You are a pirate captain",
      preferred_language: 'en-US'
    }
  });
} catch (error) {
  console.error("Error connecting to server:", error);
}
```

### 6. Pipeline Configuration Updates

Previously, the client supported updating the pipeline configuration using a specific method that took a configuration object in a generic format. Dynamic and predifined configuration updates, however, are a security concern, allowing clients to override settings and potentially abuse API keys. For this reason, it has been removed and most configuration updates need to be handled custom by your application. To do so, you should take advantage of the client-server messaging system, which allows you to send messages to the server and handle responses. This way, you can implement your own logic for updating configurations securely.

**Old**

```javascript theme={null}
const updatedConfig = [{
  "service": "tts",
  "options": [
    {
      "name": "voice",
      "value": "Janice"
    }
  ]
}];
try {
  await rtviClient.updateConfig(
    updatedConfig as RTVIClientConfigOption[],
    true
  );
} catch (e) {
  console.error("Failed to update config", e);
}
```

**New**

Check out [this section of docs](./api-reference/messages#sending-custom-messages-to-the-server) for a more complete example, along with an example on making a request (`sendClientRequest()`) to wait for a response.

```javascript theme={null}
try {
  pcClient.sendClientMessage('set-voice', { voice: 'Janice' });
} catch (error) {
  console.error("Error sending message to server:", error);
}
```

### 7. LLM Context Updates

Previously, you would use a helper class to update the context. This could be a security concern for the same reasons as mentioned above and should now be done using the [client-server messaging system](./api-reference/messages). However, in the case where you simply wish to add to the user's context, you can use the [`sendText()`](./api-reference/client-methods#sendtext) method on the `PipecatClient` instance. This allows for easy, built-in support for text-based context updates without needing to implement custom messaging.

**Old**

```javascript theme={null}
let llmHelper = new LLMHelper({});
client.registerHelper('llm', llmHelper);
await llmHelper.setContext(
  {
    messages: [
      {
        role: "user",
        content: "Tell me a joke.",
      },
    ],
  },
  true
);
```

**New**

```javascript theme={null}
client.sendText("Tell me a joke.");
```

## Breaking Changes

1. **Configuration Structure**: Connection parameters are now passed to connect() instead of constructor
2. **Helper System**: Removed in favor of direct `PipecatClient` member functions or client-server messaging.

## Migration Steps

1. Update package imports to use new names
2. Move connection configuration from constructor to startBot()/startBotAndConnect()/connect() methods
3. Replace any helper classes with corresponding `PipecatClient` methods or custom messaging
4. Update any TypeScript types referencing old names


# Daily WebRTC Transport
Source: https://docs.pipecat.ai/client/js/transports/daily



The DailyTransport class provides a WebRTC transport layer using [Daily.co's](https://daily.co) infrastructure. It wraps a Daily-JS call client to handle audio/video device management, WebRTC connections, and real-time communication between clients and bots. For complete documentation on Daily's API, see the [Daily API Reference](https://docs.daily.co/reference/daily-js).

This transport is designed for production use cases, leveraging Daily's global infrastructure for low-latency, high-quality audio and video streaming. It expects your Pipecat server to include the corresponding [`DailyTransport` server-side](/server/services/transport/daily) implementation.

## Usage

### Basic Setup

```javascript theme={null}
import { PipecatClient } from "@pipecat-ai/client-js";
import { DailyTransport } from "@pipecat-ai/daily-transport";

const pcClient = new PipecatClient({
  transport: new DailyTransport({
    // DailyTransport constructor options
    bufferLocalAudioUntilBotReady: true, // Optional, defaults to false
    inputSettings: { video: { processor: { type: "background-blur" } } },
  }),
  enableCam: false, // Default camera off
  enableMic: true, // Default microphone on
  callbacks: {
    // Event handlers
  },
  // ...
});

await pcClient.connect({
  url: "https://your-domain.daily.co/room",
  token: "your-daily-token", // Optional, if your room requires authentication
});
```

## API Reference

### Constructor Options

```typescript theme={null}
interface DailyTransportConstructorOptions extends DailyFactoryOptions {
  bufferLocalAudioUntilBotReady?: boolean;
}
```

<ParamField type="boolean">
  If set to `true`, the transport will buffer local audio until the bot is ready. This is useful for ensuring that bot gets any audio from the user that started before the bot is ready to process it.
</ParamField>

<ParamField>
  The `DailyTransportConstructorOptions` extends the `DailyFactoryOptions` type that is accepted by the underlying Daily instance. These options are passed directly through to the Daily constructor. See the [Daily API Reference](https://docs.daily.co/reference/daily-js/daily-call-client/properties) for a complete list of options.

  <Note>
    While you can provide the room url and optional token as part of your
    constructor options, the typical pattern is to provide them via a connection
    endpoint with `startBot()` or directly as part of `connect()`. See below.
  </Note>
</ParamField>

### TransportConnectionParams

On `connect()`, the `DailyTransport` optionally takes a set of [`DailyCallOptions`](https://docs.daily.co/reference/daily-js/daily-call-client/methods#dailycalloptions) to connect to a Daily room. This can be provided directly to the `PipecatClient`'s `connect()` method or via a starting endpoint passed to the `PipecatClient`'s `startBotAndConnect()` method. If using an endpoint, your endpoint should return a JSON object matching the `DailyCallOptions` type. See the [client connect()](/client/js/api-reference/client-methods#connect) documentation for more information.

<CodeGroup>
  ```typescript client theme={null}
  pcClient.connect({
    url: 'https://your.daily.co/room'
  });
  // OR...
  pcClient.startBotAndConnect({
    endpoint: '/api/start', // Your server endpoint to start the bot
  });
  ```

  ```python server theme={null}
  @app.post("/api/start")
  async def start(request: Request) -> Dict[Any, Any]:
      print("Creating room and token for RTVI connection")
      room_url, token = await create_room_and_token()

      # Start the bot process
      print("Starting bot subprocess")
      try:
          subprocess.Popen(
              [f"python3 -m bot.py -u {room_url} -t {token}"],
              shell=True,
              bufsize=1,
              cwd=os.path.dirname(os.path.abspath(__file__)),
          )
      except Exception as e:
          raise HTTPException(status_code=500, detail=f"Failed to start subprocess: {e}")

      # Return the Daily call options in format expected by DailyTransport/Daily Call Object
      return {"url": room_url, "token": token}
  ```
</CodeGroup>

### Methods

For most operations, you will not interact with the transport directly. Most methods have an equivalent in the `PipecatClient` and should be called from the `PipecatClient`. However, there are a few transport-specific methods that you may need to call directly. When doing so, be sure to access your transport via the `transport` property of the `PipecatClient` instance.

* `preAuth()`

  This is the one method meant to be called directly, which is used to allow you to gather information about the Daily room prior to connecting. As a Daily-specific action, it is not exposed through the `PipecatClient`. This method must be called prior to `connect()` and use the same `room_url` and `token` (optional) as what will be used on `connect()`.

  ```typescript theme={null}
  pcClient.transport.preAuth({
    url: "https://your.daily.co/room",
    token: "your_token",
  });
  const roomInfo = pcClient.transport.dailyCallClient.room();
  ```

## Events

The transport implements the various [`PipecatClient` event handlers](/client/js/api-reference/callbacks). For Daily-specific events, you can attach listeners to the underlying Daily call client. For a list of available events, see the [Daily API Reference](https://docs.daily.co/reference/daily-js/events).

```typescript theme={null}
pcClient.transport.dailyCallClient.on('recording-started', (ev) => {...});
```

## Advanced

### Accessing the Daily Call

For advanced use cases, where you may need to work with the Daily call client directly, you can access it via the `dailyCallClient` property.

```javascript theme={null}
const dailyCall = pcClient.transport.dailyCallClient;
```

<Note>
  The Daily call client returned is safe-guarded to not allow you to call
  functions which affect the call's lifecycle and will redirect you to use
  either a Transport method or the `PipecatClient` to perform the equivalent
  action.
</Note>

## More Information

<CardGroup>
  <Card title="Demo" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/simple-chatbot">
    Simple Chatbot Demo
  </Card>

  <Card title="Source" icon="github" href="https://github.com/pipecat-ai/pipecat-client-web-transports/tree/main/transports/daily">
    `DailyTransport`
  </Card>
</CardGroup>

<Card title="Package" icon="browser" href="https://www.npmjs.com/package/@pipecat-ai/daily-transport">
  `@pipecat-ai/daily-transport`
</Card>


# GeminiLiveWebSocketTransport
Source: https://docs.pipecat.ai/client/js/transports/gemini



## Overview

The `GeminiLiveWebsocketTransport` class implements a fully functional [Pipecat `Transport`](./transport), providing a framework for implementing real-time communication directly with the [Gemini Multimodal Live](https://ai.google.dev/api/multimodal-live) service. Like all transports, it handles media device management, audio/video streams, and state management for the connection.

<Note>
  Transports of this type are designed primarily for development and testing
  purposes. For production applications, you will need to build a server
  component with a server-friendly transport, like the
  [DailyTransport](./daily), to securely handle API keys.
</Note>

## Usage

### Basic Setup

```javascript theme={null}
import {
  GeminiLiveWebsocketTransport,
  GeminiLLMServiceOptions,
} from "@pipecat-ai/gemini-live-websocket-transport";
import { PipecatClient } from "@pipecat-ai/client-js";

const options: GeminiLLMServiceOptions = {
  api_key: "YOUR_API_KEY",
  initial_messages: [
    // Set up initial system and user messages.
    // Without the user message, the bot will not respond immediately
    // and wait for the user to speak first.
    {
      role: "model",
      content: "You are a confused jellyfish.",
    },
    { role: "user", content: "Blub blub!" },
  ],
  generation_config: {
    temperature: 0.7,
    maxOutput_tokens: 1000,
  },
};

const transport = new GeminiLiveWebsocketTransport(options);
let pcClient = new PipecatClient({
  transport: new GeminiLiveWebsocketTransport(options),
  callbacks: {
    // Event handlers
  },
});
pcClient.connect();
```

## API Reference

### Constructor Options

#### `GeminiLLMServiceOptions`

```typescript theme={null}
interface GeminiLLMServiceOptions {
  api_key: string; // Required: Your Gemini API key
  initial_messages?: Array<{
    // Optional: Initial conversation context
    content: string;
    role: string;
  }>;
  generation_config?: {
    // Optional: Generation parameters
    candidate_count?: number;
    max_output_tokens?: number;
    temperature?: number;
    top_p?: number;
    top_k?: number;
    presence_penalty?: number;
    frequency_penalty?: number;
    response_modalities?: string;
    speech_config?: {
      voice_config?: {
        prebuilt_voice_config?: {
          voice_name: "Puck" | "Charon" | "Kore" | "Fenrir" | "Aoede";
        };
      };
    };
  };
}
```

### TransportConnectionParams

The `GeminiLiveWebsocketTransport` does not take connection parameters. It connects directly to the Gemini Multimodal Live service using the API key provided as part of the initial configuration.

### Events

The GeminiLiveWebSocketTransport implements the various [PipecatClient event handlers](/client/js/api-reference/callbacks). Check out the docs or samples for more info.

## More Information

<CardGroup>
  <Card title="Demo" icon="play" href="https://github.com/pipecat-ai/pipecat-client-web-transports/tree/main/examples/directToLLMTransports">
    Gemini MultiModal Live Basic Demo
  </Card>

  <Card title="Source" icon="github" href="https://github.com/pipecat-ai/pipecat-client-web-transports/tree/main/transports/gemini-live-websocket-transport">
    `GeminiLiveWebsocketTransport`
  </Card>
</CardGroup>

<Card title="Package" icon="browser" href="https://www.npmjs.com/package/@pipecat-ai/gemini-live-websocket-transport">
  `@pipecat-ai/realtime-websocket-transport`
</Card>


# OpenAIRealTimeWebRTCTransport
Source: https://docs.pipecat.ai/client/js/transports/openai-webrtc



## Overview

The `OpenAIRealTimeWebRTCTransport` is a fully functional [Pipecat `Transport`](/client/js/transports/transport). It provides a framework for implementing real-time communication directly with the [OpenAI Realtime API using WebRTC](https://platform.openai.com/docs/guides/realtime-webrtc) voice-to-voice service. It handles media device management, audio/video streams, and state management for the connection.

<Note>
  Transports of this type are designed primarily for development and testing
  purposes. For production applications, you will need to build a server
  component with a server-friendly transport, like the
  [DailyTransport](./daily), to securely handle API keys.
</Note>

## Usage

### Basic Setup

```javascript theme={null}
import { OpenAIRealTimeWebRTCTransport, OpenAIServiceOptions } from '@pipecat-ai/openai-realtime-webrtc-transport';
import { PipecatClient } from '@pipecat-ai/client-js';

const options: OpenAIServiceOptions = {
  api_key: 'YOUR_API_KEY',
  session_config: {
    instructions: 'You are a confused jellyfish.',
  },
  initial_messages: [{ role: "user", content: "Blub blub!" }],
};

let pcClient = new PipecatClient({
  transport: new OpenAIRealTimeWebRTCTransport (options),
  ...
});
pcClient.connect();
```

## API Reference

### Constructor Options

Below is the transport's type definition for the OpenAI Session configuration you need to pass in to the `create()` method. See the [OpenAI Realtime API documentation](https://platform.openai.com/docs/api-reference/realtime-client-events/session/update) for more details on each of the options and their defaults.

```typescript theme={null}
export type OpenAIFunctionTool = {
  type: "function";
  name: string;
  description: string;
  parameters: JSONSchema;
};

export type OpenAIServerVad = {
  type: "server_vad";
  create_response?: boolean;
  interrupt_response?: boolean;
  prefix_padding_ms?: number;
  silence_duration_ms?: number;
  threshold?: number;
};

export type OpenAISemanticVAD = {
  type: "semantic_vad";
  eagerness?: "low" | "medium" | "high" | "auto";
  create_response?: boolean; // defaults to true
  interrupt_response?: boolean; // defaults to true
};

export type OpenAISessionConfig = Partial<{
  modalities?: string;
  instructions?: string;
  voice?:
    | "alloy"
    | "ash"
    | "ballad"
    | "coral"
    | "echo"
    | "sage"
    | "shimmer"
    | "verse";
  input_audio_noise_reduction?: {
    type: "near_field" | "far_field";
  } | null; // defaults to null/off
  input_audio_transcription?: {
    model: "whisper-1" | "gpt-4o-transcribe" | "gpt-4o-mini-transcribe";
    language?: string;
    prompt?: string[] | string; // gpt-4o models take a string
  } | null; // we default this to gpt-4o-transcribe
  turn_detection?: OpenAIServerVad | OpenAISemanticVAD | null; // defaults to server_vad
  temperature?: number;
  max_tokens?: number | "inf";
  tools?: Array<OpenAIFunctionTool>;
}>;

export interface OpenAIServiceOptions {
  api_key: string;
  model?: string;
  initial_messages?: LLMContextMessage[];
  settings?: OpenAISessionConfig;
}
```

### TransportConnectionParams

The `OpenAIRealTimeWebRTCTransport` does not take connection parameters. It connects directly to the OpenAI Realtime API using the API key provided as part of the initial configuration.

### Events

The transport implements the various [`PipecatClient` event handlers](/client/js/api-reference/callbacks). Check out the docs or samples for more info.

## More Information

<CardGroup>
  <Card title="Demo" icon="play" href="https://github.com/pipecat-ai/pipecat-client-web-transports/tree/main/examples/directToLLMTransports">
    OpenAI Realtime Basic Demo
  </Card>

  <Card title="Source" icon="github" href="https://github.com/pipecat-ai/pipecat-client-web-transports/tree/main/transports/openai-realtime-webrtc-transport">
    `OpenAIRealTimeWebRTCTransport`
  </Card>
</CardGroup>

<Card title="Package" icon="browser" href="https://www.npmjs.com/package/@pipecat-ai/openai-realtime-webrtc-transport">
  `@pipecat-ai/openai-realtime-webrtc-transport`
</Card>


# SmallWebRTCTransport
Source: https://docs.pipecat.ai/client/js/transports/small-webrtc

A lightweight WebRTC transport for peer-to-peer connections with Pipecat

`SmallWebRTCTransport` enables peer-to-peer WebRTC connections between clients and your Pipecat application. It implements bidirectional audio and video streaming using WebRTC for real-time communication.

This transport is intended for lightweight implementations, particularly for local development and testing. It expects your Pipecat server to include the corresponding [`SmallWebRTCTransport` server-side](/server/services/transport/small-webrtc) implementation.

## Usage

### Basic Setup

```javascript theme={null}
import { PipecatClient } from "@pipecat-ai/client-js";
import { SmallWebRTCTransport } from "@pipecat-ai/small-webrtc-transport";

const pcClient = new PipecatClient({
  transport: new SmallWebRTCTransport({
    // Optional configuration for the transport
    iceServers: ["stun:stun.l.google.com:19302"],
  }),
  enableCam: false, // Default camera off
  enableMic: true, // Default microphone on
  callbacks: {
    // Event handlers
  },
});

await pcClient.connect({
  webrtcUrl: "/api/offer", // Your WebRTC signaling server endpoint
});
```

## API Reference

### Constructor Options

```typescript theme={null}
interface SmallWebRTCTransportConstructorOptions {
  iceServers?: RTCIceServer[];
  waitForICEGathering?: boolean;
  webrtcUrl?: string;
  audioCodec?: string;
  videoCodec?: string;
  mediaManager?: MediaManager;
}
```

#### Properties

<ParamField name="iceServers" type="string[]">
  Array of STUN/TURN server URLs for ICE connection establishment. Default is `["stun:stun.l.google.com:19302"]`.

  ```javascript theme={null}
  // Set custom ICE servers
  transport.iceServers = [
    "stun:stun.l.google.com:19302",
    "stun:stun1.l.google.com:19302",
  ];
  ```
</ParamField>

<ParamField name="waitForICEGathering" type="boolean">
  If `true`, the transport will wait for ICE gathering to complete before being
  considered `'connected'`.
</ParamField>

<ParamField name="webrtcUrl" type="string">
  URL of the WebRTC signaling server's offer endpoint. This endpoint may also be provided as part of `connect()`.

  Note: This field used to be called `connectionUrl` in versions prior to
  `1.2.0`.
</ParamField>

<ParamField name="audioCodec" type="string">
  Preferred audio codec to use. If not specified, your browser default will be
  used.
</ParamField>

<ParamField name="videoCodec" type="string">
  Preferred video codec to use. If not specified, your browser default will be
  used.
</ParamField>

<ParamField name="mediaManager" type="MediaManager">
  The media manager to use for handling local audio and video streams. This
  should not be overridden unless you have a specific reason to use a different
  media manager. The default is `DailyMediaManager`, which is suitable for most
  use cases. Note that the `DailyMediaManager` does not use any of Daily's
  services, it simply takes advantage of vast media support provided by the
  Daily library.
</ParamField>

### TransportConnectionParams

```typescript theme={null}
export type SmallWebRTCTransportConnectionOptions = {
  webrtcUrl?: string;
};
```

On `connect()`, the `SmallWebRTCTransport` optionally takes a set of connection parameters. This can be provided directly to the `PipecatClient`'s `connect()` method or via a starting endpoint passed to the `PipecatClient`'s `startBotAndConnect()` method. If using an endpoint, your endpoint should return a JSON object matching the `SmallWebRTCTransportConnectionOptions` type, which currently expects a single `webrtcUrl` property.

<CodeGroup>
  ```typescript client theme={null}
  pcClient.startBotAndConnect({
    endpoint: '/api/start', // Your server endpoint to start the bot and return the webrtcUrl
  });
  // OR...
  pcClient.connect({
    webrtcUrl: '/api/offer', // Your WebRTC offer/answer endpoint
  });
  ```

  ```python server theme={null}
  # See
  # https://github.com/pipecat-ai/pipecat-examples/blob/main/p2p-webrtc/video-transform/server/server.py
  # for a complete example of how to implement the server-side endpoint.
  @app.post("/api/offer")
  async def offer(request: dict, background_tasks: BackgroundTasks):
      pipecat_connection = SmallWebRTCConnection(ice_servers)
      await pipecat_connection.initialize(sdp=request["sdp"], type=request["type"])

      @pipecat_connection.event_handler("closed")
      async def handle_disconnected(webrtc_connection: SmallWebRTCConnection):
          logger.info(f"Discarding peer connection for pc_id: {webrtc_connection.pc_id}")
          pcs_map.pop(webrtc_connection.pc_id, None)

      background_tasks.add_task(run_bot, pipecat_connection)

      answer = pipecat_connection.get_answer()

      return answer
  ```
</CodeGroup>

### Methods

For most operations, you will not interact with the transport directly. Most methods have an equivalent in the `PipecatClient` and should be called from the `PipecatClient`. However, there are a few transport-specific methods that you may need to call directly. When doing so, be sure to access your transport via the `transport` property of the `PipecatClient` instance.

<ResponseField name="setAudioCodec" type="method">
  Sets the preferred audio codec.

  ```javascript theme={null}
  transport.setAudioCodec("opus");
  ```
</ResponseField>

<ResponseField name="setVideoCodec" type="method">
  Sets the preferred video codec.

  ```javascript theme={null}
  transport.setVideoCodec("VP8");
  ```
</ResponseField>

## Events

The transport implements the various [`PipecatClient` event handlers](/client/js/api-reference/callbacks).

## Connection Process

The connection process follows these steps:

1. The transport negotiates a WebRTC connection with the corresponding pipecat transport, complete with transceivers for the media and a data channel for messaging.
2. The transport sends a message to the pipecat transport to let it know it's ready.
3. The Pipecat transport sends a message letting the client know it is ready.

## Reconnection Handling

The transport includes automatic reconnection logic:

* Up to 3 reconnection attempts after connection failures
* Detection of ICE connection state changes
* Graceful recovery from temporary disconnections
* Graceful disconnect when reconnection attempts fail

## More Information

<CardGroup>
  <Card title="Video Transform Demo" icon="video" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/p2p-webrtc/video-transform">
    Real-time video transformation example
  </Card>

  <Card title="Package" icon="browser" href="https://www.npmjs.com/package/@pipecat-ai/small-webrtc-transport">
    `@pipecat-ai/small-webrtc-transport`
  </Card>
</CardGroup>


# Transport Overview
Source: https://docs.pipecat.ai/client/js/transports/transport



Transports are the means by which `PipecatClient`s communicate with their bot services. Transports implement the underlying device management, connectivity, media transmission, and state logic that manage the lifecycle of your session.

All transport packages (such as `DailyTransport`) extend from the `Transport` base class defined in the `client-js` library. You can extend this class if you are looking to implement your own or add additional functionality.

## Transport lifecycle

Each Pipecat client instance is associated with a transport instance. The instance will re-use the transport instance across multiple calls to `connect()`, allowing you to connect to different bot services without needing to create a new transport or client each time.

```typescript theme={null}
import { PipecatClient } from "@pipecat-ai/client-js";
import { DailyTransport } from "@pipecat-ai/daily-transport";

const pcClient = new PipecatClient({
  transport: new DailyTransport(),
  ...
});

await pcClient.startBotAndConnect({ endpoint: "/api/start" });
await pcClient.disconnect();
await pcClient.connect(); // re-uses url returned from previous startBotAndConnect call, skipping the endpoint
```

## Transport states

`TransportState`

Your transport instance goes through a series of states during its lifecycle. These states are:

<Steps>
  <Step title="Disconnected">
    Transport is idle and has not yet been initialized (default state).
  </Step>

  <Step title="Initializing">
    Transport is being initialized. This occurs in response to a `pcClient.initDevices()` call, where the transport is being set up in order to enumerate local media devices. If you call `connect()` and bypass `initDevices()`, the transport will skip this state and go directly to `Connecting`.
  </Step>

  <Step title="Initialized">
    Transport has been initialized and is ready to connect. This state is reached after a successful `pcClient.initDevices()` call and skipped if `initDevices()` is not used.
  </Step>

  <Step title="Authenticating">
    Your client has called `pcClient.startBot()` or `pcClient.startBotAndConnect()` and is waiting for a response from your server containing connection details for your transport (such as a session URL and token). Note: If you provide the `TransportConnectionParams` directly to `connect()` without calling either `startBot` methods, the transport will skip this state and go directly to `Connecting`.
  </Step>

  <Step title="Authenticated">
    Your client has called `pcClient.startBot()` or `pcClient.startBotAndConnect()` and has successfully received a response. If using `startBotAndConnect()`, it will quickly move into the `Connecting` state. Note: If you provide the `TransportConnectionParams` directly to `connect()` without calling either `startBot` methods, the transport will skip this state and go directly to `Connecting`.
  </Step>

  <Step title="Connecting">
    The transport is connecting to the server.
  </Step>

  <Step title="Connected">
    The transport has successfully connected to the session and is awaiting a client-ready signal (indicated audio and video tracks are ready to be sent and received).
  </Step>

  <Step title="Ready">Transport is ready and the session can begin.</Step>

  <Step title="Disconnecting">
    Transport is disconnecting from the session.
  </Step>

  <Step title="Error">
    An error occurred during the transport lifecycle. This indicates a fatal error and the transport should move quickly into the `Disconnected` state.
  </Step>
</Steps>

You can access the current transport state via `pcClient.state`, or by defining a callback or event:

```typescript theme={null}
// Callback
const pcClient = new PipecatClient({
  transport: new DailyTransport(),
  callbacks: {
    onTransportStateChange: (state) => {
      console.log(state);
    }
  //...
});

// Event
pcClient.on(RTVIEvent.TransportStateChanged, (e) => console.log(e));

// Client getter
console.log(pcClient.state); // Disconnected <TransportState>
```


# WebSocketTransport
Source: https://docs.pipecat.ai/client/js/transports/websocket

A lightweight transport for WebSocket based connections with Pipecat

`WebSocketTransport` enables a purely WebSocket based connection between clients and your Pipecat application. It implements bidirectional audio and video streaming using a WebSocket for real-time communication.

This transport is intended for lightweight implementations, particularly for local development and testing. It expects your Pipecat server to include the corresponding [`WebSocketTransport` server-side](/server/services/transport/websocket-server) implementation.

<Warning>
  The `WebSocketTransport` is best suited for server-server applications and prototyping client/server apps.

  For client/server production applications, we strongly recommend using a WebRTC-based transport for robust network and media handling. For more on WebRTC vs. Websocket communication, check out [this article](https://voiceaiandvoiceagents.com/#websockets-webrtc).
</Warning>

## Usage

### Basic Setup

```javascript theme={null}
import { PipecatClient } from "@pipecat-ai/client-js";
import {
  WebSocketTransport,
  ProtobufFrameSerializer,
} from "@pipecat-ai/websocket-transport";

const pcClient = new PipecatClient({
  transport: new WebSocketTransport({
    serializer: new ProtobufFrameSerializer(),
    recorderSampleRate: 8000,
    playerSampleRate: 8000,
  }),
  enableCam: false, // Default camera off
  enableMic: true, // Default microphone on
  callbacks: {
    // Event handlers
  },
});

await pcClient.connect({
  wsUrl: "ws://localhost:7860/ws", // Your WebSocket server URL
});
```

## API Reference

### Constructor Options

```typescript theme={null}
type WebSocketTransportOptions = {
  wsUrl?: string;
  serializer?: WebSocketSerializer;
  recorderSampleRate?: number;
  playerSampleRate?: number;
};

export interface WebSocketTransportConstructorOptions
  extends WebSocketTransportOptions {
  mediaManager?: MediaManager;
}
```

#### Properties

<ParamField name="wsUrl" type="string">
  URL of the WebSocket server. This is the endpoint your client will connect to
  for WebSocket communication.
</ParamField>

<ParamField name="serializer" type="WebSocketSerializer">
  The serializer to use for encoding/decoding messages sent over the WebSocket
  connection. The websocket-transport package provides two serializer options: -
  `ProtobufFrameSerializer`: Uses Protocol Buffers for serialization. -
  `TwilioSerializer`: Uses Twilio's serialization format. The main purpose of
  the TwilioSerializer is to allow testing the bots built to work with Twilio
  without having to make phone calls.
</ParamField>

<ParamField name="recorderSampleRate" type="number">
  Sample rate for which to encode the audio input. Default is `16000`.
</ParamField>

<ParamField name="playerSampleRate" type="number">
  Sample rate for which to decode the incoming audio for output. Default is
  `24000`.
</ParamField>

<ParamField name="mediaManager" type="MediaManager">
  The media manager to use for handling local audio and video streams. This
  should not be overridden unless you have a specific reason to use a different
  media manager. The default is `DailyMediaManager`, which is suitable for most
  use cases. Note that the `DailyMediaManager` does not use any of Daily's
  services, it simply takes advantage of vast media support provided by the
  Daily library.
</ParamField>

### TransportConnectionParams

The `WebSocketTransport` takes the same options as the constructor; `WebSocketTransportOptions`. Anything provided here will override the defaults set in the constructor. The `wsUrl` is required to establish a connection.

<CodeGroup>
  ```typescript client theme={null}
  pcClient.connect({
    wsUrl: 'http://localhost:7860/ws'
  });
  // OR...
  pcClient.startBotAndConnect({
    endpoint: '/api/start', // returns { wsUrl }
  });
  ```

  ```python server theme={null}
  # See
  # https://github.com/pipecat-ai/pipecat-examples/blob/main/websocket/server/server.py
  # for a complete example of how to implement the server-side endpoint.
  @app.websocket("/ws")
  async def websocket_endpoint(websocket: WebSocket):
      await websocket.accept()
      print("WebSocket connection accepted")
      try:
          await run_bot(websocket)
      except Exception as e:
          print(f"Exception in run_bot: {e}")


  @app.post("/api/start")
  async def start(request: Request) -> Dict[Any, Any]:
      ws_url = "ws://localhost:7860/ws"
      return {"wsUrl": ws_url}
  ```

  ```python bot theme={null}
  # See
  # https://github.com/pipecat-ai/pipecat-examples/blob/main/websocket/server/bot_websocket_server.py
  # for a complete example of a bot script using the WebSocketTransport.
  from pipecat.serializers.protobuf import ProtobufFrameSerializer
  from pipecat.transports.websocket.fastapi import (
      FastAPIWebsocketParams,
      FastAPIWebsocketTransport,
  )

  async def run_bot(websocket_client):
      ws_transport = FastAPIWebsocketTransport(
          websocket=websocket_client,
          params=FastAPIWebsocketParams(
              audio_in_enabled=True,
              audio_out_enabled=True,
              add_wav_header=False,
              serializer=ProtobufFrameSerializer(),
          ),
      )

      llm = ... # Initialize your LLM here, e.g., OpenAI, HuggingFace, etc.

      messages = [{ role: "system", content: "You are a helpful assistant." }]
      context = LLMContext(messages)
      user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
          context,
          user_params=LLMUserAggregatorParams(
              vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
          ),
      )

      pipeline = Pipeline(
          [
              ws_transport.input(),
              user_aggregator,
              llm,  # LLM
              ws_transport.output(),
              assistant_aggregator,
          ]
      )

      task = PipelineTask(
          pipeline,
          params,
      )
      ...
  ```
</CodeGroup>

### Methods

For most operations, you will not interact with the transport directly. Most methods have an equivalent in the `PipecatClient` and should be called from the `PipecatClient`. However, there is one transport-specific methods that you may need to call directly. When doing so, be sure to access your transport via the `transport` property of the `PipecatClient` instance.

<ResponseField name="handleUserAudioStream" type="method">
  If implementing your own serializer, you will need to pass the user audio
  stream to the transport via this method, which takes an `ArrayBuffer` of audio
  data.

  ```javascript theme={null}
  transport.handleUserAudioStream(chunk.data);
  ```
</ResponseField>

## Events

The transport implements the various [`PipecatClient` event handlers](/client/js/api-reference/callbacks).

## Reconnection Handling

The WebSocketTransport does provide reconnection handling. If the WebSocket connection is lost, it will attempt to reconnect twice. If all reconnection attempts fail, the transport will gracefully disconnect.

## More Information

<CardGroup>
  <Card title="WebSocket Demo" icon="video" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/websocket">
    Basic Agent example using a WebSocket transport
  </Card>

  <Card title="Twilio Demo" icon="video" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/twilio-chatbot">
    Example using a WebSocket transport to simulate a Twilio connection to a bot
  </Card>

  <Card title="Source" icon="github" href="https://github.com/pipecat-ai/pipecat-client-web-transports/tree/main/transports/websocket-transport">
    `WebSocketTransport`
  </Card>

  <Card title="Package" icon="browser" href="https://www.npmjs.com/package/@pipecat-ai/websocket-transport">
    `@pipecat-ai/websocket-transport`
  </Card>
</CardGroup>


# RTVIClient Migration Guide
Source: https://docs.pipecat.ai/client/migration-guide

A Guide to migrating from RTVIClient to PipecatClient

This guide will cover the high-level changes between the old `RTVIClient` and the new `PipecatClient`. For specific code updates, refer to the platform-specific migration guides.

## Key changes

* **Client Name**: The class name has changed from `RTVIClient` to `PipecatClient`.
* **Pipeline Connection**: Previously, the client expected a REST endpoint for gathering connection information as part of the constructor and was difficult to update or bypass. The new client expects connection information to be provided directly as part of the `connect()` method. For convenience in typical setups where the connection information is obtained from an API endpoint that also launches the server-side bot, we've introduced `startBot()` for hitting the endpoint and returning the response as well as `startBotAndConnect()` which does both the fetching and connecting and assumes the response contains the necessary connection details.
* **Actions and helpers**: These have gone away in favor of some built-in methods for doing common actions like function call handling and appending to the llm context or in the case of custom actions, a simple set of methods for sending messages to the bot and handling responses. See `registerFunctionCallHandler()`, `sendText()`, `sendClientMessage()`, and `sendClientRequest()` for more details.
* **Bot Configuration**: This functionality as been removed as a security measure, so that a client cannot inherently have the ability to override a bot configuration and use credentials to its own whims. If you need the client to initialize or update the bot configuration, you will need to do so through an API call to your backend or building on top of the client-server messaging, which has now been made easier.

## Migration guides

<CardGroup>
  <Card title="JavaScript" icon="code" href="/client/js/migration-guide">
    Migrate your JavaScript client code to the new `PipecatClient`
  </Card>

  <Card title="React" icon="react" href="/client/react/migration-guide">
    Update your React components to use the new `PipecatClient`
  </Card>

  <Card title="iOS" icon="swift" href="/client/ios/migration-guide">
    Update your iOS components to use the new `PipecatClient`
  </Card>

  <Card title="Android" icon="android" href="/client/android/migration-guide">
    Migrate your Android client code to the new `PipecatClient`
  </Card>

  <Card title="React Native" icon="react" href="/client/react-native/migration-guide">
    Update your React Native components to use the new `PipecatClient`
  </Card>
</CardGroup>


# API Reference
Source: https://docs.pipecat.ai/client/react-native/api-reference

API reference for the Pipecat React Native SDK

<Note>
  The Pipecat React Native SDK leverages the Pipecat JavaScript SDK for seamless integration with React Native applications.
  For detailed information, please reference to the [Javascript SDK docs](/client/js/api-reference/client-constructor).

  **Just ensure you use the appropriate transport layer for React Native.**
</Note>


# SDK Introduction
Source: https://docs.pipecat.ai/client/react-native/introduction

Build React Native applications with Pipecat's React Native client library

The Pipecat React Native SDK leverages the [Pipecat JavaScript SDK](/client/js/introduction) and its `PipecatClient` to provide seamless integration for React Native applications.
Since the JavaScript SDK is designed to work across both web and React Native platforms, the core functionalities remain the same:

* Device and media stream management
* Connecting to Pipecat bots
* Messaging with Pipecat bots and handling responses using the RTVI standard
* Managing session state and errors

The primary difference lies in the transport layer, which is tailored to support the unique requirements of the React Native environment.

For example, when using the SDK with React Native, you would install `RNDailyTransport` instead of `DailyTransport`.

## Installation

Install the SDK and a transport implementation. Follow the appropriate docs for each transport:

* [Daily](https://github.com/pipecat-ai/pipecat-client-react-native-transports/tree/main/transports/daily#installation)
* [SmallWebRTC](https://github.com/pipecat-ai/pipecat-client-react-native-transports/tree/main/transports/smallwebrtc#installation)

<Note>
  Installing the React Native automatically includes
  the corresponding version of the JavaScript SDK.
</Note>

## Requirements

This package introduces some constraints on what OS/SDK versions your project can support:

* iOS: Deployment target >= 15
* Android: `minSdkVersion` >= 24

## Quick start

Here's a simple example using Daily as the transport layer:

```tsx theme={null}
import { RNDailyTransport } from "@pipecat-ai/react-native-daily-transport";
import { PipecatClient } from "@pipecat-ai/client-js";

// Create and configure the client
let pipecatClient = new PipecatClient({
  transport: new RNDailyTransport(),
  enableMic: true,
  enableCam: false
});

// Connect to your bot
await pipecatClient.startBotAndConnect({
  endpoint: `${process.env.PIPECAT_API_URL || "/start"}`,
});
```

### More Examples

<CardGroup>
  <Card title="Basic Example" icon="file-code" href="https://github.com/pipecat-ai/pipecat-client-react-native-transports/tree/main/examples">
    A basic example demonstrating how to integrate an RNDailyTransport with a
    React Native project.
  </Card>

  <Card title="Daily Full Example" icon="mobile" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/simple-chatbot/client/react-native">
    A more comprehensive Daily example showcasing a more feature-rich React Native
    application along with a server-side bot component.
  </Card>

  <Card title="SmallWebRTC Full Example" icon="mobile" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/p2p-webrtc/video-transform">
    A more comprehensive SmallWebRTC example showcasing a more feature-rich React Native
    application along with a server-side bot component.
  </Card>
</CardGroup>

## Explore the SDK

The Pipecat React Native SDK leverages the Pipecat JavaScript SDK for seamless integration with React Native applications. For detailed information, refer to our JavaScript documentation.

> Just ensure you use the appropriate transport layer for React Native.

<CardGroup>
  <Card title="Client Constructor" icon="cube" href="/client/js/api-reference/client-constructor">
    Configure your client instance with transport and callbacks
  </Card>

  <Card title="Client Methods" icon="code" href="/client/js/api-reference/client-methods">
    Core methods for interacting with your bot
  </Card>

  <Card title="API Reference" icon="book" href="/client/js/api-reference">
    Detailed documentation of all available APIs
  </Card>
</CardGroup>


# RTVIClient Migration Guide for React Native
Source: https://docs.pipecat.ai/client/react-native/migration-guide

A Guide to migrating from a React Native RTVIClient to PipecatClient

This guide covers migrating from RTVIClient to the new `PipecatClient` in a React Native application. The new client introduces simplified configuration and improved client-server messaging. For an overview of the changes, see the top-level [RTVIClient Migration Guide](/client/migration-guide).

## Key Changes

<Tip>
  As primarily a wrapper around the JavaScript SDK with a unique Transport, the
  changes are the same as those in the [JavaScript migration
  guide](/client/js/migration-guide) with only the following transport-specific
  updates. PLEASE REFER TO THE JAVASCRIPT MIGRATION GUIDE FOR FULL DETAILS.
</Tip>

### 1. Transport Configuration

Same as with the JavaScript transports, the `RNDailyTransport` now accepts a constructor argument, allowing for providing a [Daily configuration](https://docs.daily.co/reference/rn-daily-js/daily-call-client/properties).

**Old**

```javascript theme={null}
const client = new RTVIClient({
  transport: new RNDailyTransport(),
  ...
});
```

**New**

```javascript theme={null}
const client = new PipecatClient({
  transport: new RNDailyTransport({
    userName: "Tina",
    reactNativeConfig: { disableAutoDeviceManagement: { audio: true } },
  }),
});
```

### 2. Connection Methods

Again, same as with the JavaScript transport, you can provide a [Daily configuration](https://docs.daily.co/reference/rn-daily-js/daily-call-client/properties) via one of the connection methods (`connect()` or `startBotAndConnect()`) on the `PipecatClient`.

**Old**

```javascript theme={null}
await client.connect();
```

**New**

There are three options for connecting now. For migration purposes, you are likely to use option 3 (`startBotAndConnect()`):

1. **Direct Connection**: Provide the connection details directly to `connect()`.

```javascript theme={null}
const cxnDetails = {
  url: "https://your-daily-room-url",
  token: "your-daily-token",
  userData: { favoriteFlower: "black-eyed susan" },
};
await client.connect(cxnDetails);
```

2. **Using startBot()**: Fetch connection details from an API endpoint and pass them to `connect()`.

```javascript theme={null}
let cxnDetails = await client.startBot({
  endpoint: "http://localhost:7860/connect",
  requestData: {
    // Any custom data your /connect endpoint requires
    llm_provider: "openai",
    initial_prompt: "You are a pirate captain",
    // Any additional data
  },
});
cxnDetails = modifyCxnDetails(cxnDetails); // Modify if needed
await client.connect(cxnDetails);
```

3. **Using startBotAndConnect()**: Fetch connection details and connect in one step.

<Note>
  When using `startBotAndConnect()`, the response from your endpoint is passed directly to the transport and should match the Daily configuration type.
</Note>

```javascript theme={null}
await client.startBotAndConnect({
  endpoint: 'http://localhost:7860/connect',
  requestData: {
    // Any custom data your /connect endpoint requires
    llm_provider: 'openai',
    initial_prompt: "You are a pirate captain",
    // Any additional data
  }
});
```


# Daily WebRTC Transport
Source: https://docs.pipecat.ai/client/react-native/transports/daily



The DailyTransport class provides a WebRTC transport layer using [Daily.co's](https://daily.co) infrastructure. It wraps a React-Native-Daily-JS call client to handle audio/video device management, WebRTC connections, and real-time communication between clients and bots. For complete documentation on Daily's API, see the [Daily RN API Reference](https://docs.daily.co/reference/rn-daily-js).

This transport is designed for production use cases, leveraging Daily's global infrastructure for low-latency, high-quality audio and video streaming. It expects your Pipecat server to include the corresponding [`DailyTransport` server-side](/server/services/transport/daily) implementation.

## Usage

### Basic Setup

```javascript theme={null}
import { RNDailyTransport } from "@pipecat-ai/react-native-daily-transport";
import { PipecatClient } from "@pipecat-ai/client-js";

// Create and configure the client
let pipecatClient = new PipecatClient({
  transport: new RNDailyTransport(),
  enableMic: true,
  enableCam: false
});

// Connect to your bot
await pipecatClient.startBotAndConnect({
  endpoint: `${process.env.PIPECAT_API_URL || "/start"}`,
});
```

## API Reference

### Constructor Options

<ParamField>
  The `DailyTransportConstructorOptions` extends the `DailyFactoryOptions` type that is accepted by the underlying Daily instance. These options are passed directly through to the Daily constructor. See the [Daily RN API Reference](https://docs.daily.co/reference/rn-daily-js/daily-call-client/properties) for a complete list of options.
</ParamField>

<Note>
  The Pipecat React Native SDK leverages the Pipecat JavaScript SDK for seamless integration with React Native applications.
  For detailed information, please reference to the [Javascript SDK docs](/client/js/transports/daily).

  **Just ensure you use the appropriate transport layer for React Native.**
</Note>

## More Information

<CardGroup>
  <Card title="Demo" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/simple-chatbot">
    Simple Chatbot Demo
  </Card>

  <Card title="Source" icon="github" href="https://github.com/pipecat-ai/pipecat-client-react-native-transports/tree/main/transports/daily">
    `RNDailyTransport`
  </Card>
</CardGroup>

<Card title="Package" icon="browser" href="https://www.npmjs.com/package/@pipecat-ai/react-native-daily-transport">
  `@pipecat-ai/react-native-daily-transport`
</Card>


# SmallWebRTCTransport
Source: https://docs.pipecat.ai/client/react-native/transports/small-webrtc

A lightweight WebRTC transport for peer-to-peer connections with Pipecat

`SmallWebRTCTransport` enables peer-to-peer WebRTC connections between clients and your Pipecat application. It implements bidirectional audio and video streaming using WebRTC for real-time communication.

This transport is intended for lightweight implementations, particularly for local development and testing. It expects your Pipecat server to include the corresponding [`SmallWebRTCTransport` server-side](/server/services/transport/small-webrtc) implementation.

## Usage

### Basic Setup

```javascript theme={null}
import { PipecatClient } from "@pipecat-ai/client-js";
import {
  RNSmallWebRTCTransport,
  SmallWebRTCTransportConstructorOptions,
} from '@pipecat-ai/react-native-small-webrtc-transport';
import { DailyMediaManager } from '@pipecat-ai/react-native-daily-media-manager/src';

const options: SmallWebRTCTransportConstructorOptions = {
  mediaManager: new DailyMediaManager()
};
const pcClient = new PipecatClient({
  transport: new RNSmallWebRTCTransport(options),
  enableCam: false, // Default camera off
  enableMic: true, // Default microphone on
  callbacks: {
    // Event handlers
  },
});

const connectParams: APIRequest = {
  endpoint: baseUrl + '/start'
};
await client?.startBotAndConnect(connectParams);
```

## API Reference

<Note>
  The Pipecat React Native SDK leverages the Pipecat JavaScript SDK for seamless integration with React Native applications.
  For detailed information, please reference to the [Javascript SDK docs](/client/js/transports/small-webrtc).

  **Just ensure you use the appropriate transport layer for React Native.**
</Note>

## More Information

<CardGroup>
  <Card title="Video Transform Demo" icon="video" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/p2p-webrtc/video-transform">
    Real-time video transformation example
  </Card>

  <Card title="Package" icon="browser" href="https://www.npmjs.com/package/@pipecat-ai/react-native-small-webrtc-transport">
    `@pipecat-ai/small-webrtc-transport`
  </Card>
</CardGroup>


# Components
Source: https://docs.pipecat.ai/client/react/components

Ready-to-use React components for Pipecat applications

The Pipecat React SDK provides several components for handling audio, video, and visualization in your application.

## PipecatClientProvider

The root component for providing Pipecat client context to your application.

```jsx theme={null}
<PipecatClientProvider client={pcClient}>
  {/* Child components */}
</PipecatClientProvider>
```

**Props**

<ParamField type="PipecatClient">
  A singleton instance of `PipecatClient`
</ParamField>

## PipecatClientAudio

Creates a new `<audio>` element that mounts the bot's audio track.

```jsx theme={null}
<PipecatClientAudio />
```

**Props**

No props required

## PipecatClientVideo

Creates a new `<video>` element that renders either the bot or local participant's video track.

```jsx theme={null}
<PipecatClientVideo
  participant="local"
  fit="cover"
  mirror
  onResize={({ aspectRatio, height, width }) => {
    console.log("Video dimensions changed:", { aspectRatio, height, width });
  }}
/>
```

**Props**

<ParamField type="('local' | 'bot')">
  Defines which participant's video track is rendered
</ParamField>

<ParamField type="('contain' | 'cover')">
  Defines whether the video should be fully contained or cover the box. Default: 'contain'
</ParamField>

<ParamField type="boolean">
  Forces the video to be mirrored, if set
</ParamField>

<ParamField type="function">
  Triggered whenever the video's rendered width or height changes
</ParamField>

## PipecatClientCamToggle

A headless component to read and set the local participant's camera state.

```jsx theme={null}
<PipecatClientCamToggle
  onCamEnabledChanged={(enabled) => console.log("Camera enabled:", enabled)}
  disabled={false}
>
  {({ disabled, isCamEnabled, onClick }) => (
    <button disabled={disabled} onClick={onClick}>
      {isCamEnabled ? "Disable Camera" : "Enable Camera"}
    </button>
  )}
</PipecatClientCamToggle>
```

**Props**

<ParamField type="function">
  Triggered whenever the local participant's camera state changes
</ParamField>

<ParamField type="boolean">
  If true, the component will not allow toggling the camera state. Default: false
</ParamField>

<ParamField type="function">
  A render prop that provides state and handlers to the children
</ParamField>

## PipecatClientMicToggle

A headless component to read and set the local participant's microphone state.

```jsx theme={null}
<PipecatClientMicToggle
  onMicEnabledChanged={(enabled) => console.log("Microphone enabled:", enabled)}
  disabled={false}
>
  {({ disabled, isMicEnabled, onClick }) => (
    <button disabled={disabled} onClick={onClick}>
      {isMicEnabled ? "Disable Microphone" : "Enable Microphone"}
    </button>
  )}
</PipecatClientMicToggle>
```

**Props**

<ParamField type="function">
  Triggered whenever the local participant's microphone state changes
</ParamField>

<ParamField type="boolean">
  If true, the component will not allow toggling the microphone state. Default: false
</ParamField>

<ParamField type="function">
  A render prop that provides state and handlers to the children
</ParamField>

## VoiceVisualizer

Renders a visual representation of audio input levels on a `<canvas>` element.

```jsx theme={null}
<VoiceVisualizer
  participantType="local"
  backgroundColor="white"
  barColor="black"
  barGap={1}
  barWidth={4}
  barMaxHeight={24}
/>
```

**Props**

<ParamField type="string">
  The participant type to visualize audio for
</ParamField>

<ParamField type="string">
  The background color of the canvas. Default: 'transparent'
</ParamField>

<ParamField type="string">
  The color of the audio level bars. Default: 'black'
</ParamField>

<ParamField type="number">
  The number of bars to display. Default: 5
</ParamField>

<ParamField type="number">
  The gap between bars in pixels. Default: 12
</ParamField>

<ParamField type="number">
  The width of each bar in pixels. Default: 30
</ParamField>

<ParamField type="number">
  The maximum height at full volume of each bar in pixels. Default: 120
</ParamField>


# Hooks
Source: https://docs.pipecat.ai/client/react/hooks

React hooks for accessing Pipecat client functionality

The Pipecat React SDK provides hooks for accessing client functionality, managing media devices, and handling events.

## usePipecatClient

Provides access to the `PipecatClient` instance originally passed to `PipecatClientProvider`.

```jsx theme={null}
import { usePipecatClient } from "@pipecat-ai/client-react";

function MyComponent() {
  const pcClient = usePipecatClient();

  await pcClient.startBotAndConnect({
    endpoint: '/api/start',
    requestData: {
      // Any custom data your /start endpoint requires
    }
  });
}
```

## useRTVIClientEvent

Allows subscribing to RTVI client events. It is advised to wrap handlers with `useCallback`.

```jsx theme={null}
import { useCallback } from "react";
import { RTVIEvent, TransportState } from "@pipecat-ai/client-js";
import { useRTVIClientEvent } from "@pipecat-ai/client-react";

function EventListener() {
  useRTVIClientEvent(
    RTVIEvent.TransportStateChanged,
    useCallback((transportState: TransportState) => {
      console.log("Transport state changed to", transportState);
    }, [])
  );
}
```

**Arguments**

<ParamField type="RTVIEvent" />

<ParamField type="function" />

## usePipecatClientMediaDevices

Manage and list available media devices.

```jsx theme={null}
import { usePipecatClientMediaDevices } from "@pipecat-ai/client-react";

function DeviceSelector() {
  const {
    availableCams,
    availableMics,
    selectedCam,
    selectedMic,
    updateCam,
    updateMic,
  } = usePipecatClientMediaDevices();

  return (
    <>
      <select
        name="cam"
        onChange={(ev) => updateCam(ev.target.value)}
        value={selectedCam?.deviceId}
      >
        {availableCams.map((cam) => (
          <option key={cam.deviceId} value={cam.deviceId}>
            {cam.label}
          </option>
        ))}
      </select>
      <select
        name="mic"
        onChange={(ev) => updateMic(ev.target.value)}
        value={selectedMic?.deviceId}
      >
        {availableMics.map((mic) => (
          <option key={mic.deviceId} value={mic.deviceId}>
            {mic.label}
          </option>
        ))}
      </select>
    </>
  );
}
```

## usePipecatClientMediaTrack

Access audio and video tracks.

```jsx theme={null}
import { usePipecatClientMediaTrack } from "@pipecat-ai/client-react";

function MyTracks() {
  const localAudioTrack = usePipecatClientMediaTrack("audio", "local");
  const botAudioTrack = usePipecatClientMediaTrack("audio", "bot");
}
```

**Arguments**

<ParamField type="'audio' | 'video'" />

<ParamField type="'bot' | 'local'" />

## usePipecatClientTransportState

Returns the current transport state.

```jsx theme={null}
import { usePipecatClientTransportState } from "@pipecat-ai/client-react";

function ConnectionStatus() {
  const transportState = usePipecatClientTransportState();
}
```

## usePipecatClientCamControl

Controls the local participant's camera state.

```jsx theme={null}
import { usePipecatClientCamControl } from "@pipecat-ai/client-react";
function CamToggle() {
  const { enableCam, isCamEnabled } = usePipecatClientCamControl();

  return (
    <button onClick={() => enableCam(!isCamEnabled)}>
      {isCamEnabled ? "Disable Camera" : "Enable Camera"}
    </button>
  );
}
```

## usePipecatClientMicControl

Controls the local participant's microphone state.

```jsx theme={null}
import { usePipecatClientMicControl } from "@pipecat-ai/client-react";
function MicToggle() {
  const { enableMic, isMicEnabled } = usePipecatClientMicControl();

  return (
    <button onClick={() => enableMic(!isMicEnabled)}>
      {isMicEnabled ? "Disable Microphone" : "Enable Microphone"}
    </button>
  );
}
```


# SDK Introduction
Source: https://docs.pipecat.ai/client/react/introduction

Build React applications with Pipecat's React client library

The Pipecat React SDK provides React-specific components and hooks for building voice and multimodal AI applications. It wraps the core JavaScript SDK functionality in an idiomatic React interface that handles:

* React context for client state management
* Components for audio and video rendering
* Hooks for accessing client functionality
* Media device management
* Event handling through hooks

## Installation

Install the SDK, core client, and a transport implementation (e.g. Daily for WebRTC):

```bash theme={null}
npm install @pipecat-ai/client-js
npm install @pipecat-ai/client-react
npm install @pipecat-ai/daily-transport
```

## Example

Here's a simple example using Daily as the transport layer:

```tsx theme={null}
import { PipecatClient } from "@pipecat-ai/client-js";
import {
  PipecatClientProvider,
  PipecatClientAudio,
  usePipecatClient,
} from "@pipecat-ai/client-react";
import { DailyTransport } from "@pipecat-ai/daily-transport";

// Create the client instance
const client = new PipecatClient({
  transport: new DailyTransport(),
  enableMic: true,
});

// Root component wraps the app with the provider
function App() {
  return (
    <PipecatClientProvider client={client}>
      <VoiceBot />
      <PipecatClientAudio />
    </PipecatClientProvider>
  );
}

// Component using the client
function VoiceBot() {
  const client = usePipecatClient();

  const handleClick = async () => {
    await client.startBotAndConnect({
      endpoint: `${process.env.PIPECAT_API_URL || "/api"}/connect`
    });
  };

  return (
    <button onClick={handleClick}>Start Conversation</button>;
  );
}
```

## Explore the SDK

<CardGroup>
  <Card title="Components" icon="puzzle-piece" href="/client/react/components">
    Ready-to-use components for audio, video, and visualization
  </Card>

  <Card title="Hooks" icon="code" href="/client/react/hooks">
    React hooks for accessing client functionality
  </Card>
</CardGroup>

The Pipecat React SDK builds on top of the [JavaScript SDK](/client/js/introduction) to provide an idiomatic React interface while maintaining compatibility with the RTVI standard.


# RTVIClient Migration Guide for React
Source: https://docs.pipecat.ai/client/react/migration-guide

A Guide to migrating from an RTVIClient to PipecatClient in React

This guide covers migrating from RTVIClient to the new `PipecatClient` in a React application. The new client introduces simplified configuration and improved client-server messaging. For an overview of the changes, see the top-level [RTVIClient Migration Guide](/client/migration-guide).

## Key Changes

### 1. Package and Class Names

**Old**

```javascript theme={null}
import { RTVIClient } from '@pipecat-ai/client-js';
```

**New**

```javascript theme={null}
import { PipecatClient } from '@pipecat-ai/client-js';
```

### 2. React Components and Hooks

**Old**

```javascript theme={null}
import { 
  RTVIClientProvider,
  RTVIClientAudio,
  RTVIClientVideo,
  useRTVIClient,
  useRTVIClientTransportState
} from '@pipecat-ai/client-react';
```

**New**

```javascript theme={null}
import {
  PipecatClientProvider,
  PipecatClientAudio,
  PipecatClientVideo,
  usePipecatClient,
  usePipecatClientTransportState
} from '@pipecat-ai/client-react';
```

### 3. Client and Transport Configuration

**Old**

```javascript theme={null}
const transport = new DailyTransport();
const client = new RTVIClient({
  transport,
  params: {
    baseUrl: 'http://localhost:7860',
    endpoints: {
      connect: '/connect'
    }
  }
});
```

**New**

```javascript theme={null}
const client = new PipecatClient({
  transport: new DailyTransport(),
  // Connection params moved to connect() call
});
```

### 4. Connection Method

Previously, `connect()` was called on the client instance without parameters. Now, you provide connection parameters directly to the `connect()` method. This allows for more flexibility and customization of the connection process. For ease of use, we've also introduced a `startBot()` and `startBotAndConnect()` method where you can provide an API endpoint that returns the connection parameters.

**Old**

```javascript theme={null}
await client.connect();
```

**New**

There are three options for connecting now. For migration purposes, you are likely to use option 3 (`startBotAndConnect()`):

1. **Direct Connection**: Provide the connection details directly to `connect()`.

```javascript theme={null}
const cxnDetails = {
  url: 'https://your-daily-room-url',
  token: 'your-daily-token'
};
await client.connect(cxnDetails);
```

2. **Using startBot()**: Fetch connection details from an API endpoint and pass them to `connect()`.

```javascript theme={null}
let cxnDetails = await client.startBot({
  endpoint: 'http://localhost:7860/connect',
  requestData: {
    // Any custom data your /connect endpoint requires
    llm_provider: 'openai',
    initial_prompt: "You are a pirate captain",
    // Any additional data
  }
});
cxnDetails = modifyCxnDetails(cxnDetails); // Modify if needed
await client.connect(cxnDetails);
```

3. **Using startBotAndConnect()**: Fetch connection details and connect in one step.

```javascript theme={null}
await client.startBotAndConnect({
  endpoint: 'http://localhost:7860/connect',
  requestData: {
    // Any custom data your /connect endpoint requires
    llm_provider: 'openai',
    initial_prompt: "You are a pirate captain",
    // Any additional data
  }
});
```

### 5. Function Call Handling

**Old**

```javascript theme={null}
let llmHelper = new LLMHelper({});
llmHelper.handleFunctionCall(async (data) => {
  return await this.handleFunctionCall(data.functionName, data.arguments);
});
client.registerHelper('openai', llmHelper);
```

**New**

```javascript theme={null}
client.registerFunctionCallHandler('functionName', async (data) => {
  // Handle function call
  return result;
});
```

## Breaking Changes

1. **Configuration Structure**: Connection parameters are now passed to connect() instead of constructor
2. **Helper System**: Removed in favor of direct `PipecatClient` member functions or client-server messaging.
3. **Component Names**: All React components renamed from RTVI prefix to Pipecat prefix
4. **Hook Names**: All React hooks renamed from useRTVI prefix to usePipecat prefix

## Migration Steps

1. Update package imports to use new names
2. Move connection configuration from constructor to startBot()/startBotAndConnect()/connect() methods
3. Replace any helper classes with corresponding `PipecatClient` methods or custom messaging
4. Update React component and hook names
5. Update any TypeScript types referencing old names


# The RTVI Standard
Source: https://docs.pipecat.ai/client/rtvi-standard

An overview of the RTVI standard and its key features

The RTVI (Real-Time Voice and Video Inference) standard defines a set of message types and structures sent between clients and servers. It is designed to facilitate real-time interactions between clients and AI applications that require voice, video, and text communication. It provides a consistent framework for building applications that can communicate with AI models and the backends running those models in real-time.

<Note>
  This page documents version 1.0 of the RTVI standard, released in June 2025.
</Note>

## Key Features

<CardGroup>
  <Card title="Connection Management" icon="plug">
    RTVI provides a flexible connection model that allows clients to connect to AI services and coordinate state.
  </Card>

  <Card title="Transcriptions" icon="text">
    The standard includes built-in support for real-time transcription of audio streams.
  </Card>

  <Card title="Client-Server Messaging" icon="comments">
    The standard defines a messaging protocol for sending and receiving messages between clients and servers, allowing for efficient communication of requests and responses.
  </Card>

  <Card title="Advanced LLM Interactions" icon="robot">
    The standard supports advanced interactions with large language models (LLMs), including context management, function call handline, and search results.
  </Card>

  <Card title="Service-Specific Insights" icon="lightbulb">
    RTVI supports events to provide insight into the input/output and state for typical services that exist in speech-to-speech workflows.
  </Card>

  <Card title="Metrics and Monitoring" icon="chart-line">
    RTVI provides mechanisms for collecting metrics and monitoring the performance of server-side services.
  </Card>
</CardGroup>

## Terms

* **Client**: The front-end application or user interface that interacts with the RTVI server.
* **Server**: The backend-end service that runs the AI framework and processes requests from the client.
* **User**: The end user interacting with the client application.
* **Bot**: The AI interacting with the user, technically an amalgamation of a large language model (LLM) and a text-to-speech (TTS) service.

## RTVI Message Format

The messages defined as part of the RTVI protocol adhere to the following format:

```json theme={null}
{
    "id": string,
    "label": "rtvi-ai",
    "type": string,
    "data": unknown
}
```

<ParamField type="string">
  A unique identifier for the message, used to correlate requests and responses.
</ParamField>

<ParamField type="string">
  A label that identifies this message as an RTVI message. This field is required and should always be set to `'rtvi-ai'`.
</ParamField>

<ParamField type="string">
  The type of message being sent. This field is required and should be set to one of the predefined RTVI message types listed below.
</ParamField>

<ParamField type="unknown">
  The payload of the message, which can be any data structure relevant to the message type.
</ParamField>

## RTVI Message Types

Following the above format, this section describes the various message types defined by the RTVI standard. Each message type has a specific purpose and structure, allowing for clear communication between clients and servers.

<Tip>
  Each message type below includes either a 🤖 or 🏄 emoji to denote whether the message is sent from the bot (🤖) or client (🏄).
</Tip>

### Connection Management

#### client-ready 🏄

Indicates that the client is ready to receive messages and interact with the server. Typically sent after the transport media channels have connected.

* **type**: `'client-ready'`
* **data**:
  * **version**: `string`

    The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations.

  * **about**: `AboutClient Object`

    An object containing information about the client, such as its rtvi-version, client library, and any other relevant metadata. The `AboutClient` object follows this structure:

    <Expandable title="AboutClient">
      <ParamField type="string" />

      <ParamField type="string" />

      <ParamField type="string" />

      <ParamField type="string" />

      <ParamField type="any">
        Any platform-specific details that may be relevant to the server. This could include information about the browser, operating system, or any other environment-specific data needed by the server. This field is optional and open-ended, so please be mindful of the data you include here and any security concerns that may arise from exposing sensitive or personal-identifiable information.
      </ParamField>
    </Expandable>

#### bot-ready 🤖

Indicates that the bot is ready to receive messages and interact with the client. Typically send after the transport media channels have connected.

* **type**: `'bot-ready'`
* **data**:
  * **version**: `string`

    The version of the RTVI standard being used. This is useful for ensuring compatibility between client and server implementations.

  * **about**: `any` (Optional)

    An object containing information about the server or bot. It's structure and value are both undefined by default. This provides flexibility to include any relevant metadata your client may need to know about the server at connection time, without any built-in security concerns. Please be mindful of the data you include here and any security concerns that may arise from exposing sensitive information.

#### disconnect-bot 🏄

Indicates that the client wishes to disconnect from the bot. Typically used when the client is shutting down or no longer needs to interact with the bot. Note: Disconnets should happen automatically when either the client or bot disconnects from the transport, so this message is intended for the case where a client may want to remain connected to the transport but no longer wishes to interact with the bot.

* **type**: `'disconnect-bot'`
* **data**: `undefined`

#### error 🤖

Indicates an error occurred during bot initialization or runtime.

* **type**: `'error'`
* **data**:
  * **message**: `string`

    Description of the error.

  * **fatal**: `boolean`

    Indicates if the error is fatal to the session.

### Transcription

#### user-started-speaking 🤖

Emitted when the user begins speaking

* **type**: `'user-started-speaking'`
* **data**: None

#### user-stopped-speaking 🤖

Emitted when the user stops speaking

* **type**: `'user-stopped-speaking'`
* **data**: None

#### bot-started-speaking 🤖

Emitted when the bot begins speaking

* **type**: `'bot-started-speaking'`
* **data**: None

#### bot-stopped-speaking 🤖

Emitted when the bot stops speaking

* **type**: `'bot-stopped-speaking'`
* **data**: None

#### user-transcription 🤖

Real-time transcription of user speech, including both partial and final results.

* **type**: `'user-transcription'`
* **data**:
  * **text**: `string`

    The transcribed text of the user.

  * **final**: `boolean`

    Indicates if this is a final transcription or a partial result.

  * **timestamp**: `string`

    The timestamp when the transcription was generated.

  * **user\_id**: `string`

    Identifier for the user who spoke.

#### bot-output 🤖

The best-effort representation of the bot's output text, including both spoken and unspoken text. In addition to transcriptions of spoken text, this message type may also include text that the bot outputs but does not speak (e.g., text sent to the client for display purposes only). Along with the text, this event includes a `spoken` flag to indicate whether the text was spoken by the bot or not and an `aggregated_by` field to indicate what the text represents (e.g. "sentence", "word", "code", "url").

* **type**: `'bot-output'`
* **data**:
  * **text**: `string`

    The output text from the bot.

  * **spoken**: `boolean`

    Indicates if this text was spoken by the bot.

  * **aggregated\_by**: `string`

    Indicates how the text was aggregated (e.g., "sentence", "word", "code", "url"). "sentence" and "word" are reserved aggregation types defined by the RTVI standard. Other aggregation types may be defined by custom text aggregators used by the server.

#### bot-transcription 🤖

<Warning>DEPRECATED in favor of `bot-output` in Pipecat version 0.0.95 and client-js version 1.5.0</Warning>

Transcription of the bot's speech. Note: This protocol currently does not match the user transcription format to support real-time timestamping for bot transcriptions. Rather, the event is typically sent for each sentence of the bot's response. This difference is currently due to limitations in TTS services which mostly do not support (or support well), accurate timing information. If/when this changes, this protocol may be updated to include the necessary timing information. For now, if you want to attempt real-time transcription to match your bot's speaking, you can try using the `bot-tts-text` message type.

* **type**: `'bot-transcription'`
* **data**:
  * **text**: `string`

    The transcribed text from the bot, typically aggregated at a per-sentence level.

### Client-Server Messaging

#### server-message 🤖

An arbitrary message sent from the server to the client. This can be used for custom interactions or commands. This message may be coupled with the `client-message` message type to handle responses from the client.

* **type**: `'server-message'`
* **data**: `any`

  The `data` can be any JSON-serializable object, formatted according to your own specifications.

#### client-message 🏄

An arbitrary message sent from the client to the server. This can be used for custom interactions or commands. This message may be coupled with the `server-response` message type to handle responses from the server.

* **type**: `'client-message'`
* **data**:

  * **t**: `string`
  * **d**: `unknown` (optional)

  The data payload should contain a `t` field indicating the type of message and an optional `d` field containing any custom, corresponding data needed for the message.

#### server-response 🤖

An message sent from the server to the client in response to a `client-message`. **IMPORTANT**: The `id` should match the `id` of the original `client-message` to correlate the response with the request.

* **type**: `'client-message'`
* **data**:

  * **t**: `string`
  * **d**: `unknown` (optional)

  The data payload should contain a `t` field indicating the type of message and an optional `d` field containing any custom, corresponding data needed for the message.

#### error-response 🤖

Error response to a specific client message. **IMPORTANT**: The `id` should match the `id` of the original `client-message` to correlate the response with the request.

* **type**: `'error-response'`
* **data**:
  * **error**: `string`

### Advanced LLM Interactions

#### send-text 🏄

A message sent from the client to the server to send text input to the LLM, appended to the user's context.

* **type**: `'send-text'`
* **data**:
  * **content**: `string`

    The text content to be appended to the user context.

  * **options**: `object` (optional)
    * **run\_immediately**: `boolean` (optional)

      If `true`, the pipeline should be interrupted and the LLM should process the input immediately after appending it to the context. Defaults to `true`.

    * **audio\_response**: `boolean` (optional)

      If `false`, the bot should bypass the TTS so that the bot does not respond to the text in audio. Defaults to `true`.

#### append-to-context 🏄

<Warning>DEPRECATED in favor of `send-text` in Pipecat version 0.0.86 and client-js version 0.0.4</Warning>

A message sent from the client to the server to append data to the context of the current llm conversation. This is useful for providing text-based content for the user or augmenting the context for the assistant.

* **type**: `'append-to-context'`
* **data**:
  * **role**: `"user"` | `"assistant"`

    The role the context should be appended to. Currently only supports `"user"` and `"assistant"`.

  * **content**: `unknown`

    The content to append to the context. This can be any data structure the llm understand.

  * **run\_immediately**: `boolean` (optional)

    Indicates whether the context should be run immediately after appending. Defaults to `false`. If set to `false`, the context will be appended but not executed until the next llm run.

#### llm-function-call 🤖

A function call request from the LLM, sent from the bot to the client. Note that for most cases, an LLM function call will be handled completely server-side. However, in the event that the call requires input from the client or the client needs to be aware of the function call, this message/response schema is required.

* **type**: `'llm-function-call'`
* **data**:
  * **function\_name**: `string`

    Name of the function to be called.

  * **tool\_call\_id**: `string`

    Unique identifier for this function call.

  * **args**: `Record<string, unknown>`

    Arguments to be passed to the function.

#### llm-function-call-result 🏄

The result of the function call requested by the LLM, returned from the client.

* **type**: `'llm-function-call-result'`
* **data**:
  * **function\_name**: `string`

    Name of the called function.

  * **tool\_call\_id**: `string`

    Identifier matching the original function call.

  * **args**: `Record<string, unknown>`

    Arguments that were passed to the function.

  * **result**: `Record<string, unknown> | string`

    The result returned by the function.

#### bot-llm-search-response 🤖

Search results from the LLM's knowledge base.

<Note>
  Currently, Google Gemini is the only LLM that supports built-in search. However, we expect other LLMs to follow suite, which is why this message type is defined as part of the RTVI standard. As more LLMs add support for this feature, the format of this message type may evolve to accommodate discrepancies.
</Note>

* **type**: `'bot-llm-search-response'`
* **data**:
  * **search\_result**: `string` (optional)

    Raw search result text.

  * **rendered\_content**: `string` (optional)

    Formatted version of the search results.

  * **origins**: `Array<Origin Object>`

    Source information and confidence scores for search results. The `Origin Object` follows this structure:

    ```json theme={null}
    {
        "site_uri": string (optional),
        "site_title": string (optional),
        "results": Array<{
            "text": string,
            "confidence": number[]
        }>
    }
    ```

**Example:**

```json theme={null}
"id": undefined
"label": "rtvi-ai"
"type": "bot-llm-search-response"
"data": {
    "origins": [
    {
        "results": [
            {
                "confidence": [0.9881149530410768],
                "text": "* Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm."
            },
            {
                "confidence": [0.9692034721374512],
                "ext": "* A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm."
            }
        ],
        "site_title": "vanderbilt.edu",
        "site_uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQHwif83VK9KAzrbMSGSBsKwL8vWfSfC9pgEWYKmStHyqiRoV1oe8j1S0nbwRg_iWgqAr9wUkiegu3ATC8Ll-cuE-vpzwElRHiJ2KgRYcqnOQMoOeokVpWqi"
    },
    {
        "results": [
            {
                "confidence": [0.6554043292999268],
                "text": "In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields."
            }
        ],
        "site_title": "wikipedia.org",
        "site_uri": "https://vertexaisearch.cloud.google.com/grounding-api-redirect/AUZIYQESbF-ijx78QbaglrhflHCUWdPTD4M6tYOQigW5hgsHNctRlAHu9ktfPmJx7DfoP5QicE0y-OQY1cRl9w4Id0btiFgLYSKIm2-SPtOHXeNrAlgA7mBnclaGrD7rgnLIbrjl8DgUEJrrvT0CKzuo"
    }],
    "rendered_content": "<style>\n.container ... </div>\n</div>\n",
    "search_result": "Several events are happening at Vanderbilt University:\n\n* Juneteenth: A Freedom Celebration is scheduled for June 18th from 12:00 pm to 2:00 pm.\n* A Juneteenth celebration at Fort Negley Park will take place on June 19th from 5:00 pm to 9:30 pm.\n\n  In addition to these events, Vanderbilt University is a large research institution with ongoing activities across many fields.  For the most recent news, you should check Vanderbilt's official news website.\n"
}
```

### Service-Specific Insights

#### bot-llm-started 🤖

Indicates LLM processing has begun

* **type**: `bot-llm-started`
* **data**: None

#### bot-llm-stopped 🤖

Indicates LLM processing has completed

* **type**: `bot-llm-stopped`
* **data**: None

#### user-llm-text 🤖

Aggregated user input text that is sent to the LLM.

* **type**: `'user-llm-text'`
* **data**:
  * **text**: `string`

    The user's input text to be processed by the LLM.

#### bot-llm-text 🤖

Individual tokens streamed from the LLM as they are generated.

* **type**: `'bot-llm-text'`
* **data**:
  * **text**: `string`

    The token text from the LLM.

#### bot-tts-started 🤖

Indicates text-to-speech (TTS) processing has begun.

* **type**: `'bot-tts-started'`
* **data**: None

#### bot-tts-stopped 🤖

Indicates text-to-speech (TTS) processing has completed.

* **type**: `'bot-tts-stopped'`
* **data**: None

#### bot-tts-text 🤖

The per-token text output of the text-to-speech (TTS) service (what the TTS actually says).

* **type**: `'bot-tts-text'`
* **data**:
  * **text**: `string`

    The text representation of the generated bot speech.

### Metrics and Monitoring

#### metrics 🤖

Performance metrics for various processing stages and services. Each message will contain entries for one or more of the metrics types: `processing`, `ttfb`, `characters`.

* **type**: `'metrics'`
* **data**:
  * **processing**: \[See Below] (optional)

    Processing time metrics.

  * **ttfb**:  \[See Below] (optional)

    Time to first byte metrics.

  * **characters**:  \[See Below] (optional)

    Character processing metrics.

For each metric type, the data structure is an array of objects with the following structure:

* **processor**: `string`

  The name of the processor or service that generated the metric.

* **value**: `number`

  The value of the metric, typically in milliseconds or character count.

* **model**: `string` (optional)

  The model of the service that generated the metric, if applicable.

**Example:**

```json theme={null}
{
  "type": "metrics",
  "data": {
    "processing": [
      {
        "model": "eleven_flash_v2_5",
        "processor": "ElevenLabsTTSService#0",
        "value": 0.0005140304565429688
      }
    ],
    "ttfb": [
      {
        "model": "eleven_flash_v2_5",
        "processor": "ElevenLabsTTSService#0",
        "value": 0.1573178768157959
      }
    ],
    "characters": [
      {
        "model": "eleven_flash_v2_5",
        "processor": "ElevenLabsTTSService#0",
        "value": 43
      }
    ]
  }
}
```


# Overview
Source: https://docs.pipecat.ai/deployment/overview

How to deploy your Pipecat bot online

## Introduction to Pipecat deployment

You've created your Pipecat bot, had a good chat with it locally, and are eager to share it with the world. Let’s explore how to approach deployment.

<Note>
  We're continually adding further deployment example projects to the Pipecat
  repo, which you can find
  [here](https://github.com/pipecat-ai/pipecat-examples/tree/main/deployment).
</Note>

### Deployment options

You have several options for deploying your Pipecat bot:

1. **Pipecat Cloud** - A purpose-built managed service designed specifically for Pipecat deployments
2. **Self-managed cloud deployment** - Deploy to providers like Fly.io, AWS, Google Cloud Run, etc.
3. **Custom infrastructure** - Run on your own servers or specialized AI infrastructure

The best choice depends on your specific requirements, scale, and expertise.

### Things you'll need

* **Transport service** - Pipecat has existing services for various different [media transport](/server/services/supported-services#transports) modes, such as WebRTC or WebSockets. If you're not using a third-party service for handling media transport, you'll want to make sure that infrastructure is hosted and ready to receive connections.

* **Deployment target** - You can deploy and run Pipecat bots anywhere that can run Python code - Google Cloud Run, AWS, Fly.io etc. We recommend providers that offer APIs, so you can programmatically spawn new bot agents on-demand.

* **Docker** - If you're targeting cloud architecture / VMs, they will most often expect a containerized app. It's worth having Docker installed and setup to run builds. We'll step through creating a `Dockerfile` in this documentation.

## Production-ready bots

In local development, things often work great as you're testing on controlled, stable network conditions. In real-world use-cases, however, your users will likely interact with your bot across a variety of different devices and network conditions.

WebSockets are fine for server-to-server communication or for initial development. But for production use, you'll likely want client-server audio that uses a protocol designed for real-time media transport. For an explanation of the difference between WebSockets and WebRTC, [see this post](https://www.daily.co/blog/how-to-talk-to-an-llm-with-your-voice/#webrtc).

If you're targeting scalable, client-server interactions, we recommend you use WebRTC for the best results.

## Supporting models

Most chatbots require very little in the way of system resources, but if you are making use of custom models or require GPU-powered infrastructure, it's important to consider how to pre-cache local resources so that they are not downloaded at runtime. Your bot processes / VMs should aim to launch and connect as quickly as possible, so the user is not left waiting.

Designing and operating a pool of workers is out of scope for our documentation, but we'll highlight best practices in all of our examples.

As an example of a supporting model, most Pipecat [examples](https://github.com/pipecat-ai/pipecat-examples) make use of Silero VAD which we recommend including as part of your Docker image (so it's cached and readily available when your bot runs.) Since the Silero model is quite small, this doesn't inflate the size of the container too much. You may, however, want to consider making large models availabile via a network volume and ensuring your bot knows where to find it.

For Silero specifically, you can read more about how to do download it directly [here](https://pypi.org/project/silero/#:~:text=Models%20are%20downloaded%20on%20demand,downloaded%20to%20a%20cache%20folder).

```python theme={null}
# Run at buildtime
torch.hub.load(
    repo_or_dir='snakers4/silero-vad',
    model='silero_vad',
    force_reload=True
    )
```

***

## Getting started

<Card title="Basic deployment pattern" icon="draw-square" href="./pattern">
  Introduction to a model for deploying Pipecat bots
</Card>

***

## Provider guides

Once you've familiarized yourself with the Pipecat [deployment pattern](./pattern), here are some guides that walk you through the process for various deployment options. Remember, your Pipecat bots are simply Python processes, so you can host them on whichever infrastructure or service best suits your project.

<CardGroup>
  <Card title="Pipecat Cloud" icon="cloud" href="./pipecat-cloud/introduction">
    Managed service purpose-built for Pipecat deployments
  </Card>

  <Card title="Fly.io" icon="paper-plane" href="./platforms/fly">
    For service-driven / CPU bots
  </Card>

  <Card title="Modal" icon="rocket" href="./platforms/modal">
    For running custom models on GPU-accelerated infrastructure
  </Card>

  <Card title="Cerebrium" icon="brain" href="./platforms/cerebrium">
    For GPU-accelerated models & specialized AI infrastructure
  </Card>
</CardGroup>


# Deployment pattern
Source: https://docs.pipecat.ai/deployment/pattern

Basic deployment pattern for Pipecat bots

## Project structure

A Pipecat project will often consist of the following:

<Card title="1. Bot file">
  E.g. `bot.py`. Your Pipecat bot / agent, containing all the pipelines that you
  want to run in order to communicate with an end-user. A bot file may take some
  command line arguments, such as a transport URL and configuration.
</Card>

<Card title="2. Bot runner">
  E.g. `bot_runner.py`. Typically a basic HTTP service that listens for incoming
  user requests and spawns the relevant bot file in response.
</Card>

<Note>
  You can call these files whatever you like! We use `bot.py` and
  `bot_runner.py` for simplicity.
</Note>

### Typical user / bot flow

There are many ways to approach connecting users to bots. Pipecat is unopinionated about *how* exactly you should do this, but it's helpful to put an idea forward.

At a very basic level, it may look something like this:

<Steps>
  <Step title="User requests to join session via client / app">
    Client initiates a HTTP request to a hosted bot runner service.
  </Step>

  <Step title="Bot runner handles the request">
    Authenticates, configures and instantiates everything necessary for the
    session to commence (e.g. a new WebSocket channel, or WebRTC room, etc.)
  </Step>

  <Step title="Bot runner spawns bot / agent">
    A new bot process / VM is created for the user to connect with (passing
    across any necessary configuration.) Your project may load just one bot
    file, contextually swap between multiple, or launch many at once.
  </Step>

  <Step title="Bot instantiates and joins session via specified transport credentials">
    Bot initializes, connects to the session (e.g. locally or via WebSockets,
    WebRTC etc) and runs your bot code.
  </Step>

  <Step title="Bot runner returns status to client">
    Once the bot is ready, the runner resolves the HTTP request with details for
    the client to connect.
  </Step>
</Steps>

<Frame>
  <img alt="Basic pipeline image" />
</Frame>

***

## Bot runner

The majority of use-cases require a way to trigger and manage a bot session over the internet. We call these bot runners; a HTTP service that provides a gateway for spawning bots on-demand.

The anatomy of a bot runner service is entirery arbitrary, but at very least will have a method that spawns a new bot process, for example:

```python theme={null}
import uvicorn

from fastapi import FastAPI, Request, HTTPException
from fastapi.responses import JSONResponse

app = FastAPI()

@app.post("/start_bot")
async def start_bot(request: Request) -> JSONResponse:
    # ... handle / authenticate the request
    # ... setup the transport session

    # Spawn a new bot process
    try:
       #... create a new bot instance
    except Exception as e:
        raise HTTPException(
            status_code=500, detail=f"Failed to start bot: {e}")

    # Return a URL for the user to join
    return JSONResponse({...})

if __name__ == "__main__":
    uvicorn.run(
        "bot_runner:app",
        host="0.0.0.0",
        port=7860
    )

```

This pseudo code defines a `/start_bot/` endpoint which listens for incoming user `POST` requests or webhooks, then configures the session (such as creating rooms on your transport provider) and instantiates a new bot process.

A client will typically require some information regarding the newly spawned bot, such as a web address, so we also return some JSON with the necessary details.

### Data transport

Your [transport layer](/server/services/transport/daily) is responsible for sending and receiving audio and video data over the internet.

You will have implemented a transport layer as part of your `bot.py` pipeline. This may be a service that you want to host and include in your deployment, or it may be a third-party service waiting for peers to connect (such as [Daily](https://www.daily.co), or a websocket.)

For this example, we will make use of Daily’s WebRTC transport. This will mean that our `bot_runner.py` will need to do some configuration when it spawns a new bot:

1. Create and configure a new Daily room for the session to take place in.
2. Issue both the bot and the user an authentication token to join the session.

Whatever you use for your transport layer, you’ll likely need to setup some environmental variables and run some custom code before spawning the agent.

### Best practice for bot files

A good pattern to work to is the assumption that your `bot.py` is an encapsulated entity and does not have any knowledge of the `bot_runner.py`. You should provide the bot everything it needs to operate during instantiation.

Sticking to this approach helps keep things simple and makes it easier to step through debugging (if the bot launched and something goes wrong, you know to look for errors in your bot file.)

***

## Example

Let's assume we have a fully service-driven `bot.py` that connects to a WebRTC session, passes audio transcription to GPT4 and returns audio text-to-speech with ElevenLabs.

We'll also use Silero voice activity detection, to better know when the user has stopped talking.

```python bot.py theme={null}
import argparse
import asyncio
import os
import sys

import aiohttp
from dotenv import load_dotenv
from loguru import logger

from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import EndFrame, LLMContextFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response import (
    LLMAssistantResponseAggregator,
    LLMUserResponseAggregator,
)
from pipecat.services.elevenlabs.tts import ElevenLabsTTSService
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.transports.daily.transport import DailyParams, DailyTransport

load_dotenv(override=True)

logger.remove(0)
logger.add(sys.stderr, level="DEBUG")

daily_api_key = os.getenv("DAILY_API_KEY", "")
daily_api_url = os.getenv("DAILY_API_URL", "https://api.daily.co/v1")


async def main(room_url: str, token: str):
    async with aiohttp.ClientSession() as session:
        transport = DailyTransport(
            room_url,
            token,
            "Chatbot",
            DailyParams(
                api_url=daily_api_url,
                api_key=daily_api_key,
                audio_in_enabled=True,
                audio_out_enabled=True,
                video_out_enabled=False,
                transcription_enabled=True,
            ),
        )

        tts = ElevenLabsTTSService(
            aiohttp_session=session,
            api_key=os.getenv("ELEVENLABS_API_KEY", ""),
            voice_id=os.getenv("ELEVENLABS_VOICE_ID", ""),
        )

        llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"), model="gpt-4o")

        messages = [
            {
                "role": "system",
                "content": "You are Chatbot, a friendly, helpful robot. Your output will be converted to audio so don't include special characters other than '!' or '?' in your answers. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by saying hello.",
            },
        ]

        context = LLMContext(messages)
        user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
            context,
            user_params=LLMUserAggregatorParams(
                vad_analyzer=SileroVADAnalyzer(),
            ),
        )

        pipeline = Pipeline(
            [
                transport.input(),
                user_aggregator,
                llm,
                tts,
                transport.output(),
                assistant_aggregator,
            ]
        )

        task = PipelineTask(pipeline)

        @transport.event_handler("on_first_participant_joined")
        async def on_first_participant_joined(transport, participant):
            transport.capture_participant_transcription(participant["id"])
            await task.queue_frames([LLMContextFrame(LLMContext(messages))])

        @transport.event_handler("on_participant_left")
        async def on_participant_left(transport, participant, reason):
            await task.queue_frame(EndFrame())

        @transport.event_handler("on_call_state_updated")
        async def on_call_state_updated(transport, state):
            if state == "left":
                await task.queue_frame(EndFrame())

        runner = PipelineRunner()

        await runner.run(task)


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Pipecat Bot")
    parser.add_argument("-u", type=str, help="Room URL")
    parser.add_argument("-t", type=str, help="Token")
    config = parser.parse_args()

    asyncio.run(main(config.u, config.t))
```

### HTTP API

To launch this bot, let's create a `bot_runner.py` that:

* Creates an API for users to send requests to.
* Launches a bot as a subprocess.

```python bot_runner.py theme={null}
import os
import argparse
import subprocess

from pipecat.transports.services.helpers.daily_rest import DailyRESTHelper, DailyRoomObject, DailyRoomProperties, DailyRoomParams

from fastapi import FastAPI, Request, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import JSONResponse

# Load API keys from env
from dotenv import load_dotenv
load_dotenv(override=True)


# ------------ Configuration ------------ #

MAX_SESSION_TIME = 5 * 60  # 5 minutes
# List of require env vars our bot requires
REQUIRED_ENV_VARS = [
    'DAILY_API_KEY',
    'OPENAI_API_KEY',
    'ELEVENLABS_API_KEY',
    'ELEVENLABS_VOICE_ID']

daily_rest_helper = DailyRESTHelper(
    os.getenv("DAILY_API_KEY", ""),
    os.getenv("DAILY_API_URL", 'https://api.daily.co/v1'))


# ----------------- API ----------------- #

app = FastAPI()

app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"]
)

# ----------------- Main ----------------- #


@app.post("/start_bot")
async def start_bot(request: Request) -> JSONResponse:
    try:
        # Grab any data included in the post request
        data = await request.json()
    except Exception as e:
        pass

    # Create a new Daily WebRTC room for the session to take place in
    try:
        params = DailyRoomParams(
            properties=DailyRoomProperties()
        )
        room: DailyRoomObject = daily_rest_helper.create_room(params=params)
    except Exception as e:
        raise HTTPException(
            status_code=500,
            detail=f"Unable to provision room {e}")

    # Give the agent a token to join the session
    token = daily_rest_helper.get_token(room.url, MAX_SESSION_TIME)

    # Return an error if we were unable to create a room or a token
    if not room or not token:
        raise HTTPException(
            status_code=500, detail=f"Failed to get token for room: {room_url}")

    try:
        # Start a new subprocess, passing the room and token to the bot file
        subprocess.Popen(
            [f"python3 -m bot -u {room.url} -t {token}"],
            shell=True,
            bufsize=1,
            cwd=os.path.dirname(os.path.abspath(__file__)))
    except Exception as e:
        raise HTTPException(
            status_code=500, detail=f"Failed to start subprocess: {e}")

    # Grab a token for the user to join with
    user_token = daily_rest_helper.get_token(room.url, MAX_SESSION_TIME)

    # Return the room url and user token back to the user
    return JSONResponse({
        "room_url": room.url,
        "token": user_token,
    })


if __name__ == "__main__":
    # Check for required environment variables
    for env_var in REQUIRED_ENV_VARS:
        if env_var not in os.environ:
            raise Exception(f"Missing environment variable: {env_var}.")

    parser = argparse.ArgumentParser(description="Pipecat Bot Runner")
    parser.add_argument("--host", type=str,
                        default=os.getenv("HOST", "0.0.0.0"), help="Host address")
    parser.add_argument("--port", type=int,
                        default=os.getenv("PORT", 7860), help="Port number")
    parser.add_argument("--reload", action="store_true",
                        default=False, help="Reload code on change")

    config = parser.parse_args()

    try:
        import uvicorn

        uvicorn.run(
            "bot_runner:app",
            host=config.host,
            port=config.port,
            reload=config.reload
        )

    except KeyboardInterrupt:
        print("Pipecat runner shutting down...")

```

### Dockerfile

Since our bot is just using Python, our Dockerfile can be quite simple:

<CodeGroup>
  ```shell Dockerfile theme={null}
  FROM python:3.11-bullseye

  # Open port 7860 for http service

  ENV FAST_API_PORT=7860
  EXPOSE 7860

  # Install Python dependencies

  COPY \*.py .
  COPY ./requirements.txt requirements.txt
  RUN pip3 install --no-cache-dir --upgrade -r requirements.txt

  # Install models

  RUN python3 install_deps.py

  # Start the FastAPI server

  CMD python3 bot_runner.py --port ${FAST_API_PORT}

  ```

  ```python install_deps.py theme={null}
  import torch

  # Download (cache) the Silero VAD model
  torch.hub.load(repo_or_dir='snakers4/silero-vad', model='silero_vad', force_reload=True)
  ```
</CodeGroup>

The bot runner and bot `requirements.txt`:

```shell requirements.txt theme={null}
pipecat-ai[daily,openai,silero]
fastapi
uvicorn
python-dotenv
```

And finally, let's create a `.env` file with our service keys

```shell .env theme={null}
DAILY_API_KEY=...
OPENAI_API_KEY=...
ELEVENLABS_API_KEY=...
ELEVENLABS_VOICE_ID=...
```

***

### How it works

Right now, this runner is spawning `bot.py` as a subprocess. When spawning the process, we pass through the transport room and token as system arguments to our bot, so it knows where to connect.

Subprocesses serve as a great way to test out your bot in the cloud without too much hassle, but depending on the size of the host machine, it will likely not hold up well under load.

Whilst some bots are just simple operators between the transport and third-party AI services (such as OpenAI), others have somewhat CPU-intensive operations, such as running and loading VAD models, so you may find you’re only able to scale this to support up to 5-10 concurrent bots.

Scaling your setup would require virtualizing your bot with it's own set of system resources, the process of which depends on your cloud provider.

### Best practices

In an ideal world, we'd recommend containerizing your bot and bot runner independently so you can deploy each without any unnecessary dependencies or models.

Most cloud providers will offer a way to deploy various images programmatically, which we explore in the various provider examples in these docs.

For the sake of simplicity defining this pattern, we're just using one container for everything.

### Build and run

We should now have a project that contains the following files:

* `bot.py`
* `bot_runner.py`
* `requirements.txt`
* `.env`
* `Dockerfile`

You can now `docker build ...` and deploy your container. Of course, you can still work with your bot in local development too:

```shell theme={null}
# Install and activate a virtual env
python -m venv venv
source venv/bin/activate # or OS equivalent

pip install -r requirements.txt

python bot_runner.py --host localhost --reload
```


# Accounts and Organizations
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/fundamentals/accounts-and-organizations

Managing your account and collaborating on agents as part of a team

Pipecat Cloud accounts are issued a personal workspace where you can deploy agents independently.
User workspaces are unique and isolated from one another, making them unsuitable for collaboration.

## Authentication

Before you can issue any commands via the CLI, you must authenticate with your Pipecat Cloud account.

```bash theme={null}
pipecat cloud auth login
```

This command will open a browser window and prompt you to authenticate with your Pipecat Cloud account.

You can verify a login was successful by running the `whoami` command:

```bash theme={null}
pipecat cloud auth whoami
```

You can sign out of your account by running the following:

```bash theme={null}
pipecat cloud auth logout
```

**Credentials**

Access tokens are stored locally on your device at `~/.config/pipecatcloud/pipecatcloud.toml`. You can override the default location by setting the `PIPECAT_CONFIG_PATH` environment variable.
If you need to revoke access to your account, please sign out of the Pipecat Cloud dashboard to invalidate your access token.

## Organizations

Working as part of a team requires creating an organization. Organizations are multi-tenant workspaces that enable collaboration on deployments.

* Billing and integrations are managed at the organization level.
* Agents and secrets are shared across the organization, with varying roles and privileges.
* Organizations have dedicated access tokens for initializing agent instances.

<Info>
  Deployments cannot be shared or moved between organizations or user
  accounts. If you plan to collaborate with others, we recommend creating
  a new organization.
</Info>

### Creating organizations

Organization creation must be done via the [Pipecat Cloud dashboard<Icon icon="arrow-up-right-from-square" />](https://pipecat.daily.co/).

### Inviting accounts to your organization

You can invite a user to your organization by their email address using the [Pipecat Cloud dashboard<Icon icon="arrow-up-right-from-square" />](https://pipecat.daily.co/).

### Working with organizations

You can get a list of available organizations that your account is a member of by running the following:

```bash theme={null}
pipecat cloud organizations list
```

You can switch to a different organization by running the following:

```bash theme={null}
pipecat cloud organizations select
```

Switching organizations will target all CLI commands to the selected organization. For example, `pipecat cloud deploy` will issue the deployment request to your select organization.

Alternatively, you issue commands with the optional `--org` flag, for example:

```bash theme={null}
pipecat cloud deploy my-agent image-name --org my-organization
```

## API keys

Pipecat Cloud requires using an API key for some CLI commands or interfacing via the REST API.

API keys are issued at the user workspace and organization level.

You can create up to 5 private and 5 public API keys for different agents / purposes.

### Private API keys

Used to authenticate with private REST API endpoints. Private API keys
are typically used for administrative tasks and should not be shared or used by end-user applications.

Examples of where you may want to use private API keys include:

* Managing organizations and users via the REST API
* Server-to-server deployment scripts
* GitHub actions

You should not share your private API key with anyone.

### Public API keys

Public API keys are suitable for CLI usage for commands that could be invoked by end-user actions on the platform.

Examples of where you may want to use public API keys include:

* [Starting an agent instance](./active-sessions) (via the CLI or REST API)
* Server-to-server end-user applications
* Client-side while developing / iterating

Public API keys allow starting new agent processes, so while they are generally safe to share, you should still be cautious to avoid
malicious use (i.e. triggering your account rate limit.) We recommend keeping your public API key server-side and not sharing it in public repositories.

### Managing API keys

You can create, cycle, revoke or delete API keys via the [Pipecat Cloud dashboard <Icon icon="arrow-up-right-from-square" />](https://pipecat.daily.co) and CLI.

```bash theme={null}
# Create a new API key
pipecat cloud organizations keys create

# List available keys
pipecat cloud organizations keys list
```

When using the CLI, you can specify a default API key to use for all commands that require it:

```bash theme={null}
pipecat cloud organizations keys use
```

Selecting a key will modify your local configuration file and use it for all subsequent commands.

Please be aware that if the key is cycled or revoked, you will need to run the `keys use` command again to update your local configuration file with a new key.


# Starting Sessions
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/fundamentals/active-sessions

Serving user requests with active sessions.

Once you have made a successful deployment, you can start on-demand sessions to begin interacting with your agent.

An active session is a running instance of your agent code that connects with you or your end-users.
Sessions are created in response to a request and scaled up or down based on your deployment configuration (see [Scaling](./scaling)).

<Info>
  You must have an active billing account to start sessions. Usage is billed for
  the duration of each session, please refer to
  [pricing](https://www.daily.co/pricing/pipecat-cloud) for more information.
</Info>

Starting active sessions can be achieved programmatically via the REST API and Python library or directly via the CLI.

## Running an agent

Requests to start an agent requires passing a [public API key](./accounts-and-organizations#api-keys). Keys are used to ensure that only authorized users or apps can start sessions and can be cycled or revoked at any time.

### Development pattern

A typical application will start a session in response to a user request (via a web UI, mobile app, phone call or other client). Pipecat Cloud takes care of your session infrastructure and scaling, so you can focus on building your application logic.

Developers should:

1. Provide a secure endpoint to receive requests that keeps your API key secret

2. Pass any required data to the agent when starting a session (such as pipeline configuration, transport session data, etc.)

3. Handle any errors that may occur during the start process

### Using REST

The Pipecat Cloud REST API provides an endpoint to start a session with your agent over HTTP:

<CodeGroup>
  ```bash cURL theme={null}
  curl --location --request POST 'https://api.pipecat.daily.co/v1/public/{agent_name}/start' \
    --header 'Authorization: Bearer PUBLIC_API_KEY' \
    --header 'Content-Type: application/json' \
    --data-raw '{
      "createDailyRoom": true,
      "body": {"custom": "data"}
    }'
  ```

  ```python Python theme={null}
  endpoint = "https://api.pipecat.daily.co/v1/public/{service}/start"
  agent_name = "my-first-agent"
  api_key = "pk_..."

  async with aiohttp.ClientSession() as session:
      response = await session.post(
          f"{endpoint.format(service=agent_name)}",
          headers={"Authorization": f"Bearer {api_key}"},
          json={
              "createDailyRoom": True,  # Creates a Daily room
              "body": {"custom": "data"}  # Data to pass to your agent
          }
      )
  ```
</CodeGroup>

### Using Python

The Pipecat Cloud Python library (`pip install pipecatcloud`) provides helper methods to start a session with your agent:

```python theme={null}
import asyncio

from pipecatcloud.exception import AgentStartError
from pipecatcloud.session import Session, SessionParams


async def main():
    try:
        # Create session object
        session = Session(
            agent_name="my-first-agent",
            api_key=API_KEY,  # Replace with your actual API key
            params=SessionParams(
                use_daily=True,  # Optional: Creates a Daily room
                daily_room_properties={"start_video_off": False},
                data={"key": "value"},
            ),
        )

        # Start the session
        response = await session.start()

        # Get Daily room URL
        daily_url = f"{response['dailyRoom']}?t={response['dailyToken']}"
        print(f"Join Daily room: {daily_url}")

    except AgentStartError as e:
        print(f"Error starting agent: {e}")
    except Exception as e:
        print(f"Unexpected error: {e}")


# Run the async function
if __name__ == "__main__":
    asyncio.run(main())
```

### Using the CLI

To start an agent via the command line interface:

```shell theme={null}
pipecat cloud agent start my-first-agent --api-key pk_...
```

You can select and use a default API key for the agent start command:

```shell theme={null}
pipecat cloud organizations keys use my-default-key
```

See the [CLI reference](../cli-reference/agent#start) for more information.

## Passing data

Most agents will require some initial data, such pipeline configuration, transport session credentials, etc.

For sensitive data, we recommend using [secrets](./secrets) to store and access your data securely.

You pass a data blob through to the start request (please note that the data must be JSON-serializable):

```bash theme={null}
pipecat cloud agent start my-first-agent --data '{"room": "my-room"}'

# or from a file
pipecat cloud agent start my-first-agent --data-file data.json
```

Data passed to your session can be accessed in your agent code via the `bot()` method, which receives session arguments:

<Tabs>
  <Tab title="Standard Session">
    ```python theme={null}
    # bot.py
    from pipecat.runner.types import PipecatRunnerArguments

    async def bot(args: PipecatRunnerArguments):
        # Access the session ID
        logger.info(f"Session started with ID: {args.session_id}")
        
        # Access the custom data passed to the session
        logger.info(f"Session data: {args.body}")
        
        # ...
    ```
  </Tab>

  <Tab title="Daily Voice Session">
    ```python theme={null}
    # bot.py for voice agent
    from loguru import logger
    from pipecat.runner.types import DailyRunnerArguments

    async def bot(args: DailyRunnerArguments):
        """Main bot entry point compatible with the FastAPI route handler.

        Args:
            room_url: The Daily room URL
            token: The Daily room token
            body: The configuration object from the request body
        """
        logger.info(f"Bot process initialized {args.room_url} {args.token}")
        logger.info(f"Custom data: {args.body}")
        logger.info(f"Session ID: {args.session_id}")

        try:
            await main(args.room_url, args.token)
            logger.info("Bot process completed")
        except Exception as e:
            logger.exception(f"Error in bot process: {str(e)}")
            raise
    ```
  </Tab>

  <Tab title="WebSocket Session">
    ```python theme={null}
    # bot.py for WebSocket agent
    from loguru import logger
    from pipecat.runner.types import WebSocketRunnerArguments

    async def bot(args: WebSocketRunnerArguments):
        """Main bot entry point for WebSocket connections.

        Args:
            ws: The WebSocket connection
        """
        logger.info("WebSocket bot process initialized")
        logger.info(f"Custom data: {args.body}")
        logger.info(f"Session ID: {args.session_id}")

        try:
            await main(args.websocket)
            logger.info("WebSocket bot process completed")
        except Exception as e:
            logger.exception(f"Error in WebSocket bot process: {str(e)}")
            raise
    ```
  </Tab>
</Tabs>

For more advanced data requirements, your application should run its own data fetching logic before a session request is made.

If you require your agent to fetch it's own data, be mindful of any blocking requests that could slow down the start process.

## Agent capacity

Deployments have an upper limit for the number of sessions that can be started concurrently.
This helps developers implement cost control and workload management and can be adjusted per deployment.

See [Scaling](./scaling) for more information.

<Warning>
  The default instance pool capacity is **50 active sessions per
  deployment**. Please contact us at [help@daily.co](mailto:help@daily.co) or via
  [Discord](https://discord.gg/dailyco) if you require more capacity.
</Warning>

Developers are responsible for handling capacity errors when starting sessions. When your instance pool is at capacity, the start request will fail with a `429` status and code.

Here is a simple example of how to handle a capacity error:

<CodeGroup>
  ```bash REST theme={null}
  curl --location --request POST 'https://api.pipecat.daily.co/v1/public/{agent_name}/start' \
  --header 'Authorization: Bearer PUBLIC_API_KEY' \
  --header 'Content-Type: application/json' \
  --data-raw '{}'

  >> {
  >>	"status": 429,
  >>	"code": "429",
  >>	"message": "Rate limit exceeded. Please try again later"
  >> }
  ```

  ```python Python theme={null}
  from pipecatcloud import Session, SessionParams
  from pipecatcloud.exception import AgentStartError

  try:
      session = Session(
          agent_name="my-first-agent",
          api_key="pk_...",
          params=SessionParams(
              use_daily=False,
              data={}
          )
      )
  except AgentStartError as e:
      # This will catch capacity errors and other start failures
      print(f"Error starting agent: {e}")
  except Exception as e:
      print(f"Unexpected error: {e}")
  ```
</CodeGroup>

## Logging and monitoring

Session observability can be accessed via the [Dashboard <Icon icon="arrow-up-right-from-square" />](https://pipecat.daily.co) or from the CLI.

### Active sessions

To view active sessions via the CLI:

```shell theme={null}
pipecat cloud agent sessions [my-first-agent]
```

This command will list all active sessions for the specified agent, alongside a session ID.

### Session logs

To view individual session logs, use the following command:

```shell theme={null}
pipecat cloud agent logs [my-first-agent] --session [session-id]
```

For more information, see [Logging and Observability](./logging).

## Stopping sessions

You can issue a termination request to stop an active session. This will immediately stop the instance so be aware of disrupting any ongoing user interactions.

```shell theme={null}
pipecat cloud agent stop my-first-agent --session [session-id]
```

Deleting an agent deployment will block any new sessions from starting, but will not stop any active sessions. You must stop each session individually.

## Usage

You are charged for the duration of each session, see [pricing](https://www.daily.co/pricing/pipecat-cloud).

The exact duration of session is calculated based on:

* Time spent instantiating the agent
* Time spent running the agent
* Time spent tearing down the agent

You can monitor your usage on your [Dashboard <Icon icon="arrow-up-right-from-square" />](https://pipecat.daily.co).


# Agent Images
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/fundamentals/agent-images

How to containerize your agent project

Pipecat Cloud agents are designed to be run from containerized images. This allows you to run the agent in a controlled environment, with all the dependencies and configurations needed.

Your project defines the environment that your agent will run using Docker and built using a Dockerfile in the root directory of the project.

For example, your Dockerfile might look like this:

<Tabs>
  <Tab title="Dockerfile using uv">
    ```dockerfile theme={null}
    FROM dailyco/pipecat-base:latest

    # Enable bytecode compilation
    ENV UV_COMPILE_BYTECODE=1

    # Copy from the cache instead of linking since it's a mounted volume
    ENV UV_LINK_MODE=copy

    # Uncomment this if you wish to print a summary of the features available in the base image.
    # ENV PCC_LOG_FEATURES_SUMMARY=true

    # Install the project's dependencies using the lockfile and settings
    RUN --mount=type=cache,target=/root/.cache/uv \
        --mount=type=bind,source=uv.lock,target=uv.lock \
        --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
        uv sync --locked --no-install-project --no-dev

    # Copy the application code
    COPY ./bot.py bot.py
    ```
  </Tab>

  <Tab title="Dockerfile using pip">
    ```dockerfile theme={null}
    FROM dailyco/pipecat-base:latest

    COPY ./requirements.txt requirements.txt

    # Uncomment this if you wish to print a summary of the features available in the base image.
    # ENV PCC_LOG_FEATURES_SUMMARY=true

    RUN pip install --no-cache-dir --upgrade -r requirements.txt

    COPY ./bot.py bot.py
    ```
  </Tab>
</Tabs>

<Frame>
  <img />
</Frame>

## Using an official base image

Pipecat Cloud provides a series of base images that we recommend for most use-cases. Base images provide:

* Simplified development and deployment
* Optimizations for performance and security
* Pre-installed system dependencies for most multi-modal agent use-cases

Using a base image reduces complexity in your project but requires you to adhere to a specific project structure.

* Your project must contain a `bot.py` file that defines the agent pipeline
* The `bot.py` must contain a `bot()` method that is the entry point for your agent pipeline
* The `bot()` method must be asynchronous, e.g. `async def bot():`

You do not need to specify a `CMD` as part of your Dockerfile - the base image is configured to run your `bot.py` module.

You can browse available base images in the [Pipecat Cloud Docker Hub <Icon icon="arrow-up-right-from-square" />](https://hub.docker.com/u/dailyco).

### Reserved paths

The base image uses the `/app` directory for its internal operation. **Avoid copying files to `/app`** in your Dockerfile to prevent conflicts with system files.

<Warning>
  Writing to `/app` may overwrite critical system files and cause your
  deployment to fail. Your agent code should be placed in `bot.py` (required)
  and any additional modules, which the base image handles automatically.
</Warning>

### Reserved HTTP routes

The base image exposes the following HTTP routes for Pipecat Cloud platform integration:

| Route        | Method    | Description                                                                   |
| ------------ | --------- | ----------------------------------------------------------------------------- |
| `/bot`       | POST      | Main entry point called by Pipecat Cloud to start agent sessions              |
| `/ws`        | WebSocket | WebSocket endpoint for real-time communication (e.g., telephony integrations) |
| `/api/offer` | POST      | SmallWebRTC offer handling for peer-to-peer connections                       |
| `/api/offer` | PATCH     | SmallWebRTC ICE candidate handling                                            |
| `/whatsapp`  | POST      | WhatsApp Business webhook endpoint                                            |

<Note>
  These routes are automatically configured based on available features. For
  example, the `/whatsapp` route is only available when WhatsApp environment
  variables are configured.
</Note>

### Reserved environment variables

The base image uses the following environment variables for configuration:

| Variable                   | Description                                                                 |
| -------------------------- | --------------------------------------------------------------------------- |
| `PORT`                     | HTTP server port (default: `8080`)                                          |
| `SHUTDOWN_TIMEOUT`         | Server shutdown timeout in seconds (default: `7200`)                        |
| `PIPECAT_LOG_LEVEL`        | Pipecat logging level: `TRACE`, `DEBUG`, `INFO`, `WARNING`, `ERROR`, `NONE` |
| `PCC_LOG_FEATURES_SUMMARY` | Set to `true` to log available features on startup                          |
| `IMAGE_VERSION`            | Set automatically during build to track image versions                      |
| `ESP32_ENABLED`            | Enable ESP32 mode for SmallWebRTC                                           |
| `ESP32_HOST`               | ESP32 host address                                                          |
| `ICE_CONFIG_URL`           | ICE server configuration endpoint                                           |

For WhatsApp integration, the following environment variables are required:

| Variable                   | Description                                  |
| -------------------------- | -------------------------------------------- |
| `WHATSAPP_TOKEN`           | WhatsApp API access token                    |
| `WHATSAPP_PHONE_NUMBER_ID` | WhatsApp Business phone number ID            |
| `WHATSAPP_APP_SECRET`      | WhatsApp app secret for webhook verification |

### Logging available features

To see which features are available in the base image at startup, set `PCC_LOG_FEATURES_SUMMARY=true`. This outputs a summary like:

```
2025-10-22 22:04:41.411 | INFO     | feature_manager:log_features_summary:287 - ============================================================
2025-10-22 22:04:41.411 | INFO     | feature_manager:log_features_summary:288 - Features available in this base image:
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:289 - ============================================================
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:299 - ✅ Daily Transport: ENABLED
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:299 - ✅ Websocket Transport: ENABLED
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:299 - ✅ SmallWebRTC Session Arguments: ENABLED
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:299 - ✅ SmallWebRTC Transport: ENABLED
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:299 - ✅ SmallWebRTC ICE Candidates: ENABLED
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:299 - ✅ WhatsApp Integration: ENABLED
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:314 - ============================================================
```

### Available base images

| Name                   | Description                                               |
| ---------------------- | --------------------------------------------------------- |
| `dailyco/pipecat-base` | Multi-modal Pipecat optimized, suitable for most use-case |

The base image supports multiple Python versions. Starting with version 0.1.0, the default Python version is 3.12 (previously 3.10).

**Latest tags:**

* `dailyco/pipecat-base:latest` (Python 3.12, default)
* `dailyco/pipecat-base:latest-py3.10` (Python 3.10)
* `dailyco/pipecat-base:latest-py3.11` (Python 3.11)
* `dailyco/pipecat-base:latest-py3.12` (Python 3.12)
* `dailyco/pipecat-base:latest-py3.13` (Python 3.13)

**Versioned tags:**

* `dailyco/pipecat-base:0.1.0` (Python 3.12, default)
* `dailyco/pipecat-base:0.1.0-py3.10` (Python 3.10)
* `dailyco/pipecat-base:0.1.0-py3.11` (Python 3.11)
* `dailyco/pipecat-base:0.1.0-py3.12` (Python 3.12)
* `dailyco/pipecat-base:0.1.0-py3.13` (Python 3.13)

<Tip>For production use, we recommend pinning to specific versions.</Tip>

## Using a custom image

For more complex use-cases, you can use a custom image.

When doing so, we recommend following best practices to ensure your agent instance runs optimally on the platform.

Our [base image](https://github.com/daily-co/pipecat-cloud-images/tree/main/pipecat-base) is open source and serves as a useful blueprint for configuring your custom agent image.

### Agent image structure

<Warning>
  Custom agent images are for advanced use cases. For most teams,
  we recommend using our base images. If needed, consult the base
  image code as a reference.

  For unsupported use cases, contact us at [help@daily.co](mailto:help@daily.co) or via
  [Discord](https://discord.gg/dailyco).
</Warning>

Pipecat Cloud agent images must adhere to a specific structure to run on the platform. Our base images abstract away much of this complexity, but if you are building a custom image, you must ensure your agent adheres to the following:

* HTTP API that can handle requests from the platform to configure and run agent instances.
* The necessary system level dependencies (such as Python.)

In order to start an instance of your custom agent, you must expose a HTTP `POST /bot` route that will be called by the platform.

We recommend using FastAPI to create this route. Please refer to the base image code for an example of how to do this.

## Building the image

Pipecat Cloud requires all images to be built to target Linux on ARM. This is the most common platform for cloud deployments.

```bash theme={null}
docker build --platform linux/arm64 -t my-agent:latest .
```

Your agent image should include:

* All dependencies required for your agent to run.
* Assets (such as images or models) available in the container filesystem
* The entry point for your agent (usually a Python script)
* Additional system dependencies (if required)

### Best practices

* Keep your image as small as possible. Use multi-stage builds to reduce the size of the final image.
* Use a `.dockerignore` file to exclude unnecessary files from the image.
* Pipecat Cloud will automatically restart your agent if it crashes. Ensure your agent can handle this gracefully.
* Use [Secrets](./secrets) to securely store sensitive information in your agent image.
* To optimize for fast start-ups, avoid long running or blocking processes during initialization.


# Deployments
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/fundamentals/deploy

Serving your agent via Pipecat Cloud

Once you've built an agent image, it's time to deploy it to Pipecat Cloud.

## Deploying an agent

Deploying agents to Pipecat Cloud requires first building and pushing your image to a container registry, such as [Docker Hub](https://hub.docker.com/).

Use the Pipecat CLI to build and push your Docker image:

```bash theme={null}
pipecat cloud docker build-push
```

This command reads your `pcc-deploy.toml` configuration to automatically build, tag, and push your image to the configured registry.

Once your agent image is pushed, deploy it to Pipecat Cloud:

```bash theme={null}
pipecat cloud deploy
```

Agents can be deployed to a specific region, which can be used to reduce latency and meet data residency requirements. You can deploy to a specific region by using the `--region` flag. This command will instruct Pipecat Cloud to create a new deployment from your image. Agents deploy to `us-west` by default, but you can specify a different region:

```bash theme={null}
pipecat cloud deploy my-agent my-image:0.1 --region us-east
```

<Info>
  Agent names are globally unique across all regions. For multi-region
  deployments, deploy uniquely named agents to each target region (e.g.,
  `my-agent-us-west`, `my-agent-us-east`).
</Info>

Once the deployment completes, you will be able to run your agent on-demand.

For a complete list of deployment configuration options, see the [CLI deploy reference](/cli/cloud/deploy).

<Note>
  Using a private container registry? See our [Container Registry
  guides](../guides/container-registries/overview) for setup instructions for
  GCP Artifact Registry, AWS ECR, and other private registries.
</Note>

You will be asked to confirm your deployment configuration before proceeding.

### Deployment status

Once complete, Pipecat Cloud will return the status of your deployment or any errors that occurred during the deployment process.

### Using pcc-deploy.toml

To support shareable deployment configurations, you can create a `pcc-deploy.toml` file in the root of your project (adjacent to where you run your CLI commands.)

The CLI will automatically detect this file and use it to populate the deployment configuration. Any CLI arguments you provide will override the values in the `pcc-deploy.toml` file.

An example `pcc-deploy.toml` file might look like this:

```toml theme={null}
agent_name = "my-first-agent"
image = "your-docker-repository/my-first-agent:0.1"
image_credentials = "dockerhub-access"
secret_set = "my-agent-secrets"
agent_profile = "agent-1x"
region = "us-west"

[scaling]
    min_agents = 1
```

<Tip>
  For a complete reference of all configuration options, see the [deploy command
  documentation](/cli/cloud/deploy#configuration-file-pcc-deploytoml).
</Tip>

### Agent Profiles

Agent profiles determine the CPU and memory resources allocated to your deployed agents. Choose the profile that best matches your agent's requirements:

* **`agent-1x`** (default): 0.5 vCPU and 1 GB of memory. Best for voice agents.
* **`agent-2x`**: 1 vCPU and 2 GB of memory. Well suited for voice and video agents or voice agents requiring extra processing.
* **`agent-3x`**: 1.5 vCPU and 3 GB of memory. Best for voice and video agents requiring extra processing or multiple video inputs.

You can specify the agent profile in your deployment:

**Via CLI:**

```bash theme={null}
pipecat cloud deploy my-agent my-image:latest --profile agent-2x
```

**Via pcc-deploy.toml:**

```toml theme={null}
agent_profile = "agent-2x"
```

<Note>
  Agent profiles affect resource allocation and impact your usage costs. Choose
  the smallest profile that meets your agent's performance requirements.
</Note>

### Checking status of a deployment

You can check the status of a deployment using the CLI:

```bash theme={null}
pipecat cloud agent status my-first-agent
```

If a deployment was successful and ready to be started, you should see a `ready` status.

## Modifying a deployment

Agents are referenced as a mutable manifest in Pipecat Cloud.
Pushing a new image or configuration to the same agent name will update your agent manifest and be used for any new requests.

```bash theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:latest --min-agents 2
```

This command will update the `my-first-agent` deployment to use the new image and set the minimum number of instances to 2.

To avoid interruption of any active sessions, any running agent instances will continue on the old image until the session concludes. Once those sessions end, the instances running the old image are discarded and will not be returned to the pool.

<Note>
  When there is no new deployment, instances are returned to the pool after a session ends and remain available to serve new sessions during a cooldown period. A new deployment changes this behavior: instances running the prior image are terminated once their sessions conclude, rather than being recycled.
</Note>

Idle agent instances in your agent pool will be replaced with the new configuration according to the Pipecat Cloud auto-scaling strategy.
New agent requests may, therefore, start with prior deployment configuration if updates are not fully propagated. This ensures on-demand availability
remains consistent and avoids potential cold starts.

### Failed deployments

If a deployment fails (i.e. fails to enter a `ready` state), requests will be routed to any prior deployments in a ready state.

This ensures that your agent remains available to end-users even when a deployment fails.

## Delete a deployment

<Warning>
  This action is irreversible and will remove all data related to the agent from
  Pipecat Cloud.
</Warning>

If you no longer need a deployment, you can delete it using the `delete` command:

```bash theme={null}
pipecat cloud agent delete my-first-agent
```

Any running agent instances will continue to run until they conclude or are stopped.


# Troubleshooting
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/fundamentals/error-codes

Common issues and solutions when deploying agents to Pipecat Cloud

## Error codes

### PCC-1002

> Attempt to start agent without public api key.

Ensure you have provided a valid public API key when starting an agent. [Learn more](./accounts-and-organizations#api-keys) about creating and managing API keys.

### PCC-1004

> Billing credentials not set. Please set billing credentials via the Pipecat Cloud dashboard.

In order to deploy or start an agent on Pipecat Cloud, you must have valid billing credentials set up on the namespace or organization you have specified. You can set up these up via the [Pipecat Cloud dashboard](https://pipecat.daily.co/).

### PCC\_IMAGE\_NOT\_FOUND

> The provided image address or tag could not be found.

Check your provided image, tag and host address to ensure they are correct.

### PCC\_IMAGE\_PULL\_UNAUTHORIZED

> Unauthorized to pull image at given address.

If you are using a private repository, ensure you have provided valid credentials to access the repository via [image pull secrets](./secrets#image-pull-secrets).

### PCC\_IMAGE\_TOO\_LARGE

> Image is too large.

Ensure the image you are attempting to pull is not too large. Currently, Pipecat Cloud has no maximum image size limit, but excessively large images may cause issues when deploying agents.

### PCC\_IMAGE\_PULL\_RATE\_LIMIT

> Remote repository rate limited the pull request.

Check that your image repository is not rate limiting your pull requests. This can be caused by excessive requests to the repository, or by the repository owner setting rate limits.

### PCC\_INVALID\_IMAGE\_PLATFORM

> Image platform is not supported.

Ensure the image you are attempting to pull is supported by Pipecat Cloud. Currently, Pipecat Cloud only supports images for the `linux/arm64` platform.

### PCC\_IMAGE\_PULL\_UNKNOWN\_ERROR

> Unknown error occurred when attempting to pull the image.

If you receive this error, please contact support for further assistance.

### PCC\_UNKNOWN\_ERROR

> Unknown error occurred when attempting to start provided image.

Indicates an error in your image that prevent your agent from starting. Typically `pipecat cloud agent logs [AGENT_NAME]` will provide more information on the error.

### PCC-AGENT-AT-CAPACITY

> Service maximum agent instances reached

This error indicates that the maximum active sessions for your deployment have been reached. You can either increase the maximum active sessions for your deployment (`max-agents`), or place the end-user in a queue until a session becomes available.

## Enabling extra logs

If you wish to print a summary of the features available in the base image, set the environment variable `PCC_LOG_FEATURES_SUMMARY` to `true`.
This will print all the features enabled in the base image that you can use. For example:

```
2025-10-22 22:04:41.411 | INFO     | feature_manager:log_features_summary:287 - ============================================================
2025-10-22 22:04:41.411 | INFO     | feature_manager:log_features_summary:288 - Features available in this base image:
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:289 - ============================================================
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:299 - ✅ Daily Transport: ENABLED
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:299 - ✅ Websocket Transport: ENABLED
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:299 - ✅ SmallWebRTC Session Arguments: ENABLED
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:299 - ✅ SmallWebRTC Transport: ENABLED
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:299 - ✅ SmallWebRTC ICE Candidates: ENABLED
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:299 - ✅ WhatsApp Integration: ENABLED
2025-10-22 22:04:41.412 | INFO     | feature_manager:log_features_summary:314 - ============================================================
```


# Logging and Observability
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/fundamentals/logging

Obtaining logs and metrics from your agents and sessions

<Warning>
  This section of the documentation is currently work in progress. Please check
  back soon for updates.
</Warning>

## Configuring log level

You can control the Pipecat logging level for your deployed agents using the `PIPECAT_LOG_LEVEL` environment variable. This can be set as a [secret](/deployment/pipecat-cloud/fundamentals/secrets) or directly in your deployment configuration.

Available log levels:

| Level     | Description                                                          |
| --------- | -------------------------------------------------------------------- |
| `TRACE`   | The most detailed information for debugging (includes Frame logging) |
| `DEBUG`   | Verbose output for debugging (default)                               |
| `INFO`    | General operational information                                      |
| `WARNING` | Warning messages for potential issues                                |
| `ERROR`   | Error messages only                                                  |

## Agent logs

Agent logs are available via both the CLI and Dashboard. You can view logs for a specific agent by running the following command:

```bash theme={null}
pipecat cloud agent logs my-agent
```

This command accept various filters to help you narrow down the logs you are looking for. For example, you can filter logs by severity level:

```bash theme={null}
pipecat cloud agent logs my-agent -l ERROR
```

### Session logging

We recommend using the `loguru` library for logging within your agent. This will ensure any logging within your agent associated to the session it is running in.

```python theme={null}
from loguru import logger

async def bot():
	logger.info("Hello, world!") # will be associated with the session id
```

If you are handling logging manually, you can obtain the active session ID from the `PipecatRunnerArguments` object (or subclass alternative) passed to your `bot()` method:

```python theme={null}
from pipecat.runner.types import PipecatRunnerArguments

async def bot(args: PipecatRunnerArguments):
	session_id = args.session_id
```

<Note>
  See [the Session Arguments reference](../sdk-reference/session-arguments) for
  more additional SessionArgument types.
</Note>

## CPU and memory metrics

Pipecat Cloud tracks CPU and memory usage for each session, which can be helpful for troubleshooting performance issues. You can view these metrics in two ways:

### Dashboard

Navigate to your agent in the Pipecat Cloud dashboard, then go to **Sessions** and click on a specific **Session ID** to view CPU and memory usage graphs.

### CLI

Use the `sessions` command with a specific session ID to see CPU and memory usage with sparkline visualizations and percentile summaries:

```bash theme={null}
pipecat cloud agent sessions my-agent --id <session-id>
```

See the [CLI reference](/cli/cloud/agent#sessions) for more details.


# Scaling
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/fundamentals/scaling

Production-ready agent deployment and scaling strategies

Agentic applications demand near-instant agent availability to maintain user engagement and responsiveness. Pipecat Cloud manages the complexity of scaling these agent deployments in production environments, providing granular controls for compute resources and cost optimization.

## Core concepts

### Instances

An instance represents a single unit of compute that runs your agent.

Instance costs are determined by:

* [Active session](./active-sessions) runtime duration
* Warm instance maintenance time
* The compute profile specified in your deployment

Pipecat Cloud automatically provisions and manages agent instances to handle active sessions, ensuring that your deployment can scale to meet demand
within the limits of your deployment configuration.

### Instance pool

Making a deployment to Pipecat Cloud creates a managed pool of agent instances that:

* Routes requests to available agent instances
* Scales based on demand within configured limits
* Maintains optimal performance through auto-scaling

<Frame>
  <img />
</Frame>

Developers can configure the upper and lower limit of a deployment's instance pool, providing a cost-effective way to handle varying loads.

### Minimum agents

`min-agents`

* Maintains specified number of warm agent instances to serve incoming requests
* Immediately ready to become active, reducing cold starts
* Defaults to `0` if unspecified

<Frame>
  <img />
</Frame>

Developers specify a `min-agents` configuration to determines the number of agent instances that should be kept warm in their deployment pool. A warm instance is kept running and can immediately be used to serve an active session.

Maintaining a minimum number of agent instances is important to keep agent start times fast and reduce [cold starts](./scaling#cold-starts).

```shell theme={null}
# Defaults to zero
pipecat cloud deploy [agent-name]

# Maintain 1 warm instance at all times
pipecat cloud deploy [agent-name] --min-agents 1
```

<Warning>
  Setting `--min-agents` to 1 or greater will incur charges even when the agent
  is not in use.
</Warning>

### Maximum agents

`max-agents`

* Sets hard limit on concurrent sessions
* Acts as a cost control / load mechanism
* Returns HTTP 429 when pool capacity is reached

<Frame>
  <img />
</Frame>

<Note>
  Each deployment made to Pipecat Cloud has a maximum allowed pool
  size of 50. Please contact us at [help@daily.co](mailto:help@daily.co) or via
  [Discord](https://discord.gg/dailyco) if you require more capacity.
</Note>

Deployments can optionally be made with a `max-agents` configuration that limits the number of agent instances that your pool can contain.
This exists as a cost control measure, allowing developers to limit the total number of active sessions that can be run at any one time.

The maximum instance count is a hard limit, meaning requests made to a pool that is at capacity will receive a `429` response. See [starting sessions](./active-sessions#agent-capacity) for more information for how to handle this in your application code.

```shell theme={null}
# Limit pool to 25 active sessions running concurrently
pipecat cloud deploy [agent-name] --max-agents 25
```

### Agent lifecycle

<Steps>
  <Step title="Pool Initialization">
    * Provisions the minimum number of warm agent instances based on `min-agents` configuration (defaults to `0`)
    * Listens for session requests to route to available agent instances
  </Step>

  <Step title="Session Assignment">
    * ✅ If a warm instance is available, the session will be assigned to that instance
    * ⏳ If no warm agent instances are available, and your pool is not at capacity, a new instance will be provisioned to handle the request (e.g a cold-start)
    * ❌ If your pool is at capacity, your application will receive a `429` response from the start request
  </Step>

  <Step title="Auto-scaling">
    * The Pipecat Cloud auto-scaler determines if additional warm agent instances should be created to support further requests.
    * Once a session concludes, the instance is returned to the pool and can immediately serve another session. However, if a new deployment has been pushed since the instance was created, the instance is discarded instead and will not be reused.
  </Step>
</Steps>

<Note>
  <b>Your are billed for warm agent instances</b>, even if they are not handling
  active sessions. Developers should consider their deployment strategy when
  cost optimizing, adjusting the minimum and maximum instance count accordingly.
  See [current pricing](https://www.daily.co/pricing/pipecat-cloud) for details.
</Note>

## Cold-starts

A cold start may occur when an active session request is made and no warm agent instances are available in the pool to handle it. In this case, Pipecat Cloud will provision a new instance to handle the request.

Cold starts require additional time to provision the instance and load the agent instances, which may result in a delay for the user. To minimize cold starts, you can configure your pool to maintain a minimum number of warm agent instances at all times.

<Frame>
  <img />
</Frame>

Pipecat Cloud aims to mitigate cold starts as much as possible through [auto-scaling](#auto-scaling).

<AccordionGroup>
  <Accordion title="How long does a cold start take?">
    The exact duration of a cold start depends on multiple factors, such as the size of your agent image and the complexity of your agent. In general, cold starts take around 10 seconds. Once you have a running instance of a deployed image, new sessions start immediately with no cold start delay.

    For developers, especially during test and development, using scale-to-zero with cold starts is a good place to start as it saves money. Once you understand your utilization pattern, you can build a strategy around optimizing start times by maintaining warm agent instances.
  </Accordion>

  <Accordion title="Mitigation strategies">
    To avoid cold starts, you can:

    * Adjust the number of warm agent instances (`min-agents`) in your pool to ensure that there are always agent instances available to handle requests.
    * Adjust your maximum instance count (`max-agents`) and issue capacity notifications in your application.
  </Accordion>
</AccordionGroup>

### Scale-to-zero

For some deployments, using a minimum instance count of 0 is preferable (e.g. while in development.) Since you are only charged for warm agents instances and active sessions, this can be a cost-effective way to manage deployments where fast start times are not required.

When the minimum instance count is set to 0, the pool will scale down to 0 agent instances when there are no active sessions. Idle agent instances are maintained for 5 minutes before being terminated.

<Tip>
  Scale-to-zero is not recommended for production deployments where immediate
  response is required.
</Tip>

## Auto-scaling

Pipecat Cloud performs auto-scaling by default on all deployments. Auto-scaling is accomplished through the following mechanisms:

* Scaling up based on request velocity
* Maintaining efficiency within max-agents limit
* Scaling down to min-agents (or zero) during low usage
* Supporting burst workloads automatically

### Auto-Scaling Buffer

<Note>
  <b>Pipecat Cloud maintains a free auto-scaling buffer</b> in addition to your
  paid reserved agent instances. This saves you from over-provisioning warm
  agent instances while still ensuring fast response times during traffic
  increases.
</Note>

When your traffic increases and you have active sessions running, our system automatically:

* Proactively provisions additional idle agent instances based on your current usage patterns
* Provides these buffer agent instances at no additional cost to you
* Ensures you can continue handling traffic spikes even when all your paid warm agents are in use

For example, with multiple active sessions:

* The system is already spinning up additional buffer agent instances in the background
* These buffer agent instances become available within \~10 seconds
* You can continue calling the `/start` endpoint without worrying about configuring additional capacity

This approach means:

* You can set a lower `min-agents` value than your peak traffic requirements
* You'll still avoid cold starts in most scenarios
* You get better cost efficiency without sacrificing performance

The only scenario where you need to consider higher `min-agents` values is for extremely rapid traffic spikes (tens or hundreds of calls per second) where the buffer can't be provisioned fast enough.

### Scaling Philosophy

Our scaling system is designed to minimize the need for manual capacity planning. Here's how we recommend thinking about scaling:

* **Start simple:** We encourage you to set `min-agents` to 0 initially and test how the system performs for your specific use case. Many applications work well without any pre-warmed agent instances.

* **Optimize as needed:** We work hard to make cold starts rare and as fast as possible so that, for many applications, you don't have to worry about warm instances at all.

* **Tune for traffic patterns:** If you have spiky workloads with bursty traffic patterns, setting an appropriate `min-agents` value can help prevent cold starts during critical periods. Consider scheduling higher `min-agents` values only during your peak usage hours.

Our goal is to make scaling decisions as simple as possible while giving you the controls you need for optimization when required.

## Updating scaling configuration

You can update your deployment's configuration at any time via the CLI or Pipecat Cloud Dashboard.

```shell theme={null}
pipecat cloud deploy [agent-name] [image] --min-agents 1 --max-agents 5
```

Please note that changing your scaling parameters will not disrupt any active sessions.
If you reduce your max instance count below the number of currently active sessions, you will still be billed for the duration of those sessions.

## Capacity Planning

Effective capacity planning is crucial for production deployments to ensure your agents respond immediately.

Pipecat Cloud auto-scales your agents. For most cases, the only action you need to take is to set the `--min-agents` parameter to 1. However, if your application experiences fluctuations in traffic, you may need to plan for additional warm capacity to ensure your agents are always ready to respond immediately.

<Card title="Capacity Planning Guide" icon="calculator" href="../guides/capacity-planning">
  See our guide for calculating reserved agents, understanding warm capacity,
  and implementing scaling strategies for production.
</Card>

## Usage summary

Pipecat Cloud bills based on:

* **Active session minutes**: Time your agents spend handling live sessions
* **Reserved session minutes**: Time your warm agent instances are kept running, even when idle

An active session starts when you call the `/start` endpoint (or CLI or SDK equivalent) and ends when your agent's pipeline shuts down.

Reserved session minutes are optional and controlled by setting `--min-agents` in your deployment configuration.

<Note>
  Both active and reserved session time is measured to the second and billed in
  minutes.
</Note>


# Secrets
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/fundamentals/secrets

Providing your agent with sensitive information

Most agents will require access to sensitive information, such as API keys, passwords, and other secrets.
To keep this information secure, we recommend using Pipecat Cloud's secret management feature.

Secrets are created as "sets" of key-value pairs, and defined at the user / organization level.
This means that secrets can be shared across all agent deployments within the same user workspace or organization.

To access secrets, your deployment must specify the secret set to use.

## Working with secrets

Secrets can be managed via either the CLI or the [Dashboard <Icon icon="arrow-up-right-from-square" />](https://pipecat.daily.co).

### Creating a secret set and adding secrets

```bash theme={null}
pipecat cloud secrets set my-secrets SECRET_NAME secret-value SECRET_NAME_2 secret-value-2
```

This command will create or modify the secret set with the name `my-secrets`, and add or update the key-value pairs `SECRET_NAME` and `SECRET_NAME_2`.

You can add additional secrets to the set by specifying more key-value pairs.

```bash theme={null}
pipecat cloud secrets set my-secrets SECRET_NAME_3 secret-value-3
```

<Info>
  Whenever a secret is added or updated in an existing set, any deployments
  using that set will need to be redeployed to access the new values.
</Info>

#### Regions

Secret sets are created in a specific region. By default, secrets are created in `us-west` if no region is specified.

```bash theme={null}
pipecat cloud secrets set my-secrets SECRET_NAME secret-value --region us-east
```

<Warning>
  Secret set names are globally unique, but each secret set exists in only one
  region. Secrets must be in the same region as the agents that use them. For
  multi-region deployments, create separate secret sets for each region (e.g.,
  `my-secrets-us-west`, `my-secrets-us-east`).
</Warning>

#### Special characters

There are several ways to specify key-value pairs:

1. **Simple values** (no spaces or special characters):

   ```bash theme={null}
   pipecat cloud secrets set my-secrets KEY1=simple KEY2=value
   ```

2. **Values with spaces**:

   ```bash theme={null}
   pipecat cloud secrets set my-secrets KEY1="value with spaces"
   # or
   pipecat cloud secrets set my-secrets "KEY1=value with spaces"
   ```

3. **Values containing equals signs**:

   ```bash theme={null}
   pipecat cloud secrets set my-secrets KEY1="value=with=equals"
   # or
   pipecat cloud secrets set my-secrets KEY1==value=with=equals
   ```

4. **Values containing quotes**:

   ```bash theme={null}
   pipecat cloud secrets set my-secrets KEY1="value with \"quotes\""
   # or
   pipecat cloud secrets set my-secrets 'KEY1=value with "quotes"'
   ```

5. **Values containing backslashes**:
   ```bash theme={null}
   pipecat cloud secrets set my-secrets 'KEY1=value with \backslashes'
   # or
   pipecat cloud secrets set my-secrets "KEY1=value with \\backslashes"
   ```

#### Important Notes

* Keys must contain only alphanumeric characters, underscores, and hyphens (`[a-zA-Z0-9_-]`)
* Keys must not exceed 64 characters in length
* Values are preserved exactly as entered, including any spaces, quotes, or special characters
* If a value contains spaces, you must either:
  * Enclose the value in quotes: `KEY="value with spaces"`
  * Enclose the entire key-value pair in quotes: `"KEY=value with spaces"`
* When using quotes within values, you can either:
  * Use single quotes around the pair containing double quotes: `'KEY=value "quoted" here'`
  * Use escaped double quotes: `KEY="value \"quoted\" here"`

### List secret sets

You can view a list of available secret sets in your currently selected workspace or organization.

```bash theme={null}
pipecat cloud secrets list
```

You can pass an optional secret set name to view the secret keys within that set. Values are not displayed.

```bash theme={null}
pipecat cloud secrets list my-secrets
```

### Removing a secret

To remove a secret from a set, use the `unset` command.

```bash theme={null}
pipecat cloud secrets unset my-secrets SECRET_NAME
```

To remove a secret-set entirely, use the `delete` command:

```bash theme={null}
pipecat cloud secrets delete my-secrets
```

### Image pull secrets

Image pull secrets are used to authenticate with private Docker registries when deploying agents.

They can be passed as part of the `deploy` command.

```bash theme={null}
pipecat cloud secrets image-pull-secret my-image-pull-secret https://index.docker.io/v1/
```

Running this command will prompt you for account credentials.
You can optionally encode your credentials in base64 with the `--base64encode` flag.

Like regular secret sets, image pull secrets are region-specific. Specify the region with the `--region` flag:

```bash theme={null}
pipecat cloud secrets image-pull-secret my-image-pull-secret https://index.docker.io/v1/ --region us-east
```

<Warning>
  Unlike secret sets, image pull secrets can not be updated. You must delete and
  recreate the secret if you need to change the credentials. Image pull secrets
  must also be in the same region as the agents that use them.
</Warning>

#### Using Private Docker Hub Images with Pipecat Cloud

If you want to use private Docker Hub images with your Pipecat Cloud deployment, follow these steps:

**Step 1: Create a Docker Hub Access Token**

1. Log in to [Docker Hub](https://hub.docker.com)
2. Go to Account Settings → Personal access tokens
3. Click "Generate new token"
4. Name it (e.g., "pcc-images") and set permissions (e.g. "Read & Write")
5. Copy and save the generated token securely

**Step 2: Make Your Docker Hub Image Private**

1. Go to your repository on Docker Hub
2. Click "Settings"
3. Under "Visibility," change from "Public" to "Private"
4. Save changes

**Step 3: Set Up Image Pull Secret in Pipecat Cloud**

```bash theme={null}
# Create image pull secret
pipecat cloud secrets image-pull-secret my-image-pull-secret https://index.docker.io/v1/
# Enter your Docker Hub username when prompted
# Enter your access token (not password) when prompted
```

<Info>
  `dockerhub-access` is the name of your image pull secret, so name it whatever
  you'd like.
</Info>

**Step 4: Deploy Your Agent with the Image Pull Secret**

Update your `pcc-deploy.toml` file:

```toml theme={null}
agent_name = "my-first-agent"
image = "your-docker-repository/my-first-agent:0.1"
image_credentials = "my-image-pull-secret"
secret_set = "my-secrets"

[scaling]
	min_agents = 0
	max_agents = 5
```

Finally, deploy your agent with the `pcc-deploy.toml` configuration:

```bash theme={null}
pipecat cloud deploy
```

## Accessing secrets in your agent code

Secrets are mounted as environment variables in your agent process.

For example, if you define a secret with the key MY\_SECRET, you can access it in your agent code like so:

```python theme={null}
import os

secret_value = os.environ.get('MY_SECRET')
```

When deploying, you have two options to specify which secret set to use:

1. Specify a secret set as part of your deploy command:

```bash theme={null}
pipecat cloud deploy my-first-agent your-docker-repository/my-first-agent:0.1 --secrets my-secrets
```

2. Specify a `secret_set` key in your `pcc-deploy.toml` file:

```toml theme={null}
agent_name = "my-first-agent"
image = "your-docker-repository/my-first-agent:0.1"
secret_set = "my-secrets"
```

With either approach, the CLI will automatically inject the secrets from the specified set into your agent deployment as environment variables.

## Other methods

If you prefer to manage secrets outside of Pipecat Cloud, you can use environment variables or other secret management tools.

You could, for example, set environment variables in your Dockerfile:

```dockerfile theme={null}
FROM dailyco/pipecat-base:latest

# Enable bytecode compilation
ENV UV_COMPILE_BYTECODE=1

# Copy from the cache instead of linking since it's a mounted volume
ENV UV_LINK_MODE=copy

# Set the secret as an environment variable
ENV MY_SECRET=secret-value

# Install the project's dependencies using the lockfile and settings
RUN --mount=type=cache,target=/root/.cache/uv \
    --mount=type=bind,source=uv.lock,target=uv.lock \
    --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
    uv sync --locked --no-install-project --no-dev

# Copy the application code
COPY ./bot.py bot.py
```

We recommend using Pipecat Cloud's built-in management for the most secure and versatile method of managing secrets.


# Capacity Planning
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/capacity-planning

Optimize your agent deployments to minimize cold starts

## Overview

By default, Pipecat Cloud auto-scales your agent to help minimize the likelihood that your agents experience a cold start. For many applications, you can simply set the `--min-agents` parameter to 1 in order to avoid scaling from zero and let Pipecat Cloud handle the rest.

However, for applications where traffic can fluctuate, you may need to plan for additional warm capacity to ensure your agents are always ready to respond immediately. For those cases, this guide will help you understand how warm capacity works in Pipecat Cloud, when you need to plan to use reserves, and how you can optimize your plan for both performance and cost.

## Agent Types

Before discussing capacity planning, it's important to understand the different types of agent instances in Pipecat Cloud:

* **Active Agents**: Agent instances currently running and handling user sessions.
* **Idle Agents**: When you start an active agent, Pipecat Cloud automatically provisions an additional idle agent to help with scaling. These take approximately 30 seconds to become available, though this time may vary based on system load and image size. You are not charged for these idle agents.
* **Reserved Agents**: Agent instances maintained according to your `--min-agents` deployment setting, ensuring immediate availability regardless of current traffic.

## Understanding Warm Capacity

Your "warm capacity" represents agent instances that are immediately available to handle new sessions without a cold start. Pipecat Cloud automatically manages this based on:

1. Your configured **reserved agents** (`min-agents`)
2. Your current **active sessions**
3. Automatically provisioned **idle agents**

The system ensures you always have the following warm capacity available:

* When **Reserved > Active**: Your warm capacity equals the number of **Reserved Agents**
* When **Active ≥ Reserved**: Your warm capacity equals the number of **Active Agents** (through automatically provisioned idle agents)

To illustrate the point, here are a few scenarios showing how warm capacity works:

| Reserved | Active | Warm Capacity |
| -------- | ------ | ------------- |
| 10       | 1      | 10            |
| 10       | 10     | 10            |
| 1        | 10     | 10            |

This shows how:

* Reserved agents provide a guaranteed minimum warm capacity
* As active sessions increase beyond your reserved count, your warm capacity grows to match through automatically provisioned idle agents

## Agent Cooldown

When an active session ends, the agent instance behavior is governed by a cooldown period:

* The agent instance remains available in your warm capacity pool for a 5-minute cooldown period
* During this time, it can immediately serve another request without any cold start
* After the 5-minute cooldown expires, if the agent instance hasn't been used, it will be terminated
* This cooldown provides a buffer in your warm capacity pool, helping to smooth transitions between traffic peaks

## Planning for Traffic Patterns

To determine the optimal reserved instance count for your deployment, consider:

1. **Peak Concurrent Sessions**: How many simultaneous sessions do you expect during peak periods?

2. **Growth Rate**: How quickly do new sessions start during peak periods?

   * If new sessions start faster than the \~30 second warm-up time for auto-scaled agent instances, you'll need more reserved capacity

3. **Cold Start Tolerance**: How important is immediate response for your use case?

<Tip>
  Completely avoiding cold starts may not be practical for all applications. We
  strongly recommend considering longer or variable start up times when building
  your application. For phone use cases, consider a hold message or for web
  apps, consider a waiting UX or message.
</Tip>

## Cost-Efficient Scaling Strategies

* **Development/Testing**: Use `min-agents: 0` to minimize costs during development
* **Production Voice AI**: Set `min-agents` to cover your baseline traffic to avoid cold starts
* **Time-Based Scaling**: Consider modifying your reserved count for known high-traffic periods
* **Monitoring**: Regularly review your warm capacity utilization to optimize your configuration

<Info>A cold start typically takes around 10 seconds.</Info>

## Calculating Optimal Reserved Agents

For production deployments where immediate response is essential, you can calculate your optimal reserved agent count using a growth rate approach:

```
Optimal Reserved = MAX(Baseline Sessions, CPS × Idle Creation Delay)
```

Where:

* **Baseline Sessions**: Minimum concurrent sessions you typically maintain
* **CPS (Calls Per Second)**: Your expected session growth rate during peak periods
* **Idle Creation Delay**: Time for new idle agents to become available (\~30 seconds)

This formula addresses the fundamental challenge: "Can our warm capacity creation keep pace with our call growth rate?" By reserving capacity based on your growth rate and the idle creation delay, you ensure sufficient capacity is available during the critical period before auto-scaling can catch up.

### Growth Rate Examples

Here's how the formula works with different growth rates:

| Scenario      | Baseline | CPS                | Idle Creation Delay | Calculation       | Optimal Reserved |
| ------------- | -------- | ------------------ | ------------------- | ----------------- | ---------------- |
| High volume   | 10       | 1.0 (1 call/sec)   | 30s                 | MAX(10, 1.0 × 30) | 30               |
| Medium volume | 10       | 0.5 (1 call/2sec)  | 30s                 | MAX(10, 0.5 × 30) | 15               |
| Low volume    | 10       | 0.1 (1 call/10sec) | 30s                 | MAX(10, 0.1 × 30) | 10               |

### Call Center Example

Consider a voice AI call center that:

* Normally handles 10 concurrent calls (baseline)
* During promotions, receives new calls at a rate of 1 call per second
* Has a 30-second idle creation delay

Applying our formula:

```
Optimal Reserved = MAX(10, 1 × 30)
                 = MAX(10, 30)
                 = 30
```

With 30 reserved agents:

* You can handle the growth rate of 1 call per second for the full 30 seconds until new idle agents start becoming available
* This prevents any cold starts during the critical catch-up period
* Auto-scaling will create additional idle agents to handle continued growth

### Understanding the Growth Rate Approach

This approach works because:

1. When your call volume starts increasing, you immediately begin consuming your warm capacity
2. At the same time, each new active call triggers the creation of a new idle agent
3. However, these new idle agents take \~30 seconds to become available
4. Your reserved capacity must be sufficient to handle all calls during this 30-second "catch-up period"

### Trading Cost for Performance

The formula provides a starting point, which you can adjust based on your specific needs:

* **Cost-sensitive**: Use a lower reserved count and accept some cold starts during traffic spikes
* **Performance-sensitive**: Use the calculated reserved count to ensure zero cold starts
* **Hybrid approach**: Monitor your actual traffic patterns and adjust based on real-world performance

<Tip>
  For most production voice AI applications, we recommend using at least the
  calculated optimal reserved agent count during business hours or peak usage
  periods, and then scaling down during off-hours to optimize costs.
</Tip>

## Summary and Next Steps

Proper capacity planning ensures your agents are always ready to respond immediately, providing the best possible user experience while optimizing costs:

1. **Understand your traffic patterns**: Baseline sessions, peak sessions, and growth rate
2. **Calculate your optimal reserved agents**: Use the formula to determine the right level of warm capacity
3. **Monitor and adjust**: Refine your capacity planning based on real-world performance and costs

<CardGroup>
  <Card title="Explore Scaling Options" icon="server" href="../fundamentals/scaling">
    Learn more about scaling configuration options and deployment strategies.
  </Card>

  <Card title="Deploy Your Agent" icon="rocket" href="../fundamentals/deploy">
    Apply your capacity planning knowledge to your agent deployment.
  </Card>
</CardGroup>


# CI with GitHub Actions
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/ci-with-github-actions

Automate image builds and deploys for continuous integration

Automate your Pipecat Cloud deployment using the official [Deploy to Pipecat Cloud](https://github.com/marketplace/actions/deploy-to-pipecat-cloud) GitHub Action. This guide shows you how to build and deploy your agent whenever you push changes to your main branch.

## Prerequisites

Before setting up your workflow, ensure you have:

* A Pipecat Cloud account with a **Private** API key
* A Docker Hub account (or another container registry)
* An image pull secret configured in Pipecat Cloud
* A secret set configured for your agent

## Configure GitHub Secrets

Add the following secrets to your repository (Settings → Secrets and variables → Actions):

* `DOCKERHUB_USERNAME`: Your Docker Hub username
* `DOCKERHUB_TOKEN`: Your Docker Hub access token
* `PCC_API_KEY`: Your Pipecat Cloud **Private** API key

<Note>
  For Docker Hub, use a [personal access
  token](https://docs.docker.com/docker-hub/access-tokens/) rather than your
  password.
</Note>

## Basic Workflow Configuration

Create `.github/workflows/deploy.yml` in your repository:

```yml theme={null}
name: Deploy to Pipecat Cloud

on:
  push:
    branches: ["main"]

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Build and Deploy to Pipecat Cloud
        uses: daily-co/pipecat-cloud-deploy-action@v1
        with:
          api-key: ${{ secrets.PCC_API_KEY }}
          agent-name: ${{ github.event.repository.name }}
          build: true
          image: ${{ secrets.DOCKERHUB_USERNAME }}/${{ github.event.repository.name }}
          registry-username: ${{ secrets.DOCKERHUB_USERNAME }}
          registry-password: ${{ secrets.DOCKERHUB_TOKEN }}
          secret-set: my-secret-set
          image-credentials: my-image-pull-secret
          min-agents: 0
          max-agents: 10
```

This workflow will:

1. Trigger on every push to the `main` branch
2. Build your Docker image for `linux/arm64` (the action sets this automatically)
3. Push the image to Docker Hub with a tag defaulting to the commit SHA
4. Deploy the new image to Pipecat Cloud and wait for readiness

<Warning>
  Avoid reusing the same image tag for multiple builds. The action defaults to tagging with `${{ github.sha }}` so each deploy uses a unique tag. If you override `tag`, use a value that changes per build (e.g. date or SHA) so Pipecat Cloud does not serve cached, outdated images.
</Warning>

## Image Tagging

By default the action tags the image with the git commit SHA. To use a custom tag:

```yml theme={null}
- name: Build and Deploy to Pipecat Cloud
  uses: daily-co/pipecat-cloud-deploy-action@v1
  with:
    api-key: ${{ secrets.PCC_API_KEY }}
    agent-name: ${{ github.event.repository.name }}
    build: true
    image: ${{ secrets.DOCKERHUB_USERNAME }}/${{ github.event.repository.name }}
    tag: ${{ github.ref_name }}-${{ github.sha }}
    registry-username: ${{ secrets.DOCKERHUB_USERNAME }}
    registry-password: ${{ secrets.DOCKERHUB_TOKEN }}
    secret-set: my-secret-set
```

## Monorepo Configuration

If your agent lives in a subdirectory (e.g. `server/`, as with Pipecat CLI–generated code), set the build context and optionally restrict the workflow to that path:

```yml theme={null}
name: Deploy to Pipecat Cloud

on:
  push:
    branches: ["main"]
    paths:
      - "server/**"

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Build and Deploy to Pipecat Cloud
        uses: daily-co/pipecat-cloud-deploy-action@v1
        with:
          api-key: ${{ secrets.PCC_API_KEY }}
          agent-name: ${{ github.event.repository.name }}
          build: true
          image: ${{ secrets.DOCKERHUB_USERNAME }}/${{ github.event.repository.name }}
          docker-context: ./server
          dockerfile: ./server/Dockerfile
          registry-username: ${{ secrets.DOCKERHUB_USERNAME }}
          registry-password: ${{ secrets.DOCKERHUB_TOKEN }}
          secret-set: my-secret-set
          image-credentials: my-image-pull-secret
```

The `paths` filter ensures the workflow runs only when files under `server/` change.

## Platform and Build Behavior

Pipecat Cloud requires images built for `linux/arm64`. When `build` is enabled, the action automatically passes `--platform linux/arm64` to Docker and sets up QEMU emulation on x64 runners, so no extra steps are needed.

## Customizing Deployment

Configure the deployment via the action inputs:

* **image-credentials**: Name of the image pull secret set in Pipecat Cloud (for private registries)
* **secret-set**: Name of the secret set used for runtime environment variables
* **region**: Deployment region (uses your organization default if omitted)
* **min-agents**: Minimum agents to keep warm (0–50)
* **max-agents**: Maximum concurrent agents (1–50)
* **agent-profile**: Agent profile name, if used
* **wait-for-ready**: Whether to poll until the deployment is ready (default: true)
* **wait-timeout**: Max seconds to wait for readiness (default: 90)

<Tip>
  Tune `min-agents` and `max-agents` based on your traffic. See the [Capacity
  Planning guide](./capacity-planning) for guidance.
</Tip>

## Deploying a Pre-built Image

If you build the image in another job or CI system, deploy by setting `build: false` and passing a fully tagged image:

```yml theme={null}
- name: Deploy to Pipecat Cloud
  uses: daily-co/pipecat-cloud-deploy-action@v1
  with:
    api-key: ${{ secrets.PCC_API_KEY }}
    agent-name: ${{ github.event.repository.name }}
    image: ${{ secrets.DOCKERHUB_USERNAME }}/${{ github.event.repository.name }}:v1.2.3
    secret-set: my-secret-set
    image-credentials: my-image-pull-secret
```

## Next Steps

<CardGroup>
  <Card title="Secrets Management" icon="key" href="../fundamentals/secrets">
    Learn how to securely manage image pull secrets and environment variables.
  </Card>

  <Card title="Scaling Configuration" icon="chart-line" href="../fundamentals/scaling">
    Configure auto-scaling to handle varying traffic loads.
  </Card>
</CardGroup>


# AWS ECR
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/container-registries/aws-ecr

Deploy Pipecat Cloud agents using Amazon Elastic Container Registry

If you're using AWS ECR with private repositories for Pipecat Cloud deployments, you'll need to configure image pull secrets to authenticate with your registry. **ECR tokens expire every 12 hours**, so you'll also need to set up automatic token refresh.

## Authenticate Docker

First, authenticate Docker with ECR:

```bash theme={null}
aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <aws_account_id>.dkr.ecr.<region>.amazonaws.com
```

## Configure Image Pull Secrets

Use the Pipecat Cloud REST API to store your ECR registry credentials:

```bash theme={null}
# Get ECR login token
ECR_TOKEN=$(aws ecr get-login-password --region <region>)

# Create image pull secret using REST API
curl --request PUT \
  --url https://api.pipecat.daily.co/v1/secrets/my-ecr-credentials \
  --header 'Authorization: Bearer <your-private-api-token>' \
  --header 'Content-Type: application/json' \
  --data '{
    "secretValue": "'"$(echo -n "AWS:$ECR_TOKEN" | base64)"'",
    "host": "https://<accountid>.dkr.ecr.<region>.amazonaws.com",
    "isImagePullSecret": true
  }'
```

Replace:

* `<region>` with your AWS region (e.g., `us-east-1`, `us-west-2`)
* `<aws_account_id>` with your AWS account ID
* `<your-private-api-token>` with your Pipecat Cloud private API token

## Configure Your Deployment

Create a `pcc-deploy.toml` file with your ECR image configuration:

```toml theme={null}
agent_name = "my-ecr-agent"
image = "<aws_account_id>.dkr.ecr.<region>.amazonaws.com/your-repo:tag"
secret_set = "my-agent-secrets"
image_credentials = "my-ecr-credentials"

[scaling]
    min_instances = 0
```

## Build and Push to ECR

Build and push your agent image to ECR using the Pipecat Cloud CLI:

```bash theme={null}
# Build and push using your pcc-deploy.toml configuration
pipecat cloud docker build-push
```

This command automatically builds for the correct platform (`linux/arm64`) and pushes to your configured ECR repository.

<Note>
  Pipecat Cloud may pull your image on deploy and again during scale-outs, so
  credentials must be valid whenever new pods start.
</Note>

## Deploy Your Agent

Deploy using your configured `pcc-deploy.toml`:

```bash theme={null}
pipecat cloud deploy
```

## Automatic Token Refresh (Required)

**ECR passwords expire every 12 hours**, so set up a scheduled job to refresh the token:

```bash theme={null}
#!/bin/bash
# refresh-ecr-token.sh

# Get fresh ECR token
ECR_TOKEN=$(aws ecr get-login-password --region <region>)

# Update the existing image pull secret
curl --request PUT \
  --url https://api.pipecat.daily.co/v1/secrets/my-ecr-credentials \
  --header 'Authorization: Bearer <your-private-api-token>' \
  --header 'Content-Type: application/json' \
  --data '{
    "secretValue": "'"$(echo -n "AWS:$ECR_TOKEN" | base64)"'",
    "host": "https://<accountid>.dkr.ecr.<region>.amazonaws.com",
    "isImagePullSecret": true
  }'

```

Schedule this script to run every 6-8 hours using cron or your preferred scheduler.

## Operational Considerations

**Critical operational tips:**

* **Image pulls can happen during scale-outs**, not just at initial deploy—keep the secret valid continuously
* **If you see agents failing to become ready with no logs**, check that your ECR credentials aren't expired
* **Consider setting up monitoring alerts** for ECR token expiration
* **Test your refresh script** to ensure it works before relying on it in production

<Note>
  Ensure your AWS credentials have the necessary permissions to access ECR,
  including `ecr:GetAuthorizationToken` and `ecr:BatchGetImage` policies.
</Note>


# GCP Artifact Registry
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/container-registries/gcp-artifact-registry

Deploy Pipecat Cloud agents using Google Cloud Artifact Registry

For GCP Artifact Registry, you can use a service account JSON key to authenticate with your private registry.

## Authenticate Docker

First, authenticate Docker with your service account key:

```bash theme={null}
# Authenticate Docker with your service account key
cat key-file.json | docker login -u _json_key --password-stdin https://<location>-docker.pkg.dev
```

## Configure Image Pull Secrets

Create image credentials in Pipecat Cloud using the same service account:

```bash theme={null}
pipecat cloud secrets image-pull-secret my-gcp-credentials https://<location>-docker.pkg.dev
```

When prompted, enter:

* Username: `_json_key`
* Password: The contents of your service account JSON key file

## Configure Your Deployment

Create a `pcc-deploy.toml` file with your Artifact Registry image configuration:

```toml theme={null}
agent_name = "my-gcp-agent"
image = "<location>-docker.pkg.dev/<project-id>/<repo-name>/my-agent:0.1"
secret_set = "my-agent-secrets"
image_credentials = "my-gcp-credentials"

[scaling]
    min_instances = 0
```

## Build and Push to Artifact Registry

Build and push your agent image using the Pipecat Cloud CLI:

```bash theme={null}
# Build and push using your pcc-deploy.toml configuration
pipecat cloud docker build-push
```

This command automatically builds for the correct platform (`linux/arm64`) and pushes to your configured Artifact Registry repository.

## Deploy Your Agent

Deploy using your configured `pcc-deploy.toml`:

```bash theme={null}
pipecat cloud deploy
```

## Configuration Reference

Replace the following placeholders with your actual values:

* `<location>` with your Artifact Registry location (e.g., `us-central1`, `asia-east1`)
* `<project-id>` with your Google Cloud project ID
* `<repo-name>` with your Artifact Registry repository name
* `key-file.json` with the path to your service account JSON key file

<Note>
  The service account used for image credentials should have the `Artifact
      Registry Reader` role or equivalent permissions to pull images from your
  private repository.
</Note>


# Container Registry Overview
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/container-registries/overview

Deploy Pipecat Cloud agents from private container registries

When deploying agents to Pipecat Cloud, you can use images from both public and private container registries. While public images work out of the box, private images from any registry (including Docker Hub) require authentication setup.

## Public vs Private Images

**Public images** from any registry, including Docker Hub, work with Pipecat Cloud without additional configuration. However, **we strongly recommend using private images** for better data privacy and security when deploying production agents.

For **Docker Hub specifically**, even when using public images, we recommend setting up image pull secrets due to Docker Hub's low rate limits for unauthenticated pulls. See our [Image pull secrets guide](../../fundamentals/secrets#image-pull-secrets) for setup instructions.

## Private Registry Setup

For **private images** from any registry (including private Docker Hub repositories), you'll need to configure image pull secrets for authentication.

### Docker Hub Private Images

For private Docker Hub repositories, see the [Image pull secrets section](../../fundamentals/secrets#image-pull-secrets) in our Secrets guide for complete setup instructions.

### Other Private Registries

For other private registry providers, choose your platform below for detailed setup instructions:

<CardGroup>
  <Card title="Google Cloud Artifact Registry" icon="google" href="./gcp-artifact-registry">
    Complete setup guide for GCP Artifact Registry, including service account
    configuration and authentication.
  </Card>

  <Card title="Amazon ECR" icon="aws" href="./aws-ecr">
    Detailed AWS ECR integration with automatic token refresh setup for
    continuous deployment.
  </Card>
</CardGroup>

## General Process

All private registry integrations follow a similar pattern:

1. **Authenticate Docker** with your registry locally
2. **Create image credentials** in Pipecat Cloud using the CLI
3. **Reference credentials** in your deployment configuration
4. **Deploy your agent** using the private image

```bash theme={null}
# Create image pull secret
pipecat cloud secrets image-pull-secret my-registry-creds https://your-registry-url
```

Then configure your `pcc-deploy.toml` file:

```toml theme={null}
agent_name = "my-agent"
image = "your-registry-url/my-agent:tag"
image_credentials = "my-registry-creds"
secret_set = "my-secrets"

[scaling]
    min_agents = 0
```

Finally, deploy:

```bash theme={null}
pipecat cloud deploy
```

## Need Another Registry?

If you're using a different container registry (Azure Container Registry, GitHub Container Registry, Harbor, etc.), the general process above applies. The main differences are typically in the authentication method and registry URL format.

For assistance with other registry providers, please reach out on our [Discord community](https://discord.gg/pipecat).


# Daily WebRTC
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/daily-webrtc

Using Daily as a WebRTC transport for your Pipecat Cloud agents

[Daily](https://www.daily.co) is a WebRTC platform that provides real-time voice and video capabilities to connect users with voice agents.

Pipecat Cloud offers a first-class integration with Daily, making it straightforward to deploy WebRTC-enabled agents without managing complex infrastructure yourself.

## Using an Integrated Daily API Key

When you create a Pipecat Cloud account, you're automatically provisioned with a Daily API key that's fully integrated with the platform.

The integrated Daily API key provides:

* **Zero configuration**: No need to separately sign up for Daily or manage API keys
* **Free voice minutes**: Voice minutes for one human participant and one agent are free when using your Pipecat Cloud-provisioned Daily key
* **Simplified deployment**: Create Daily rooms and launch agents with a single command
* **Built-in compatibility**: All Pipecat base images work with Daily out of the box

<Info>
  Free voice minutes apply only to 1:1 voice sessions running on Pipecat Cloud.
  Video sessions, WebRTC sessions outside of Pipecat Cloud, and additional Daily
  features (recording, transcription, PSTN/SIP) are billed according to [Daily's
  standard pricing](https://www.daily.co/pricing/webrtc-infrastructure/).
</Info>

## Using Daily with Pipecat Cloud Agents

### Starting an Agent with Daily Transport

When starting a Pipecat Cloud agent, you can specify Daily as the transport using the `--use-daily` flag with the CLI or setting the appropriate parameters in the SDK or REST API calls.

<Tabs>
  <Tab title="CLI">
    ```bash theme={null}
    # Start an agent instance with a Daily room
    pipecat cloud agent start my-agent-name --use-daily

    # Start with custom Daily room properties
    pipecat cloud agent start my-agent-name --use-daily --daily-properties '{"enable_recording": "cloud"}'
    ```

    This command creates a Daily room and provides a URL you can open in your browser to interact with your agent using voice.
  </Tab>

  <Tab title="Python SDK">
    ```python theme={null}
    from pipecatcloud import Session, SessionParams

    # Create and start a session
    session = Session(
        agent_name="my-agent",
        api_key="pk_...",
        params=SessionParams(
            use_daily=True,
            # Optionally customize the Daily room
            daily_room_properties={
                "enable_recording": "cloud"
            }
        )
    )
    response = await session.start()

    # Access the Daily room URL and token
    daily_url = f"{response['dailyRoom']}?t={response['dailyToken']}"
    print(f"Join the call at: {daily_url}")
    ```
  </Tab>

  <Tab title="REST API">
    ```bash theme={null}
    curl --location --request POST 'https://api.pipecat.daily.co/v1/public/my-agent-name/start' \
      --header 'Authorization: Bearer YOUR_API_KEY' \
      --header 'Content-Type: application/json' \
      --data-raw '{
        "createDailyRoom": true,
        "dailyRoomProperties": {
          "enable_recording": "cloud"
        }
      }'
    ```

    The response includes `dailyRoom` (URL) and `dailyToken` fields that can be used to join the call.
  </Tab>
</Tabs>

### Using Your Own Daily API Key

While Pipecat Cloud provides a Daily API key with included 1:1 voice minutes, you can optionally use your own Daily API key if you have specific requirements. When using your own key, all usage will be billed according to your Daily account's pricing plan rather than being included with your Pipecat Cloud subscription.

## WebRTC vs. WebSockets for Voice AI

When building voice AI applications, choosing the right transport technology is crucial:

* **WebRTC** is purpose-built for real-time audio and video communication between browsers and devices:

  * Optimized for real-time media streaming over unpredictable networks
  * Uses UDP protocol, prioritizing speed over guaranteed packet delivery
  * Provides built-in echo cancellation and noise suppression
  * Intelligently adapts bitrate based on changing network conditions
  * Handles NAT traversal for connections across different networks
  * Includes device management for cameras and microphones

* **WebSockets** work well for server-to-server communication because:
  * They operate in controlled network environments with stable connections
  * When server-to-server latency is low, packet retransmission doesn't add substantial delay
  * Server environments don't need device access or media quality enhancements
  * They're simpler to implement for pure data transmission
  * Many server platforms have built-in WebSocket support

<Info>
  For browser or mobile app voice applications, WebRTC delivers superior
  performance across varying network conditions. For more details, see [How to
  Talk to an LLM with Your
  Voice](https://www.daily.co/blog/how-to-talk-to-an-llm-with-your-voice/#webrtc).
</Info>

When connecting to telephony systems like Twilio or implementing server-to-server communication where network conditions are controlled and reliable, WebSockets remain an appropriate choice. However, for any user-facing voice or video application, WebRTC offers significant advantages in quality and reliability.

## How it Works

When you start an agent with Daily integration:

1. Pipecat Cloud creates a Daily room using your API key (either the provisioned one or your custom key)
2. A URL and access token is generated, creating a means to access the Daily room
3. The agent is started and configured to join the Daily room
4. Your code (in `bot.py`) receives the room URL and token as parameters
5. The Pipecat framework automatically connects to the Daily room using these parameters

The basic flow is handled for you, allowing you to focus on building your agent's conversation logic rather than WebRTC infrastructure.

## Next Steps

<CardGroup>
  <Card title="Daily WebRTC docs" icon="globe" href="https://docs.daily.co">
    Explore Daily's complete documentation for additional features
  </Card>

  <Card title="DailyTransport Pipecat Docs" icon="puzzle-piece" href="/server/services/transport/daily">
    Explore Pipecat's DailyTransport documentation
  </Card>

  <Card title="Pipecat Examples" icon="code" href="https://github.com/pipecat-ai/pipecat-examples">
    View example Pipecat projects using Daily WebRTC
  </Card>
</CardGroup>


# Krisp VIVA Noise Cancellation
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/krisp-viva

Eliminate background noise and voices for more reliable agent interactions using Krisp's VIVA noise cancellation model

## Overview

Voice agents are vulnerable to background noise, especially other voices that trigger false interruptions. When this happens, your bot stops mid-sentence unnecessarily, creating a poor user experience. Pipecat Cloud's Krisp integration processes incoming audio to remove ambient noise and background voices before they reach your agent, resulting in cleaner audio input with fewer false interruptions and better transcription.

<Info>
  Krisp processes the audio before it reaches your agent's speech recognition
  and VAD systems, helping to ensure that only the intended user's voice
  triggers interruptions.
</Info>

## Enabling Krisp VIVA

Enabling Krisp VIVA with Pipecat Cloud involves the following steps:

### 1. Use the Pipecat Base Image

Krisp VIVA dependencies are pre-installed in the official Pipecat base image. You don't need to add any Krisp VIVA-related packages to your `requirements.txt` file or `pyproject.toml` file.

<Note>
  Krisp VIVA support is available in the `dailyco/pipecat-base` image starting
  with version `0.1.8`.
</Note>

### 2. Add the KrispVivaFilter() to Your Transport

In your bot code, import and add the `KrispVivaFilter()` to your transport:

```python theme={null}
from pipecat.audio.filters.krisp_viva_filter import KrispVivaFilter
from pipecat.transports.daily.transport import DailyParams, DailyTransport

transport = DailyTransport(
    room_url,
    token,
    "Voice AI Bot",
    DailyParams(
        audio_in_enabled=True,
        audio_in_filter=KrispVivaFilter(),  # Add Krisp VIVA filter here
        audio_out_enabled=True,
    ),
)
```

### 3. Enable Krisp VIVA in Your Deployment

You can enable Krisp VIVA when deploying your agent through:

<Tabs>
  <Tab title="Config File">
    In your `pcc-deploy.toml`:

    ```toml theme={null}
    agent_name = "voice-starter"
    image = "your-repo/voice-starter:0.1"
    image_credentials = "my-dockerhub-secret"
    secret_set = "voice-starter-secrets"

    [krisp_viva]
    audio_filter = "tel"

    [scaling]
    min_agents = 0
    ```
  </Tab>

  <Tab title="CLI">
    ```bash theme={null}
    pipecat cloud deploy --krisp-viva-audio-filter tel
    ```
  </Tab>

  <Tab title="REST API">
    When creating or updating an agent via the REST API, include the `krispViva` parameter:

    ```bash theme={null}
    curl -X POST https://api.pipecat.daily.co/v1/agents \
      -H "Authorization: Bearer YOUR_PRIVATE_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{
        "serviceName": "voice-starter",
        "image": "your-repo/voice-starter:0.1",
        "secretSet": "voice-starter-secrets",
        "krispViva": {
          "audioFilter": "tel"
        },
        "autoScaling": {
          "minAgents": 1,
          "maxAgents": 10
        }
      }'
    ```

    To update an existing agent:

    ```bash theme={null}
    curl -X POST https://api.pipecat.daily.co/v1/agents/voice-starter \
      -H "Authorization: Bearer YOUR_PRIVATE_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{
        "krispViva": {
          "audioFilter": "tel"
        }
      }'
    ```

    See the [REST API reference](/deployment/pipecat-cloud/rest-reference/endpoint/agent-create) for more details.
  </Tab>

  <Tab title="Dashboard">
    In the Pipecat Cloud dashboard:

    1. Navigate to your agent
    2. Go to Settings > Agent Configuration
    3. Select your desired model from the dropdown
    4. Save your changes which will automatically deploy your agent with the selected model
  </Tab>
</Tabs>

Supported values are:

* `tel`: Supports up to 16kHz audio
* `pro`: Supports up to 32kHz audio

<Tip>
  By default, Pipecat processes input audio at 16kHz, making the `tel` value
  appropriate for most use cases.
</Tip>

## Testing Your Integration

To verify Krisp is working:

1. Deploy your agent with Krisp enabled
2. Connect to your agent in an environment with background noise; try turning on a TV or podcast
3. Observe whether background noises and voices affect your agent's behavior

## Pricing

Krisp processing is billed based on active session minutes:

| Usage Tier                     | Price               |
| ------------------------------ | ------------------- |
| First 10,000 minutes per month | Free                |
| Additional minutes             | \$0.0015 per minute |

<Note>
  For agents deployed with Krisp enabled, active session minutes count as Krisp
  usage, accruing towards your monthly bill.
</Note>

## Local Development

Krisp cannot be used in local development environments, as it requires a proprietary SDK and model which can only be distributed on Pipecat Cloud. When running locally be sure to not import the KrispFilter module and don't apply it to your transport.

## Troubleshooting

If you're experiencing issues with Krisp noise cancellation:

* Verify Krisp VIVA is enabled in your deployment configuration
* Confirm you've added the `KrispVivaFilter()` to your transport configuration
* Check your logs for any Krisp-related error messages
* Confirm that you have **not** added any Krisp-related dependencies to your `requirements.txt` or `pyproject.toml` files, as they are not needed and may cause conflicts.


# Managed API Keys
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/managed-api-keys

Using managed API keys for your Pipecat Cloud agents

## Overview

Pipecat Cloud provides API keys for some Pipecat services that you can use directly in your applications. These managed keys give you access to competitive pricing, higher rate limits, pre-signed BAAs and DPAs for compliance, and integrated billing with your Pipecat Cloud account.

<Note>
  Managed keys are currently in beta. We are planning to add more services based
  on demand. If you're interested in us supporting a specific service, let us
  know by emailing [pipecat-ai@daily.co](mailto:pipecat-ai@daily.co).
</Note>

## Supported Services

* **Deepgram Flux (STT)**

## Pricing

Usage of managed keys is billed through your Pipecat Cloud account. For detailed pricing information, visit our [pricing page](https://www.daily.co/pricing/pipecat-cloud/).

## How to Use Managed Keys

### Enable in Deployment Configuration

When deploying your agent, you need to enable managed keys in your configuration.

**Using pcc-deploy.toml:**

```toml theme={null}
enable_managed_keys = true
```

**Using the CLI:**

```bash theme={null}
pipecat cloud deploy --enable-managed-keys true
```

<Info>
  This setting is configured per agent deployment. Any changes require
  redeploying your agent to take effect.
</Info>

### Update Your Code

When initializing a supported service in your Pipecat application, use the special string `"PIPECATCLOUD"` as the API key. This tells Pipecat Cloud to use the managed key when your agent is deployed.

To ensure your code works both locally (with your own API key) and when deployed (with managed keys), use this pattern:

```python theme={null}
import os
from pipecat.services.deepgram.flux.stt import DeepgramFluxSTTService

tts = DeepgramFluxSTTService(api_key=os.getenv("DEEPGRAM_API_KEY", "PIPECATCLOUD"))
```

This approach:

* Uses your local `DEEPGRAM_API_KEY` environment variable during development
* Automatically switches to the managed key when deployed to Pipecat Cloud

<Note>
  The `PIPECATCLOUD` key is only available for Deepgram Flux (STT) when there is
  no value for `DEEPGRAM_API_KEY` in your corresponding secrets set.
</Note>

## Getting Help

If you have any questions about using managed keys, reach out to us on [Discord](https://discord.gg/pipecat). Our community and team are happy to help!


# Regions
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/regions

Deploy agents to geographic regions for optimal performance and compliance

Pipecat Cloud supports deploying agents to geographic regions, allowing you to reduce latency for your users and meet data residency requirements.

## Available Regions

You can view all available regions using the CLI:

```bash theme={null}
pipecat cloud regions list
```

Or via the [REST API](/deployment/pipecat-cloud/rest-reference/endpoint/regions-list).

Available regions include:

* **us-west**: US West (Oregon)
* **us-east**: US East (Virginia)
* **eu-central**: Europe (Frankfurt)
* **ap-south**: Asia Pacific (Mumbai)

## Regional Requirements

When deploying agents and managing secrets, keep these requirements in mind:

* **Agent names are globally unique**: Each agent must have a unique name across all regions
* **Secret sets are region-specific (with globally unique names)**: Create a separate secret set in each region where you deploy agents
* **Image pull secrets are region-specific**: Create separate image pull secrets in each region if using private registries
* **Secrets must match agent regions**: An agent can only use secrets and image pull secrets from the same region

## Regional WebSocket Endpoints

When connecting to Pipecat Cloud via WebSocket for telephony services, you can specify a region to reduce latency and ensure your traffic stays within a geographic area.

### Default Endpoint

By default, WebSocket connections use the us-west:

```
wss://api.pipecat.daily.co/ws/{service}
```

Where `{service}` is one of:

* `twilio` - For Twilio WebSocket connections
* `telnyx` - For Telnyx WebSocket connections
* `plivo` - For Plivo WebSocket connections
* `exotel` - For Exotel WebSocket connections

### Regional Endpoints

To connect to a specific region, use the regional endpoint format:

```
wss://{region}.api.pipecat.daily.co/ws/{service}
```

**Example:**

Connect to Twilio in Europe:

```
wss://eu-central.api.pipecat.daily.co/ws/twilio
```

<Note>
  Regional WebSocket endpoints route to agents deployed in that region. Make
  sure your agent is deployed in the same region as the WebSocket endpoint
  you're connecting to.
</Note>

## Setting a Default Region

You can specify a default region for your organization via the [REST API](/deployment/pipecat-cloud/rest-reference/endpoint/properties-update) or the [CLI](/cli/cloud/organizations#default-region). If you don't specify a region, resources are created in your organization's default region.

## Deploying an Agent to a Region

This walkthrough will guide you through deploying an agent to a specific region.

### Step 1: Build and Push Your Docker Image

Build your agent Docker image and push it to a container registry. This guide assumes you already have a working agent image.

<Tip>
  Need help building agent images? See [Agent
  Images](/deployment/pipecat-cloud/fundamentals/agent-images) for detailed
  instructions.
</Tip>

### Step 2: Create a Secret Set

Create a secret set in your target region with the environment variables your agent needs.

```bash theme={null}
pipecat cloud secrets set my-agent-secrets \
  OPENAI_API_KEY=sk-xxx \
  DEEPGRAM_API_KEY=xxx \
  --region us-east
```

<Tip>
  Learn more about managing secrets in the
  [Secrets](/deployment/pipecat-cloud/fundamentals/secrets) guide.
</Tip>

### Step 3: Create an Image Pull Secret

If you're using a private container registry, create an image pull secret in the same region.

```bash theme={null}
pipecat cloud secrets image-pull-secret my-registry-creds \
  https://index.docker.io/v1/ \
  --region us-east
```

When prompted, enter your registry username and access token.

<Note>
  Using a public registry? You can skip this step and omit `image_credentials`
  from your deployment configuration.
</Note>

### Step 4: Create a Deployment Configuration

Create a `pcc-deploy.toml` file in your project directory:

```toml theme={null}
agent_name = "my-agent"
image = "your-username/my-agent:latest"
region = "us-east"
secret_set = "my-agent-secrets"
image_credentials = "my-registry-creds"
agent_profile = "agent-1x"

[scaling]
min_agents = 1
max_agents = 10
```

<Tip>
  See the [deploy command
  reference](/cli/cloud/deploy#configuration-file-pcc-deploytoml) for all
  available configuration options.
</Tip>

### Step 5: Deploy Your Agent

Deploy your agent using the configuration file:

```bash theme={null}
pipecat cloud deploy
```

The CLI will read your `pcc-deploy.toml` file and deploy the agent to the specified region. Once the deployment is ready, you can start sessions with your agent.

## Deploying to Multiple Regions

To serve users globally, repeat the steps above for each target region. Use region-specific naming for agents, secrets, and image pull secrets (e.g., `my-agent-us-east`, `my-agent-secrets-us-east`).

Your application determines which regional agent to connect users to based on their location, preferences, or latency requirements.

## Learn More

* [Deployments](/deployment/pipecat-cloud/fundamentals/deploy) - Comprehensive deployment guide
* [Secrets](/deployment/pipecat-cloud/fundamentals/secrets) - Managing secrets and credentials


# Session API
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/session-api

Send HTTP requests to your running Pipecat Cloud sessions

The Session API lets you send HTTP requests directly to your running Pipecat Cloud agents. This enables real-time control and data exchange with active sessions, such as updating conversation context, triggering actions, or retrieving session state.

## How It Works

1. [Start a session](/deployment/pipecat-cloud/fundamentals/active-sessions) with your agent
2. Capture the `sessionId` header from the start response
3. Send HTTP requests to your session using the session endpoint

## Endpoint Format

```
https://api.pipecat.daily.co/v1/public/{service_name}/sessions/{session_id}/{path}
```

| Parameter      | Description                                   |
| -------------- | --------------------------------------------- |
| `service_name` | The name of your deployed agent               |
| `session_id`   | The `sessionId` value from the start response |
| `path`         | The endpoint path you defined in your bot     |

### Supported Methods

* `GET`
* `POST`
* `PUT`
* `PATCH`
* `DELETE`
* `OPTIONS`
* `HEAD`

## Authentication

Include your Pipecat Cloud public API key in the `Authorization` header:

```bash theme={null}
Authorization: Bearer pk_...
```

## Setting Up Your Bot

To handle Session API requests, define endpoints in your `bot.py` file using the `app` object from `pipecatcloud_system`:

```python theme={null}
from pipecatcloud_system import app
from pipecat.runner.types import PipecatRunnerArguments

# Define your API endpoints
@app.get("/status")
async def get_status():
    return {"status": "active", "message": "Bot is running"}

@app.post("/update-context")
async def update_context(data: dict):
    # Handle context updates
    return {"updated": True}

# Your main bot function
async def bot(args: PipecatRunnerArguments):
    # Bot implementation
    pass
```

<Note>Requires base image version `0.1.2` or later.</Note>

## Examples

### Get Session Status

Define an endpoint that returns the current session state:

```python theme={null}
from pipecatcloud_system import app

session_data = {"messages": [], "user_name": None}

@app.get("/status")
async def get_status():
    return {
        "message_count": len(session_data["messages"]),
        "user_name": session_data["user_name"]
    }
```

Call the endpoint:

```bash theme={null}
curl -X GET \
  'https://api.pipecat.daily.co/v1/public/my-agent/sessions/57af5437-97a2-4646-9873-a5c5935bd705/status' \
  -H 'Authorization: Bearer pk_...'
```

### Update Conversation Context

Define an endpoint to inject information into the conversation:

```python theme={null}
from pipecatcloud_system import app
from pydantic import BaseModel

class ContextUpdate(BaseModel):
    user_name: str
    preferences: dict

@app.post("/context")
async def update_context(update: ContextUpdate):
    # Update your bot's context with the new information
    # This could modify the LLM system prompt, add to conversation history, etc.
    return {"status": "context updated", "user_name": update.user_name}
```

Call the endpoint:

```bash theme={null}
curl -X POST \
  'https://api.pipecat.daily.co/v1/public/my-agent/sessions/57af5437-97a2-4646-9873-a5c5935bd705/context' \
  -H 'Authorization: Bearer pk_...' \
  -H 'Content-Type: application/json' \
  -d '{"user_name": "Alice", "preferences": {"language": "Spanish"}}'
```

### Trigger an Action

Define an endpoint to trigger bot behavior mid-session:

```python theme={null}
from pipecatcloud_system import app

@app.post("/speak")
async def trigger_speech(data: dict):
    message = data.get("message", "Hello!")
    # Queue the message for your TTS pipeline
    # Implementation depends on your bot architecture
    return {"queued": True, "message": message}
```

## Important Notes

* **Startup latency**: If you call the Session API before your bot finishes initializing, the request may take longer while waiting for the bot to become available.
* **Session scope**: Each endpoint call is routed to the specific session identified by `session_id`. Different sessions run independently.
* **Error handling**: If a session has ended or the session ID is invalid, you'll receive an error response.

## Getting Help

If you have questions about the Session API, reach out on [Discord](https://discord.gg/pipecat).


# Smart Turn Detection
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/smart-turn

Advanced conversational turn detection

## Overview

Smart Turn Detection uses an advanced machine learning model to determine when a user has finished speaking and your bot should respond. Unlike basic Voice Activity Detection (VAD) which only detects speech vs. non-speech, Smart Turn Detection recognizes natural conversational cues like intonation patterns and linguistic signals for more natural conversations.

<CardGroup>
  <Card title="GitHub Repository" icon="github" href="https://github.com/pipecat-ai/smart-turn">
    Open source model for conversational turn detection
  </Card>

  <Card title="Model weights" icon="face-smiling-hands" href="https://huggingface.co/pipecat-ai/smart-turn-v3/tree/main">
    ONNX weights file for Smart Turn v3
  </Card>
</CardGroup>

## Key Benefits

* **Natural conversations**: More human-like turn-taking patterns
* **Free to use**: The model is fully open source
* **Scalable**: Smart Turn v3 supports fast CPU inference directly inside your Pipecat Cloud instance

## Quick Start

To enable Smart Turn Detection in your Pipecat Cloud bot, configure a `TurnAnalyzerUserTurnStopStrategy` with `LocalSmartTurnAnalyzerV3` in your context aggregator.

The model weights are bundled with Pipecat, so there's no need to download them separately.

```python theme={null}
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.audio.vad.vad_analyzer import VADParams
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.transports.daily.transport import DailyParams, DailyTransport
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies

async def main(room_url: str, token: str):
    transport = DailyTransport(
        room_url,
        token,
        "Voice AI Bot",
        DailyParams(
            audio_in_enabled=True,
            audio_out_enabled=True,
        ),
    )

    # Configure Smart Turn Detection via user turn strategies
    user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
        context,
        user_params=LLMUserAggregatorParams(
            user_turn_strategies=UserTurnStrategies(
                stop=[TurnAnalyzerUserTurnStopStrategy(
                    turn_analyzer=LocalSmartTurnAnalyzerV3()
                )]
            ),
            vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
        ),
    )

    # Continue with your pipeline setup...
```

<Tip>
  Smart Turn Detection requires VAD to be enabled with `stop_secs=0.2`. This
  value mimics the training data and allows Smart Turn to dynamically adjust
  timing based on the model's predictions.
</Tip>

## How It Works

1. **Audio Analysis**: The system continuously analyzes incoming audio for speech patterns
2. **VAD Processing**: Voice Activity Detection segments audio into speech and silence
3. **Turn Classification**: When VAD detects a pause, the ML model analyzes the speech segment for natural completion cues
4. **Smart Response**: The model determines if the turn is complete or if the user is likely to continue speaking

## Training Data Collection

The smart-turn model is trained on real conversational data collected through these applications. Help us improve the model by contributing your own data or classifying existing data:

<CardGroup>
  <Card title="Data Collector" icon="microphone" href="https://turn-training.pipecat.ai/">
    Contribute conversational data to improve the model
  </Card>

  <Card title="Data Classifier" icon="check-circle" href="https://smart-turn-dataset.pipecat.ai/">
    Help classify turn completion patterns in conversations
  </Card>
</CardGroup>

## More information

For more details on Smart Turn, see the following links:

<CardGroup>
  <Card title="Smart Turn Overview" href="../../../server/utilities/smart-turn/smart-turn-overview">
    More details about the Pipecat Smart Turn integration
  </Card>
</CardGroup>


# Daily PSTN Dial-in
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/telephony/daily-dial-in

Using Daily's Transport to receive calls from external SIP Addresses and PSTN numbers

When someone calls your Daily phone number, the call is placed on hold and triggers a webhook to initialize your Pipecat bot.

## Setup Options

### Option 1: Automatic Webhook (Recommended)

Let Pipecat Cloud handle the webhook automatically.

**Using the Dashboard:**

1. Go to **Settings > Telephony** in Pipecat Cloud
2. Select your phone number
3. Choose your deployed agent
4. Save

**Using the REST API:**

```bash theme={null}
curl --location 'https://api.daily.co/v1' \
--header 'Authorization: Bearer YOUR_DAILY_API_KEY' \
--header 'Content-Type: application/json' \
--data '{
    "properties": {
        "pinless_dialin": [{
            "phone_number": "YOUR_DAILY_NUMBER",
            "room_creation_api": "https://api.pipecat.daily.co/v1/public/webhooks/{org_id}/{agent_name}/dialin"
        }]
    }
}'
```

### Option 2: Custom Webhook Server

Host your own webhook server for custom logic (e.g., CRM integration, advanced routing, caller-based customization).

<CardGroup>
  <Card title="FastAPI Webhook Server" icon="server" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/deployment/pipecat-cloud-daily-pstn-server/fastapi-webhook-server">
    Python webhook server example
  </Card>

  <Card title="Next.js Webhook Server" icon="js" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/deployment/pipecat-cloud-daily-pstn-server/nextjs-webhook-server">
    JavaScript webhook server example
  </Card>
</CardGroup>

**Webhook payload from Daily:**

```json theme={null}
{
  "To": "+15559876543",
  "From": "+15551234567",
  "callId": "uuid",
  "callDomain": "uuid"
}
```

**Forward to Pipecat Cloud:**

```bash theme={null}
curl --request POST \
--url https://api.pipecat.daily.co/v1/public/{service}/start \
--header 'Authorization: Bearer $API_KEY' \
--header 'Content-Type: application/json' \
--data '{
    "createDailyRoom": true,
    "dailyRoomProperties": {
        "sip": {
            "display_name": "+15551234567",
            "sip_mode": "dial-in",
            "num_endpoints": 1
        }
    },
    "body": {
        "dialin_settings": {
            "from": "+15551234567",
            "to": "+15559876543",
            "call_id": "uuid",
            "call_domain": "uuid"
        }
    }
}'
```

## Bot Configuration

Configure your bot to handle dial-in calls:

```python theme={null}
from pipecat.runner.types import DailyDialinRequest, RunnerArguments
from pipecat.transports.daily import DailyTransport, DailyParams, DailyDialinSettings

async def bot(runner_args: RunnerArguments):
    # Parse dial-in request
    request = DailyDialinRequest.model_validate(runner_args.body)

    # Configure dial-in settings
    dialin_settings = DailyDialinSettings(
        call_id=request.dialin_settings.call_id,
        call_domain=request.dialin_settings.call_domain
    )

    # Create transport
    transport = DailyTransport(
        runner_args.room_url,
        runner_args.token,
        "Voice Bot",
        DailyParams(
            api_key=request.daily_api_key,
            api_url=request.daily_api_url,
            dialin_settings=dialin_settings,
            audio_in_enabled=True,
            audio_out_enabled=True,
        )
    )

    # Start speaking when caller joins
    @transport.event_handler("on_first_participant_joined")
    async def on_first_participant_joined(transport, participant):
        await task.queue_frames([LLMRunFrame()])

    # Your bot pipeline setup...
```

**Personalize with caller info:**

```python theme={null}
caller_phone = request.dialin_settings.From
customer = await get_customer_by_phone(caller_phone)
# Use customer data in your system prompt
```

## Complete Example

<Card title="Daily PSTN Dial-in Bot" icon="phone" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-pstn-dial-in">
  Full implementation with STT, LLM, TTS, and caller personalization
</Card>


# Daily PSTN Dial-out
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/telephony/daily-dial-out

Using Daily’s Transport to make calls to external SIP Addresses and PSTN numbers

To initiate outbound calls, you first need to configure your Daily rooms to allow dial-out by setting `enable_dialout` in `dailyRoomProperties`. Then, include the target phone number(s) in the request `body` of `{service}/start`.

In this example, we use the `dialout_settings` array to pass one or more phone numbers for the bot to dial. Each object in the array must include a `phoneNumber` and can optionally specify a `callerId` to control the outbound caller ID.

<Info>
  If no `callerId` is provided, a random phone number will be used to make the
  call. To specify the caller ID, use the [phone number's
  `id`](/guides/telephony/daily-phone-numbers#list-all-purchased-phone-numbers)
  as the `callerId`.
</Info>

```bash theme={null}
curl --request POST \
--url https://api.pipecat.daily.co/v1/public/{service}/start \
--header 'Authorization: Bearer $API_KEY' \
--header 'Content-Type: application/json' \
--data-raw '{
    'createDailyRoom': true,
    'dailyRoomProperties': {
        'enable_dialout': true,
        'exp': 1742353929
    },
    'body': {
        'dialout_settings': \
        [{'phoneNumber': '+1TARGET', 'callerId': 'UUID_OF_PURCHASED_NUM'}]
    }
}
```

<Info>
  The API\_KEY here is Pipcat Cloud's [public API
  key](../../fundamentals/accounts-and-organizations#api-keys), typically begins
  with "pk\_..."
</Info>

The bot should begin dialing out as soon as the call state transitions to the `joined` state.

Unlike dial-in scenarios where the bot typically speaks first, dial-out calls may require the bot to wait for the remote user to speak. In these cases, you can start listening for the remote user when the `on_dialout_answered` event fires.

This is also a good point to initiate voicemail detection—for example, by analyzing the remote user's audio to determine if the call has reached a voicemail.

```python theme={null}
@transport.event_handler("on_call_state_updated")
async def on_call_state_updated(transport, state):
    if state == "joined" and dialout_settings:
        await start_dialout(transport, dialout_settings)

@transport.event_handler("on_dialout_answered")
    async def on_dialout_answered(transport, data):
        # the bot will wait for the user to speak before responding
        await rtvi.set_bot_ready()
```

### Examples of `dialout_settings`

Here are various configurations you can use in the `dialout_settings` array:

<CodeGroup>
  ```json Twilio SIP Address theme={null}
  "dialout_settings": [
    { 
      "sipUri": "sip:+1xxxx@domain.sip.twilio.com" 
    }
  ]
  ```

  ```json Phone Number with CallerID theme={null}

  "dialout_settings": [
    {
      "phoneNumber": "+1xxx",
      "callerId": "uuid"
    }
  ]

  ```

  ```json Multiple Phone Numbers theme={null}
  "dailout_settings": [
    {
      "phoneNumber": "+1xxx",
      "callerId": "uuid-1"
    },
    {
      "sipUri": "sip:+1xxxx@domain.sip.twilio.com"
    },
    {
      "phoneNumber": "+1xxx",
      "callerId": "uuid-1"
    }
  ]
  ```
</CodeGroup>

## Next Steps

After configuring your dial-out functionality, you can implement a Pipecat bot to handle the call interactions:

<Card title="Dial-in/Dial-out Bot Example" icon="phone" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-pstn-dial-out">
  Complete example of a Pipecat bot implementation that handles both incoming
  (dial-in) and outgoing (dial-out) calls with voicemail detection
</Card>


# Exotel Websocket Transport
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/telephony/exotel-websocket

Using Exotel's WebSocket Transport for your Pipecat Cloud agents

<Note>
  This guide covers Pipecat Cloud-specific configuration for Exotel WebSocket
  integration. For a complete guide including dial-in, dial-out, and advanced
  features, see the [Exotel WebSocket Integration
  guide](/guides/telephony/exotel-websockets).
</Note>

Native support for Exotel's WebSocket Transport with Pipecat Cloud allows you to connect your AI agents with Exotel's voice infrastructure. This integration enables your Pipecat bots to handle real phone calls using Exotel's Voice Streaming.

## How It Works

Pipecat Cloud implements Exotel's bidirectional voice streaming protocol. While audio streams flow through WebSockets, the call session is controlled through Exotel's App Bazaar configuration and the Voicebot Applet.

When Pipecat Cloud receives an incoming WebSocket connection from Exotel, it processes the `Connected` and `Start` messages to initialize a new bot instance. All WebSocket messages are forwarded to your bot, including any custom parameters set in your App Bazaar configuration. This allows your bot to leverage Exotel's API for advanced call control - such as recording conversations, transferring to human agents, or implementing complex call flows.

## Prerequisites

Before setting up this integration, ensure you have:

* An Exotel account with voice streaming enabled
* A Pipecat Cloud account with an Exotel Websockets-compatible bot

A ready-to-build example of an Exotel websockets bot with complete source code is available in the [pipecat-examples repo](https://github.com/pipecat-ai/pipecat-examples/tree/main/exotel-chatbot).

## Pipecat Cloud Configuration

### 1. Purchase and Configure Phone Number

[Purchase a phone number from Exotel](https://my.exotel.com/) if you haven't already:

* Navigate to ExoPhones and purchase a number
* Complete KYC verification if required

### 2. Enable Voice Streaming

Voice streaming may not be enabled by default on all accounts:

* Contact Exotel support at `hello@exotel.com`
* Request: "Enable Voicebot Applet for voice streaming for account \[Your Account SID]"
* Include your use case: "AI voice bot integration"

### 3. Get Your Organization Name

Retrieve your Pipecat Cloud organization name using the Pipecat CLI:

```bash theme={null}
$ pipecat cloud organizations list
```

This command will output a list of organizations associated with your account. For example:

```bash theme={null}
Organization        Name
──────────────────────────────────────
Default Workspace   three-random-words-randomnumber (active)
```

### 4. Create App Bazaar Flow

1. Navigate to App Bazaar in your Exotel dashboard
2. Click "Create" to create a new app
3. Build your call flow:

**Add Voicebot Applet:**

* Drag the "Voicebot" applet to your call flow (not "Stream" or "Passthru")
* Configure the WebSocket URL:

```
wss://api.pipecat.daily.co/ws/exotel?serviceHost=AGENT_NAME.ORGANIZATION_NAME
```

Replace the placeholder values:

* `AGENT_NAME` with your deployed bot's name (e.g., `customer-support`)
* `ORGANIZATION_NAME` with your organization name from step 3 (e.g., `three-random-words-randomnumber`)

For example: `wss://api.pipecat.daily.co/ws/exotel?serviceHost=customer-support.industrious-purple-cat-12345`

<Note>
  **Using Regional Endpoints**

  If you deployed your agent to a specific region, use the regional WebSocket endpoint:

  `wss://{region}.api.pipecat.daily.co/ws/exotel?serviceHost=AGENT_NAME.ORGANIZATION_NAME`

  For example, for Europe: `wss://eu-central.api.pipecat.daily.co/ws/exotel?serviceHost=my-agent.my-org`

  Learn more about [regional endpoints](/deployment/pipecat-cloud/guides/regions#regional-websocket-endpoints).
</Note>

**Add Hangup Applet:**

* Drag a "Hangup" applet at the end to properly terminate calls

Your final flow should look like:

```
Call Start → [Voicebot Applet] → [Hangup Applet]
```

### 5. Link Phone Number to App

1. Navigate to "ExoPhones" in your dashboard
2. Find your purchased number and click the edit/pencil icon
3. Under "App", select the custom app you created
4. Save the configuration

## Testing Your Integration

To test your integration, simply dial your Exotel phone number from any phone. The call will connect to your Pipecat Cloud bot, which will respond according to your bot's configuration.

## Custom Data Limitations

<Warning>
  Currently, there are limitations with passing custom data through query
  parameters in Exotel's WebSocket URLs. While the platform theoretically
  supports query parameters, they may be stripped during call processing.
</Warning>

## Next Steps

For complete implementation details including dial-out, caller personalization, and advanced call control features, see the [Exotel WebSocket Integration guide](/guides/telephony/exotel-websockets).


# Plivo Websocket Transport
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/telephony/plivo-websocket

Using Plivo's WebSocket Transport for your Pipecat Cloud agents

<Note>
  This guide covers Pipecat Cloud-specific configuration for Plivo WebSocket
  integration. For a complete guide including dial-in, dial-out, and advanced
  features, see the [Plivo WebSocket Integration
  guide](/guides/telephony/plivo-websockets).
</Note>

Native support for Plivo's WebSocket Transport with Pipecat Cloud allows you to connect your AI agents with Plivo's voice infrastructure. This integration enables your Pipecat bots to handle real phone calls using Plivo's WebSocket streaming.

## How It Works

Pipecat Cloud implements Plivo's bidirectional [Media Streaming protocol](https://www.plivo.com/docs/voice/guides/stream-audio/). While audio streams flow through WebSockets, the call session is controlled by XML responses that tell Plivo how to handle each call.

When Pipecat Cloud receives an incoming WebSocket connection from Plivo, it processes the `start` message to initialize a new bot instance. All WebSocket messages are forwarded to your bot, including call information such as the caller's number and call UUID. This allows your bot to leverage Plivo's Voice API for advanced call control - such as recording conversations, transferring to human agents, or implementing complex call flows.

## Prerequisites

Before setting up this integration, ensure you have:

* A Plivo account with voice capabilities
* A Pipecat Cloud account with a Plivo WebSocket-compatible bot
* A web server to host XML responses (we'll show you how to set this up)

A ready-to-build example of a Plivo websockets bot with complete source code is available in the [pipecat-examples repo](https://github.com/pipecat-ai/pipecat-examples/tree/main/plivo-chatbot).

## Pipecat Cloud Configuration

### 1. Get Your Organization Name

Retrieve your Pipecat Cloud organization name using the Pipecat CLI:

```bash theme={null}
$ pipecat cloud organizations list
```

This command will output a list of organizations associated with your account. For example:

```bash theme={null}
Organization        Name
──────────────────────────────────────
Default Workspace   three-random-words-randomnumber (active)
```

### 2. Set Up an XML Server

Unlike Twilio's TwiML Bins, Plivo requires a web server to host your XML responses. Create a file called `server.py`:

```python theme={null}
from fastapi import FastAPI, Query, HTTPException
from starlette.responses import Response
import uvicorn

app = FastAPI(title="Plivo XML Server")

@app.get("/plivo-xml")
async def plivo_xml(
    agent: str = Query(..., description="Agent name"),
    org: str = Query(..., description="Organization name"),
):
    """
    Returns XML for Plivo to start WebSocket streaming to Pipecat Cloud
    Example: /plivo-xml?agent=my-bot&org=my-org-123
    """
    if not agent or not org:
        raise HTTPException(status_code=400, detail="Both 'agent' and 'org' parameters are required")

    xml = f"""<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Stream bidirectional="true" keepCallAlive="true" contentType="audio/x-mulaw;rate=8000">
    wss://api.pipecat.daily.co/ws/plivo?serviceHost={agent}.{org}
  </Stream>
</Response>"""

    return Response(content=xml, media_type="application/xml")

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=7860)
```

<Note>
  **Using Regional Endpoints**

  If you deployed your agent to a specific region, use the regional WebSocket endpoint:

  `wss://{region}.api.pipecat.daily.co/ws/plivo?serviceHost=AGENT_NAME.ORGANIZATION_NAME`

  For example, for Europe: `wss://eu-central.api.pipecat.daily.co/ws/plivo?serviceHost=my-agent.my-org`

  Learn more about [regional endpoints](/deployment/pipecat-cloud/guides/regions#regional-websocket-endpoints).
</Note>

### 3. Deploy Your XML Server

Install dependencies and run your server:

```bash theme={null}
pip install fastapi uvicorn
python server.py
```

For local testing, use ngrok to make your server publicly accessible:

```bash theme={null}
ngrok http 7860
```

This will give you a public URL like `https://abc123.ngrok.io`.

<Tip>
  Use the `--subdomain` flag with ngrok to keep your URL consistent across
  restarts.
</Tip>

For production, deploy your XML server to a reliable hosting platform.

### 4. Configure Plivo

[Purchase a phone number from Plivo](https://www.plivo.com/docs/numbers/quickstart/) if you haven't already, then:

1. Navigate to Voice → XML in your Plivo dashboard
2. Select Add New Application
3. Enter a name for your application (e.g., "Pipecat WebSocket")
4. Set the **Answer URL** to your server URL with agent and organization parameters:
   ```
   https://your-server.com/plivo-xml?agent=AGENT_NAME&org=ORGANIZATION_NAME
   ```
   * Replace `AGENT_NAME` with your deployed bot's name (e.g., `my-first-agent`)
   * Replace `ORGANIZATION_NAME` with your organization name from step 1 (e.g., `three-random-words-randomnumber`)
5. Set **HTTP Method** to `GET`
6. Click Create Application

### 5. Assign to Your Phone Number

1. Navigate to Phone Numbers and select your phone number
2. Select XML Application as the Application Type
3. Select your XML Application from the Plivo Application dropdown
4. Click Update to apply your changes

## Testing Your Integration

To test your integration, simply dial your Plivo phone number from any phone. The call will connect to your Pipecat Cloud bot, which will respond according to your bot's configuration.

## Next Steps

For complete implementation details including dial-out and advanced call control features, see the [Plivo WebSocket Integration guide](/guides/telephony/plivo-websockets).


# Telnyx Websocket Transport
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/telephony/telnyx-websocket

Using Telnyx's WebSocket Transport for your Pipecat Cloud agents

<Note>
  This guide covers Pipecat Cloud-specific configuration for Telnyx WebSocket
  integration. For a complete guide including dial-in, dial-out, custom
  parameters, and advanced features, see the [Telnyx WebSocket Integration
  guide](/guides/telephony/telnyx-websockets).
</Note>

Native support for Telnyx's WebSocket Transport with Pipecat Cloud allows you to connect your AI agents with Telnyx's voice infrastructure. This integration enables your Pipecat bots to handle real phone calls using Telnyx's WebSocket streaming.

## How It Works

Pipecat Cloud implements Telnyx's bidirectional [Media Streaming protocol](https://developers.telnyx.com/docs/voice/programmable-voice/media-streaming). While audio streams flow through WebSockets, the call session is controlled by updating the Telnyx Extensible Markup Language (TeXML) associated with each call's unique identifier (`call_control_id`).

When Pipecat Cloud receives an incoming WebSocket connection from Telnyx, it processes the `connected` and `start` messages to initialize a new bot instance. All WebSocket messages are forwarded to your bot, including any custom parameters set in your TeXML. This allows your bot to leverage Telnyx's Call Control API for advanced call control - such as recording conversations, transferring to human agents, or implementing complex call flows.

<Frame>
  <img />
</Frame>

## Prerequisites

Before setting up this integration, ensure you have:

* A Telnyx account with voice capabilities
* A Pipecat Cloud account with a Telnyx WebSocket-compatible bot

A ready-to-build example of a Telnyx websockets bot with complete source code is available in the [pipecat-examples repo](https://github.com/pipecat-ai/pipecat-examples/tree/main/telnyx-chatbot).

## Pipecat Cloud Configuration

### 1. Get Your Organization Name

Retrieve your Pipecat Cloud organization name using the Pipecat CLI:

```bash theme={null}
$ pipecat cloud organizations list
```

This command will output a list of organizations associated with your account. For example:

```bash theme={null}
Organization        Name
──────────────────────────────────────
Default Workspace   three-random-words-randomnumber (active)
```

### 2. Create a TeXML Application

[Purchase a phone number from Telnyx](https://telnyx.com/resources/purchase-a-phone-number-with-telnyx) if you haven't already, then create a [TeXML Application](https://developers.telnyx.com/docs/voice/programmable-voice/texml-setup) with the following configuration:

```xml theme={null}
<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Connect>
    <Stream url="wss://api.pipecat.daily.co/ws/telnyx?serviceHost=AGENT_NAME.ORGANIZATION_NAME" bidirectionalMode="rtp"></Stream>
  </Connect>
  <Pause length="40"/>
</Response>
```

Replace the placeholder values:

* `AGENT_NAME` with your deployed bot's name (e.g., `my-first-agent`)
* `ORGANIZATION_NAME` with your organization name from step 1 (e.g., `three-random-words-randomnumber`)

For example, if your agent is named "customer-support" and your organization is "industrious-purple-cat-12345", your serviceHost would be: `customer-support.industrious-purple-cat-12345`

<Note>
  **Using Regional Endpoints**

  If you deployed your agent to a specific region, use the regional WebSocket endpoint to connect:

  `wss://{region}.api.pipecat.daily.co/ws/telnyx?serviceHost=AGENT_NAME.ORGANIZATION_NAME`

  For example, for Europe: `wss://eu-central.api.pipecat.daily.co/ws/telnyx?serviceHost=my-agent.my-org`

  Learn more about [regional endpoints](/deployment/pipecat-cloud/guides/regions#regional-websocket-endpoints).
</Note>

### 3. Assign TeXML Application to Your Phone Number

1. Navigate to Voice → Programmable Voice in your Telnyx dashboard
2. In the TeXML Applications tab, select the pencil icon for the TeXML Application you created in step 2
3. In the Numbers tab, select Assign numbers
4. Select the number you would like to assign the TeXML Application to
5. Click Save to apply your changes

## Testing Your Integration

To test your integration, simply dial your Telnyx phone number from any phone. The call will connect to your Pipecat Cloud bot, which will respond according to your bot's configuration.

## Next Steps

For complete implementation details including dial-out, caller personalization, custom parameters, and advanced call control features, see the [Telnyx WebSocket Integration guide](/guides/telephony/telnyx-websockets).


# Twilio Websocket Transport
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/telephony/twilio-websocket

Using Twilio's WebSocket Transport for your Pipecat Cloud agents

<Note>
  This guide covers Pipecat Cloud-specific configuration for Twilio WebSocket
  integration. For a complete guide including dial-in, dial-out, custom
  parameters, and advanced features, see the [Twilio WebSocket Integration
  guide](/guides/telephony/twilio-websockets).
</Note>

Native support for Twilio's WebSocket Transport with Pipecat Cloud allows you to connect your AI agents with Twilio's voice infrastructure. This integration enables your Pipecat bots to handle real phone calls using Twilio's Websockets.

## How It Works

Pipecat Cloud implements Twilio's bidirectional [Media Streams protocol](https://www.twilio.com/docs/voice/media-streams/websocket-messages). While audio streams flow through WebSockets, the call session is controlled by updating the Twilio Markup Language (TwiML) associated with each call's unique identifier (`CallSid`).

When Pipecat Cloud receives an incoming WebSocket connection from Twilio, it processes the `Connected` and `Start` messages to initialize a new bot instance. All WebSocket messages are forwarded to your bot, including any custom parameters set in your TwiML. This allows your bot to leverage Twilio's REST API for advanced call control - such as recording conversations, transferring to human agents, or implementing complex call flows.

<Frame>
  <img />
</Frame>

## Prerequisites

Before setting up this integration, ensure you have:

* A Twilio account with voice capabilities
* A Pipecat Cloud account with a Twilio Websockets-compatible bot

A ready-to-build example of a Twilio websockets bot with complete source code is available in the [pipecat-examples repo](https://github.com/pipecat-ai/pipecat-examples/tree/main/twilio-chatbot).

## Pipecat Cloud Configuration

### 1. Get Your Organization Name

Retrieve your Pipecat Cloud organization name using the Pipecat CLI:

```bash theme={null}
$ pipecat cloud organizations list
```

This command will output a list of organizations associated with your account. For example:

```bash theme={null}
Organization        Name
──────────────────────────────────────
Default Workspace   three-random-words-randomnumber (active)
```

### 2. Create a TwiML Bin

[Purchase a phone number from Twilio](https://help.twilio.com/articles/223135247-How-to-Search-for-and-Buy-a-Twilio-Phone-Number-from-Console) if you haven't already, then create a [TwiML Bin](https://help.twilio.com/articles/360043489573-Getting-started-with-TwiML-Bins) with the following configuration:

```xml theme={null}
<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Connect>
    <Stream url="wss://api.pipecat.daily.co/ws/twilio">
      <Parameter name="_pipecatCloudServiceHost"
         value="AGENT_NAME.ORGANIZATION_NAME"/>
    </Stream>
  </Connect>
</Response>
```

Replace the placeholder values:

* `AGENT_NAME` with your deployed bot's name (e.g., `my-first-agent`)
* `ORGANIZATION_NAME` with your organization name from step 1 (e.g., `three-random-words-randomnumber`)

For example, if your agent is named "customer-support" and your organization is "industrious-purple-cat-12345", your value would be: `customer-support.industrious-purple-cat-12345`

<Note>
  **Using Regional Endpoints**

  If you deployed your agent to a specific region, use the regional WebSocket endpoint to connect:

  `wss://{region}.api.pipecat.daily.co/ws/twilio`

  For example, for Europe: `wss://eu-central.api.pipecat.daily.co/ws/twilio`

  Learn more about [regional endpoints](/deployment/pipecat-cloud/guides/regions#regional-websocket-endpoints).
</Note>

### 3. Assign TwiML Bin to Your Phone Number

1. Navigate to the Phone Numbers section in your Twilio dashboard
2. Select your phone number from the list
3. In the Configure tab, under "Voice Configuration" section, find "A call comes in"
4. Set this dropdown to "TwiML Bin"
5. Select the TwiML Bin you created in step 2
6. Click Save to apply your changes

## Testing Your Integration

To test your integration, simply dial your Twilio phone number from any phone. The call will connect to your Pipecat Cloud bot, which will respond according to your bot's configuration.

## Next Steps

For complete implementation details including dial-out, custom parameters, caller personalization, and advanced call control features, see the [Twilio WebSocket Integration guide](/guides/telephony/twilio-websockets).


# Using Datadog
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/using-datadog

How to add Datadog logging, traces, and custom metrics to your Pipecat Cloud agents

This guide explains how to use [Datadog](https://www.datadoghq.com/) with your Pipecat Cloud agents for logging, traces, and custom metrics.

<Warning>
  This guide is a work in progress and may have some sharp edges. However, it
  has been vetted for basic usage.
</Warning>

## Prerequisites

* Have a Datadog account and API key.
* Use at least version `0.0.7` of the Pipecat Cloud base image:

```dockerfile theme={null}
# dailyco/pipecat-base:latest should also work
FROM dailyco/pipecat-base:0.0.7
```

## Add Datadog API key as a secret

If you're using keeping your secrets in an `.env` file [as recommended](../fundamentals/secrets), add your Datadog API key to that file:

```bash theme={null}
DD_API_KEY=<your-datadog-api-key>
```

Then update your secret set from that file:

```bash theme={null}
pipecat cloud secrets set <your-agent-secrets> --file .env
```

See [Secrets](../fundamentals/secrets) for more details.

## Create `datadog.yaml`

This file contains the Datadog Agent configuration.

The configuration shown below assumes you want all Datadog features covered in this guide (logging, traces, and custom metrics).

```yaml theme={null}
site: <your-datadog-site> # e.g. us5.datadoghq.com
hostname: pipecat.daily.co # (or whatever would be useful for you to see in the Datadog dashboards)

# Include the following line only if you need logging (see "Enabling Logging")
logs_enabled: true

# Uncomment the below lines if you DON'T need traces (see "Enabling Traces")
# apm_config:
#   enabled: false

# Uncomment the below lines if you DON'T need custom metrics (see "Enabling Custom Metrics")
# use_dogstatsd: false

kubelet_tls_verify: false
autoconfig_exclude_features:
  - kubernetes
  - orchestratorexplorer

```

## Create `pre-app.sh`

This script runs before the agent launcher runs.

The file shown below assumes you want all Datadog features covered in this guide (logging, traces, and custom metrics).

```bash theme={null}
#!/bin/bash

# Write the Datadog API key to the config file
echo "api_key: $DD_API_KEY" >> /etc/datadog-agent/datadog.yaml

# Start the Datadog Agent service
service datadog-agent start || true

# Include the below line only if you need traces (see "Enabling Traces")
# HACK: here we restart the Datadog Trace Agent specifically, since we know it failed to start above
service datadog-agent-trace restart || true

## DEBUGGING

# Check the status of the Datadog Agent subsystems for logging, custom metrics, and traces
# service datadog-agent status
# service datadog-agent-trace status
# datadog-agent status

# Display the Datadog agent log
# cat /var/log/datadog/agent.log
```

<Warning>
  You may see `Starting Datadog Agent: datadog-agent failed!` in your logs,
  triggered by `service datadog-agent start`. You can safely ignore this! As
  long as `service datadog-agent status` and `service datadog-agent-trace
      status` indicate that both of those services end up running, then we should
  have no trouble sending logs, traces, and custom metrics to Datadog.
</Warning>

## Update `Dockerfile` with the Datadog Agent

The snippet shown below assumes you want all Datadog features covered in this guide (logging, traces, and custom metrics).

```dockerfile theme={null}
# Install the Datadog Agent
# (note that you can literally use "dummy"—or anything else—as the API key)
RUN apt-get -y update; apt-get -y install curl
RUN DD_API_KEY=dummy \
    DD_INSTALL_ONLY=true \
    bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)"

# Configure Datadog Agent
COPY ./datadog.yaml /etc/datadog-agent
# Include the below line if you need logging (see "Enabling Logging")
COPY ./python.d /etc/datadog-agent/conf.d/python.d

# Include the below lines only if you need traces or custom metrics (see "Enabling Traces" or "Enabling Custom Metrics")
# Prepare file system for the Datadog Agent to use for Unix Sockets
RUN mkdir -p /var/run/datadog/
RUN chown dd-agent:dd-agent /var/run/datadog/

# Start Datadog Agent (move pre-app script to expected place where it'll be invoked by Pipecat Cloud base image)
COPY --chmod=755 ./pre-app.sh /app/
```

## Enabling logging

### Enable logging in `datadog.yaml`

Ensure the following is present in your [`datadog.yaml`](#create-datadog-yaml):

```yaml theme={null}
logs_enabled: true
```

### Create `python.d/conf.yaml`

Configure log collection for your agent:

```yaml theme={null}
# Use whatever agent name would be useful for you to see in the Datadog dashboard
logs:
  - type: file
    path: /var/log/<my-agent-name>/datadog.log
    service: <my-agent-name>
    source: python
    sourcecategory: sourcecode
```

### Update `Dockerfile` for logging

Ensure the following is present in your [`Dockerfile`](#update-dockerfile-with-the-datadog-agent):

```dockerfile theme={null}
COPY ./python.d /etc/datadog-agent/conf.d/python.d
```

This configures the Datadog Agent to collect logs from your agent.

### Configure `logger` in `bot.py`

Add a Datadog sink to your logger.

This sink will write log lines into a file that your Datadog Agent will read from, in a Datadog-friendly format.

```python theme={null}
import json

# Format log lines so that:
# - Datadog can parse and display log levels, messages, and timestamps
# - You can search Datadog logs by session_id
def datadog_format(record):
    format_string_safe_message = (
        json.dumps(record["message"]).replace("{", "{{").replace("}", "}}")
    )
    session_id = record["extra"].get("session_id", "")
    return (
        "{{"
        '"timestamp":"{time:YYYY-MM-DD HH:mm:ss.SSS}",'
        '"levelname":"{level}",'
        '"name":"{name}",'
        '"function":"{function}",'
        '"line":{line},'
        f'"message":{format_string_safe_message},'
        f'"session_id": "{session_id}"'
        "}}\n"
    )

async def bot(args: DailySessionArguments):
    # Add Datadog-specific sink
    logger.add(
        "/var/log/<my-agent-name>/datadog.log", # Use same path as in python.d/conf.yaml
        rotation="10 MB",
        retention="7 days",
        format=datadog_format,
    )
    # ...
```

## Enabling traces

### Enable traces in `datadog.yaml`

Ensure `apm_config.enabled` is NOT set to `false` in your [`datadog.yaml`](#create-datadog-yaml):

```yaml theme={null}
# DELETE or COMMENT OUT the following lines, if they're present
# apm_config:
#  enabled: false
```

### Start the Datadog Trace Agent in `pre-app.sh`

Ensure the following is present in your [`pre-app.sh`](#create-pre-app-sh):

```bash theme={null}
service datadog-agent-trace restart || true
```

### Prepare file system in `Dockerfile`

Ensure the following is present in your [`Dockerfile`](#update-dockerfile-with-the-datadog-agent):

```dockerfile theme={null}
RUN mkdir -p /var/run/datadog/
RUN chown dd-agent:dd-agent /var/run/datadog/
```

### Add `ddtrace` Python library

In your `requirements.txt`:

```
ddtrace
```

### Record traces in `bot.py`

```python theme={null}
from ddtrace import tracer

@tracer.wrap()
def my_function_call():
    # ...
```

## Enabling custom metrics

### Enable custom metrics in `datadog.yaml`

Ensure `use_dogstatsd` is NOT set to `false` in your [`datadog.yaml`](#create-datadog-yaml):

```yaml theme={null}
# DELETE or COMMENT OUT the following line, if it's present
# use_dogstatsd: false
```

### Prepare file system in `Dockerfile`

Ensure the following is present in your [`Dockerfile`](#update-dockerfile-with-the-datadog-agent):

```dockerfile theme={null}
RUN mkdir -p /var/run/datadog/
RUN chown dd-agent:dd-agent /var/run/datadog/
```

### Add `datadog` Python library

In your `requirements.txt`:

```
datadog
```

### Record custom metrics in `bot.py`

```python theme={null}
from datadog import statsd

def my_function_call():
    statsd.increment("myagent.my_function_call")
```


# WhatsApp Business Calling API
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/guides/whatsapp

Learn how to receive WhatsApp voice calls in your Pipecat Cloud application

## Overview

This guide explains how to integrate WhatsApp Business voice calling into your **Pipecat Cloud** application using the **WhatsApp Cloud API** and **SmallWebRTC**. Once configured, your Pipecat Cloud bot will be able to receive and handle real-time voice calls directly from WhatsApp users.

> For a more detailed explanation of WhatsApp Business setup, voice calling, and webhook configuration, refer to our [full WhatsApp Business Calling guide](/guides/features/whatsapp).

***

## Pipecat Cloud–Specific Setup

In Pipecat Cloud you only need to ensure your bot is deployed and your environment variables are configured correctly.

### Configure Webhook for Pipecat Cloud

Your webhook URL must point to your deployed Pipecat agent:

```
https://api.pipecat.daily.co/v1/public/webhooks/$ORGANIZATION_ID/$AGENT_NAME/whatsapp
```

* **Important:** Use your **public API key** as the `Verify Token` for Meta Developer Console verification.
* Always include the `/whatsapp` path at the end of the webhook URL.
* Make sure your Pipecat agent is **deployed and running**.

### Pipecat Cloud Secrets

In Pipecat Cloud, configure the following environment variables as **secrets**.

```bash theme={null}
WHATSAPP_TOKEN=<your_access_token>
WHATSAPP_PHONE_NUMBER_ID=<your_phone_number_id>
WHATSAPP_APP_SECRET=<your_app_secret>
```

If you have them in a `.env` file, you can upload to Pipecat Cloud as a secret set like this:

```bash theme={null}
uv run pipecat cloud secrets set whatsapp-secrets --file .env
```

This creates a secret set named `whatsapp-secrets` and securely stores your API keys for your Pipecat Cloud agent.

> For instructions on generating the access token, phone number ID, and app secret, refer to the [WhatsApp Cloud API Configuration](/guides/features/whatsapp#whatsapp-configuration).

### Example

Your bot must use `SmallWebRTCTransport` with audio enabled:

```python theme={null}
transport = SmallWebRTCTransport(
    webrtc_connection=webrtc_connection,
    params=TransportParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
    ),
)
```

## Full demo

<Card title="Demo" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/p2p-webrtc/pipecat-cloud">
  Pipecat Cloud WhatsApp Demo
</Card>

### Testing Calls

1. From WhatsApp, call your registered WhatsApp Business number.
2. The call will be routed to your Pipecat Cloud agent.
3. The bot should automatically answer and handle the conversation.

### Troubleshooting

Common Pipecat Cloud issues:

1. **Webhook Verification Failed**

   * Verify the public API key matches the verify token in Meta Developer Console.
   * Ensure the webhook URL is using the right organization and service names.

2. **Bot Not Answering Calls**
   * Ensure the WhatsApp Business number has calling enabled.
   * Ensure that the WhatsApp token is still valid.

> For a complete troubleshooting section, see [WhatsApp Business Calling guide](/guides/features/whatsapp#troubleshooting).


# Introduction
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/introduction

Deploy your AI agents to production at scale

Pipecat Cloud is a managed platform for hosting and scaling AI agents into production. Purpose-built for the Pipecat framework, Pipecat Cloud makes it easy for developers
to deploy Pipecat pipelines or custom agentic workflows.

<Frame>
  <img />
</Frame>

**Platform summary:**

* Built from the ground up for agentic use-cases and the Pipecat ecosystem
* Flexible configuration options for customizing your deployment
* Blazing fast availability and performance tuned for agentic use-cases
* Cross-platform client SDKs for frictionless application development
* Real-time monitoring and observability
* Cost effective and scalable
* Secure and compliant
* First-party integrations from best-in-class providers
* Pre-made templates for common agentic workflows

Pipecat Cloud is built by [Daily <Icon icon="arrow-up-right-from-square" />](https://daily.co), leveraging over a decade of experience building infrastructure
management and real-time voice and video capabilities.

***

<CardGroup>
  <Card title="Create your account" icon="user" href="https://pipecat.daily.co" />

  <Card title="Deploy your first agent" icon="cloud" href="/getting-started/quickstart" />
</CardGroup>


# Create an Agent
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/agent-create

POST /agents
Create a new agent with a container image and configuration settings.



# Delete an Agent
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/agent-delete

DELETE /agents/{agentName}
Permanently delete an agent and its associated resources.



# Retrieve Agent Logs
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/agent-get-logs

GET /agents/{agentName}/logs
Get execution logs for an agent with filtering and pagination options.



# Retrieve Agent Sessions
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/agent-get-sessions

GET /agents/{agentName}/sessions
Get sessions for an agent with filtering and pagination options.



# List All Agents
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/agent-list-all

GET /agents
Retrieve a list of all agents in your organization with their status and configuration.



# Get Agent Details
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/agent-list-one

GET /agents/{agentName}
Retrieve detailed information about a specific agent including its deployment status.



# Update an Agent
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/agent-update

POST /agents/{agentName}
Update an existing agent's configuration or deploy a new version.



# Get Organization Properties
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/properties-get

GET /properties
Retrieve current values of configurable properties for your organization.



# Update Organization Properties
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/properties-update

PATCH /properties
Update configurable properties for your organization, such as the default deployment region.



# List Regions
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/regions-list

GET /regions
Retrieve a list of all available regions for deploying agents and storing secrets.



# Create or Update Secrets
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/secret-create-update

PUT /secrets/{setName}
Create or update a secret set and its values



# Delete Entire Secret Set
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/secret-delete-secret-set

DELETE /secrets/{setName}
Delete an entire secret set by its name. This operation removes the secret set and all associated key-value pairs.



# Delete Specific Secret From Set
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/secret-delete-specific-secret

DELETE /secrets/{setName}/{secretKey}
Delete a specific secret from a set by its key. This operation removes the key-value pair from the specified secret set.



# List All Secret Sets
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/secret-list-all

GET /secrets
Retrieve a list of all secret sets in your organization with their name and secret type.



# Get Secret Set Details
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/secret-list-one

GET /secrets/{setName}
Retrieve key and value pairs for a specific secret set.



# Session API
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/session-proxy

GET /{serviceName}/sessions/{sessionId}/{path}
Send HTTP requests directly to your running Pipecat Cloud agent sessions.

Send HTTP requests to endpoints defined in your running bot. Supports `GET`, `POST`, `PUT`, `PATCH`, `DELETE`, `OPTIONS`, and `HEAD` methods.

## Request Headers

Headers are forwarded to your bot with these exceptions:

* `host` - Excluded
* `content-length` - Excluded
* `authorization` - Excluded (authentication is handled by the API gateway)

<Note>
  Requires base image version `0.1.2` or later. See the [Session API
  guide](/deployment/pipecat-cloud/guides/session-api) for setup instructions
  and examples.
</Note>


# Start an Agent session
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/start

POST /{agentName}/start
Start a new session with a deployed agent.

This endpoint starts a new instance of a deployed agent. You can optionally create a Daily room for the service to connect to.


# Stop an Agent session
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/rest-reference/endpoint/stop

DELETE /agents/{agentName}/sessions/{sessionId}
Stop a running agent session and clean up its resources.



# Examples
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/sdk-reference/examples

Common usage patterns for the Pipecat Cloud Python SDK

This page provides examples of common tasks with the Pipecat Cloud Python SDK.

## Starting an Agent Session

This example shows how to start a session with various configurations:

```python theme={null}
import asyncio

from pipecatcloud.exception import AgentStartError
from pipecatcloud.session import Session, SessionParams


async def main():
    try:
        # Create session object
        session = Session(
            agent_name="my-first-agent",
            api_key=API_KEY,  # Replace with your actual API key
            params=SessionParams(
                use_daily=True,  # Optional: Creates a Daily room
                daily_room_properties={"start_video_off": False},
                data={"key": "value"},
            ),
        )

        # Start the session
        response = await session.start()

        # Get Daily room URL
        daily_url = f"{response['dailyRoom']}?t={response['dailyToken']}"
        print(f"Join Daily room: {daily_url}")

    except AgentStartError as e:
        print(f"Error starting agent: {e}")
    except Exception as e:
        print(f"Unexpected error: {e}")


# Run the async function
if __name__ == "__main__":
    asyncio.run(main())
```

## Building a Bot Entry Point with Daily Arguments

```python theme={null}
from loguru import logger
from pipecat.runner.types import DailyRunnerArguments

async def bot(args: DailyRunnerArguments):
    """Main bot entry point compatible with the FastAPI route handler.

    Args:
        room_url: The Daily room URL
        token: The Daily room token
        body: The configuration object from the request body
    """
    logger.info(f"Bot process initialized {args.room_url} {args.token}")

    try:
        await main(args.room_url, args.token)
        logger.info("Bot process completed")
    except Exception as e:
        logger.exception(f"Error in bot process: {str(e)}")
        raise
```

## Building a Bot Entry Point with WebSocket Arguments

```python theme={null}
from loguru import logger
from pipecat.runner.types import WebSocketRunnerArguments

async def bot(args: WebSocketRunnerArguments):
    """Main bot entry point for WebSocket connections.

    Args:
        ws: The WebSocket connection
    """
    logger.info("WebSocket bot process initialized")

    try:
        await main(args.websocket)
        logger.info("WebSocket bot process completed")
    except Exception as e:
        logger.exception(f"Error in WebSocket bot process: {str(e)}")
        raise
```


# Error Handling
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/sdk-reference/exceptions

Managing errors with the Pipecat Cloud Python SDK

The Pipecat Cloud SDK defines several exception classes to help you handle different error conditions.

## Base Exception Class

### Error

Base class for all Pipecat Cloud exceptions.

```python theme={null}
from pipecatcloud.exception import Error

try:
    # SDK operation
except Error as e:
    print(f"A Pipecat Cloud error occurred: {e}")
```

## Session Errors

### AgentStartError

Raised when an agent fails to start.

```python theme={null}
from pipecatcloud.exception import AgentStartError

try:
    response = await session.start()
except AgentStartError as e:
    print(f"Failed to start agent: {e}")
    if e.error_code == "429":
        print("Agent pool at capacity. Try again later.")
    elif e.error_code == "404":
        print("Agent not found.")
```

#### Properties

<ParamField type="string">
  Error message with details about the failure.
</ParamField>

<ParamField type="string">
  Error code that can be used for conditional handling of different error types.
</ParamField>

### AgentNotHealthyError

Raised when attempting to interact with an agent that is not in a ready state.

```python theme={null}
from pipecatcloud.exception import AgentNotHealthyError

try:
    response = await session.start()
except AgentNotHealthyError as e:
    print(f"Agent is not ready: {e}")
    print("Check agent status with: pipecat cloud agent status my-agent")
```

#### Properties

<ParamField type="string">
  Error message with details about the agent's status.
</ParamField>

<ParamField type="string">
  Error code identifying the specific issue with the agent.
</ParamField>

## Authentication Errors

### AuthError

Raised when authentication fails or token has expired.

```python theme={null}
from pipecatcloud.exception import AuthError

try:
    # Operation requiring authentication
except AuthError:
    print("Your session has expired. Please log in again.")
    # Prompt user to reauthenticate
```

#### Properties

<ParamField type="string">
  Message explaining the authentication failure.
</ParamField>

## Configuration Errors

### ConfigError

Raised when there are issues with configuration storage or retrieval.

```python theme={null}
from pipecatcloud.exception import ConfigError

try:
    # Operation requiring config
except ConfigError as e:
    print(f"Configuration error: {e.message}")
    # Guide user to fix configuration
```

#### Properties

<ParamField type="string">
  Message explaining the configuration issue.
</ParamField>

### ConfigFileError

Raised when the configuration file is malformed.

```python theme={null}
from pipecatcloud.exception import ConfigFileError

try:
    # Operation requiring config file
except ConfigFileError:
    print("Your configuration file is invalid or corrupted.")
    print("Try recreating it with: pipecat cloud auth login")
```

### InvalidError

Raised when an invalid operation is attempted.

```python theme={null}
from pipecatcloud.exception import InvalidError

try:
    # Potentially invalid operation
except InvalidError as e:
    print(f"Invalid operation: {e}")
```


# Python SDK Overview
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/sdk-reference/overview

Introduction to the Pipecat Cloud Python SDK

The Pipecat Cloud Python SDK provides a programmatic interface for managing and interacting with your agents. It allows you to start and manage agent sessions, handle different session types, and respond to various error conditions.

## Installation

Install the SDK using pip:

```bash theme={null}
pip install pipecatcloud
```

## Key Components

The SDK contains several main components:

* [**Session Management**](./sessions) - Start and interact with agent sessions
* [**Session Arguments**](./session-arguments) - Types of arguments received by `bot()` entry points
* [**Error Handling**](./exceptions) - Exception classes for handling different error scenarios

## Quick Start

Here's a simple example to get started:

```python theme={null}
import asyncio

from pipecatcloud.exception import AgentStartError
from pipecatcloud.session import Session, SessionParams


async def main():
    try:
        # Create session object
        session = Session(
            agent_name="my-first-agent",
            api_key=API_KEY,  # Replace with your actual API key
            params=SessionParams(
                use_daily=True,  # Optional: Creates a Daily room
                daily_room_properties={"start_video_off": False},
                data={"key": "value"},
            ),
        )

        # Start the session
        response = await session.start()

        # Get Daily room URL
        daily_url = f"{response['dailyRoom']}?t={response['dailyToken']}"
        print(f"Join Daily room: {daily_url}")

    except AgentStartError as e:
        print(f"Error starting agent: {e}")
    except Exception as e:
        print(f"Unexpected error: {e}")


# Run the async function
if __name__ == "__main__":
    asyncio.run(main())
```

For more detailed examples and use cases, see the [Examples](./examples) section.


# Session Arguments
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/sdk-reference/session-arguments

Understanding the arguments received by bot entry points

When creating agents with Pipecat Cloud, your `bot()` entry point function receives different types of arguments depending on the session type. These classes represent the structure of those arguments.

## PipecatSessionArguments

Standard Pipecat Cloud agent session arguments, used for basic sessions.

```python theme={null}
from pipecatcloud.agent import PipecatSessionArguments

def bot(args: PipecatSessionArguments):
    print(f"Session ID: {args.session_id}")
    print(f"Custom data: {args.body}")
```

### Properties

<ParamField type="Optional[str]">
  The unique identifier for the current session.
</ParamField>

<ParamField type="Any">
  The custom data passed to the agent via the session parameters.
</ParamField>

## DailySessionArguments

Arguments for sessions that involve Daily WebRTC rooms for voice/video interaction.

```python theme={null}
from pipecat.runner.types import DailyRunnerArguments

def bot(args: DailyRunnerArguments):
    print(f"Session ID: {args.session_id}")
    print(f"Daily room URL: {args.room_url}")
    print(f"Daily token: {args.token}")
    print(f"Custom data: {args.body}")
```

### Properties

<ParamField type="Optional[str]">
  The unique identifier for the current session.
</ParamField>

<ParamField type="str">
  The URL for the Daily room.
</ParamField>

<ParamField type="str">
  The authentication token for the Daily room.
</ParamField>

<ParamField type="Any">
  The custom data passed to the agent via the session parameters.
</ParamField>

## WebSocketSessionArguments

Arguments for sessions that use WebSocket connections for real-time communication.

```python theme={null}
from pipecat.runner.types import WebSocketRunnerArguments

async def bot(args: WebSocketRunnerArguments):
    print(f"Session ID: {args.session_id}")
    await args.websocket.send_text("Hello from the agent!")
```

### Properties

<ParamField type="Optional[str]">
  The unique identifier for the current session.
</ParamField>

<ParamField type="WebSocket">
  The FastAPI WebSocket connection used to communicate with the client.
</ParamField>


# Session Management
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/sdk-reference/sessions

Managing agent sessions with the Pipecat Cloud Python SDK

The session management classes allow you to start and interact with agent sessions.

## Session

The `Session` class is the primary way to start and interact with agent sessions.

```python theme={null}
from pipecatcloud.session import Session, SessionParams

session = Session(
    agent_name="my-agent",
    api_key="pk_...",
    params=SessionParams(use_daily=True)
)
```

### Constructor Parameters

<ParamField type="string">
  Name of the deployed agent to interact with.
</ParamField>

<ParamField type="string">
  Public API key for authentication.
</ParamField>

<ParamField type="SessionParams">
  Optional parameters to configure the session.
</ParamField>

### Methods

<ResponseField name="start()" type="async">
  Starts a new session with the specified agent.

  **Returns**

  A dictionary containing session information. If `use_daily` is True, includes `dailyRoom` URL and `dailyToken`.

  **Raises**

  `AgentStartError`: If the session fails to start due to missing API key, agent not found, agent not ready, or capacity limits.
</ResponseField>

## SessionParams

The `SessionParams` class allows you to configure a session.

```python theme={null}
from pipecatcloud.session import SessionParams

params = SessionParams(
    data={"custom_field": "value"},
    use_daily=True,
    daily_room_properties={"enable_recording": "cloud"}
)
```

### Parameters

<ParamField type="Dict[str, Any]">
  Optional dictionary of data to pass to the agent. Must be JSON-serializable.
</ParamField>

<ParamField type="boolean">
  If True, creates a Daily WebRTC room for the session, enabling voice
  interaction.
</ParamField>

<ParamField type="Dict[str, Any]">
  Optional dictionary of properties to configure the Daily room. Only used when
  `use_daily=True`.

  See [Daily API
  documentation](https://docs.daily.co/reference/rest-api/rooms/config) for
  available properties.
</ParamField>


# HIPAA
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/security/hipaa

HIPAA compliance for Pipecat Cloud

Pipecat Cloud provides HIPAA-compliant infrastructure for healthcare applications that need to process Protected Health Information (PHI).

## Infrastructure Compliance

Pipecat Cloud's infrastructure meets HIPAA controls for:

* Data processing
* Data transmission
* Secure handling of PHI

## Data Logging with HIPAA Controls

When HIPAA controls are enabled for your account:

**No application data is logged** - This includes conversation history and any PHI that may be contained in your application logs.

**Rationale**: Pipecat follows the HIPAA "minimum necessary" rule by logging only what's essential to run the service, rather than storing your PHI data.

**Alternative logging**: Configure an independent logging solution to route your application logs to your own secure log storage system.

## Business Associate Agreement (BAA)

Daily can sign a Business Associate Agreement covering:

* Pipecat Cloud
* Daily WebRTC infrastructure
* Daily transcription services

A single BAA covers all these services.

## Getting Started

Contact us at [help@daily.co](mailto:help@daily.co) to:

* Enable HIPAA controls for your account
* Sign a Business Associate Agreement
* Learn more about HIPAA compliance features


# Security Guide
Source: https://docs.pipecat.ai/deployment/pipecat-cloud/security/security-and-compliance

Securing your agents on Pipecat Cloud

This guide addresses security considerations throughout your journey with Pipecat Cloud, from account creation to running production agents. Each section focuses on practical security measures you can take and explains how Pipecat Cloud protects your agents.

## Account & Organization Security

### How do I secure access to my Pipecat Cloud account?

Pipecat Cloud implements different authentication mechanisms depending on how you interact with the platform:

| Interface | Authentication Method          | Security Notes                              |
| --------- | ------------------------------ | ------------------------------------------- |
| Dashboard | Email + OTP                    | Two-factor authentication by default        |
| CLI       | JWT via Clerk                  | Token-based auth with automatic expiration  |
| REST API  | Bearer token (public API keys) | Scoped to specific organizations and agents |

<Tip>
  For CLI usage, we recommend running `pipecat cloud auth login` regularly to
  refresh your credentials.
</Tip>

### How do I manage access for my team?

Organizations provide a secure way to collaborate with team members:

* Create separate organizations for different teams or environments
* Invite members with appropriate permission levels
* Use organization-specific API keys for automation

Each organization has its own isolated environment with separate resources, agents, and secrets.

## Development Security

### How do I keep API keys and secrets secure during development?

Never include sensitive credentials in your agent code or images:

```python theme={null}
# DON'T do this
api_key = "sk_1234567890abcdef"  # Hardcoded keys are a security risk

# DO this instead
import os
api_key = os.environ.get("SERVICE_API_KEY")  # Use environment variables
```

During local development, use environment files (`.env`) and ensure they're in your `.gitignore`.

For deployment, use Pipecat Cloud's secrets management:

```bash theme={null}
# Create a secret set with your API keys
pipecat cloud secrets set my-agent-secrets \
  OPENAI_API_KEY=sk_... \
  ANTHROPIC_API_KEY=sk_...

# Reference in deployment
pipecat cloud deploy my-agent my-image:latest --secret-set my-agent-secrets
```

<Info>
  Secrets are encrypted at rest and in transit, then securely injected as
  environment variables in your agent's runtime.
</Info>

<Warning>
  When you update or rotate API keys in a secret set, you must redeploy your
  agent to apply these changes. Updated secrets are not automatically propagated
  to running deployments.
</Warning>

### Is my agent code secure during development?

Your local development environment and version control security are your responsibility. We recommend:

* Using private repositories
* Implementing branch protection
* Scanning code for security issues
* Never committing secrets or credentials

## Deployment Security

### How do I secure my agent container?

Follow these container security best practices:

1. Start with our [official base image](https://github.com/daily-co/pipecat-cloud-images/tree/main/pipecat-base) for a pre-hardened configuration
2. Include only necessary components to reduce attack surface
3. Keep dependencies updated to patch vulnerabilities
4. Use container scanning tools before deployment

### Are my private container images secure?

If you use private container registries, Pipecat Cloud supports secure image pulls:

```bash theme={null}
# Create an image pull secret for private registry
pipecat cloud secrets image-pull-secret dockerhub https://index.docker.io/v1/

# Alternatively, use base64 encoding for credentials
pipecat cloud secrets image-pull-secret dockerhub https://index.docker.io/v1/ --base64encode

# Use it during deployment
pipecat cloud deploy my-agent my-private-repo/image:latest --credentials dockerhub
```

Credentials are securely stored and only used during image pulls.

<Tip>
  The optional `--base64encode` flag provides an additional layer of obfuscation
  when entering credentials in environments where terminal history or session
  logging might be a concern.
</Tip>

### How does Pipecat Cloud protect my agent code?

Your deployed code remains private and protected through multiple layers:

* Containers run in isolated Kubernetes pods
* Network policies restrict cross-agent communication
* Each organization has separate namespaces
* Access controls prevent unauthorized access to running containers

## Runtime Security

### Can other customers access my agents or data?

Pipecat Cloud implements strong multi-tenant isolation:

* Organizational namespacing creates logical boundaries between customers
* Network policies prevent cross-namespace communication
* Kubernetes RBAC controls limit access to running pods
* Authentication is required for all API access

### How are my agents protected at runtime?

Pipecat Cloud provides several runtime security measures:

* **Network Controls**: Restrict egress to necessary services
* **Access Monitoring**: Track authentication and access patterns
* **Secret Injection**: Environment variables for secure credential access

### Who can start sessions with my agents?

Only clients with valid API keys for your organization can start agent sessions. For security best practices:

1. **Use server-side code** to start agent sessions, keeping API keys secure on your server
2. **Never expose API keys** in client-side code

```bash theme={null}
# Create a public API key for client applications
pipecat cloud organizations keys create

# Associate with specific agents
pipecat cloud organizations keys use
```

<Warning>
  While called "public API keys," these should still be treated as sensitive
  credentials and not shared with end-users or embedded in client-side code.
  They are "public" only in the sense that they have limited permissions
  compared to admin keys.
</Warning>

We recommend implementing a server-side endpoint that starts sessions on behalf of your users, similar to how you might handle other sensitive API operations.

## Data Security

### Is my data encrypted?

Pipecat Cloud encrypts your data throughout the platform:

* **In Transit**: TLS 1.3 encryption for all API communications
* **At Rest**: AES-256 encryption for stored data
* **Database**: Hardware-level encryption for database content, such as public API keys which developers control the rotation of

### Where is my data stored?

All Pipecat Cloud infrastructure is currently located in the US (AWS us-west-2 region). Contact us at `help@daily.co` if you have specific data residency requirements.

### What happens to my data when agents terminate?

Ephemeral data used during agent sessions is removed when the session ends. By default, stdout and stderr logs are stored in Pipecat Cloud. You have control over what is logged and where the logging is stored. Refer to [Logging & Observability](../fundamentals/logging) for more information.

## Monitoring & Incident Response

### How does Pipecat Cloud detect security issues?

We maintain comprehensive logging and monitoring:

* Administrative API calls are logged and retained for 1 year
* Authentication attempts and access patterns are monitored
* System events are tracked for security analysis

<Info>
  Request access to [Daily's Trust Center](https://trust.daily.co/) to learn
  more about our incident response and security practices.
</Info>

### How do I report security vulnerabilities?

If you discover a potential security vulnerability:

1. Email us at [disclosures@daily.co](mailto:disclosures@daily.co) (we'll acknowledge within one week)
2. Provide reasonable time for us to resolve the issue before disclosure
3. Make a good faith effort to avoid violating privacy or degrading service

<Note>
  Reporting security findings directly to Daily helps to ensure continued
  protection for all users.
</Note>

## Security Responsibilities

Security in cloud environments works best as a partnership. Pipecat Cloud follows a shared responsibility model where we secure the underlying infrastructure and platform, while you maintain responsibility for your application code and configuration. Understanding these boundaries helps ensure comprehensive protection for your agents and data.

### What security aspects does Pipecat Cloud handle?

Pipecat Cloud is responsible for:

* Infrastructure and platform security
* Network security and isolation
* Secret storage and injection
* Authentication systems
* Platform monitoring and logging

### What security aspects am I responsible for?

As a customer, you're responsible for:

* Application code security
* API key management
* Agent configuration security
* Following container security best practices
* Proper secrets management

<Tip>
  While Pipecat Cloud secures log storage, you should ensure your agent code
  doesn't log sensitive information like credentials, personal data, or
  proprietary content in stdout/stderr streams.
</Tip>

## Additional Resources

For security-related questions or to report security issues:

* **Security Issues**: Email [disclosures@daily.co](mailto:disclosures@daily.co) to report potential vulnerabilities
* **General Support**: Contact [help@daily.co](mailto:help@daily.co) for security questions


# Cerebrium
Source: https://docs.pipecat.ai/deployment/platforms/cerebrium

Deploy Pipecat applications to Cerebrium

[Cerebrium](https://www.cerebrium.ai) is a serverless Infrastructure platform that makes it easy for companies to build, deploy and scale AI applications.
Cerebrium offers both CPUs and GPUs (H100s, A100s etc) with extremely low cold start times allowing us to create highly performant applications in the most cost efficient manner.

### Install the Cerebrium CLI

To get started, let us run the following commands:

1. Run `pip install cerebrium` to install the Python package.
2. Run `cerebrium login` to authenticate yourself.

If you don't have a Cerebrium account, you can create one and get started with \$30 in free credits.

### Create a Cerebrium project

1. Create a new Cerebrium project:

```bash theme={null}
cerebrium init pipecat-agent
```

2. This will create two key files:

* `main.py` - Your application entrypoint
* `cerebrium.toml` - Configuration for build and environment settings

Update your `cerebrium.toml` with the necessary configuration:

```toml theme={null}
[cerebrium.hardware]
region = "us-east-1"
provider = "aws"
compute = "CPU"
cpu = 4
memory = 18.0

[cerebrium.dependencies.pip]
torch = ">=2.0.0"
"pipecat-ai[silero, daily, openai, cartesia]" = "latest"
aiohttp = "latest"
torchaudio = "latest"
```

In order for our application to work, we need to copy our API keys from the various platforms. Navigate to the Secrets section in your Cerebrium dashboard to store your API keys:

* `OPENAI_API_KEY` - We use OpenAI For the LLM. You can get your API key from [here](https://platform.openai.com/api-keys)
* `DAILY_TOKEN` - For WebRTC communication. You can get your token from [here](https://dashboard.daily.co/developers)
* `CARTERSIA_API_KEY` - For text-to-speech services. You can get your API key from [here](https://play.cartesia.ai/keys)

We access these secrets in our code as if they are normal ENV vars. For the above, You can swap in any LLM or TTS service you wish to use.

### Agent setup

We create a basic pipeline setup in our `main.py` that combines our LLM, TTS and Daily WebRTC transport layer.

```python theme={null}
import os
import sys
import time

import aiohttp
from loguru import logger

from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.audio.vad.vad_analyzer import VADParams
from pipecat.frames.frames import EndFrame, LLMMessagesAppendFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import LLMContextAggregatorPair
from pipecat.services.cartesia.tts import CartesiaTTSService
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.transports.daily.transport import DailyParams, DailyTransport

logger.remove(0)
logger.add(sys.stderr, level="DEBUG")


async def main(room_url: str, token: str):
    async with aiohttp.ClientSession() as session:
        transport = DailyTransport(
            room_url,
            token,
            "Friendly bot",
            DailyParams(
                audio_in_enabled=True,
                audio_out_enabled=True,
                transcription_enabled=True,
            ),
        )

        messages = [
            {
                "role": "system",
                "content": "You are a helpful AI assistant that can switch between two services to showcase the difference in performance and cost: 'openai_realtime' and 'custom'. Respond to user queries and switch services when asked.",
            },
        ]

        llm = OpenAILLMService(
            name="LLM",
            api_key=os.environ.get("OPENAI_API_KEY"),
            model="gpt-4",
        )

        tts = CartesiaTTSService(
            api_key=os.getenv("CARTESIA_API_KEY"),
            voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22",  # British Lady
        )

        vad_processor = VADProcessor(vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)))

        custom_context = LLMContext(messages)
        custom_user_aggregator, custom_assistant_aggregator = LLMContextAggregatorPair(context)
        context_aggregator_custom = LLMContextAggregatorPair(custom_context)

        pipeline = Pipeline(
            [
                transport.input(),  # Transport user input
                vad_processor,
                custom_user_aggregator,
                llm,
                tts,
                custom_assistant_aggregator,
                transport.output(),  # Transport bot output
            ]
        )

        task = PipelineTask(
            pipeline,
            params=PipelineParams(
                enable_metrics=True,
                enable_usage_metrics=True,
            ),
        )

        @transport.event_handler("on_first_participant_joined")
        async def on_first_participant_joined(transport, participant):
            transport.capture_participant_transcription(participant["id"])
            time.sleep(1.5)
            await task.queue_frame(
                LLMMessagesAppendFrame(
                    [
                        {
                            "role": "system",
                            "content": "Introduce yourself.",
                        }
                    ],
                    run_llm=True,
                )
            )

        @transport.event_handler("on_participant_left")
        async def on_participant_left(transport, participant, reason):
            await task.queue_frame(EndFrame())

        @transport.event_handler("on_call_state_updated")
        async def on_call_state_updated(transport, state):
            if state == "left":
                await task.queue_frame(EndFrame())

        runner = PipelineRunner()
        await runner.run(task)
        await session.close()
```

First, in our main function, we initialize the daily transport layer to receive/send the audio/video data from the Daily room we will connect to. You can see we pass the room\_url we would like to join as well as a token to authenticate us programmatically joining. We also set our VAD stop seconds which is the amount of time we wait for a pause before our bot will respond - in this example, we set it to 600 milliseconds.

Next we connect to our LLM (OpenAI) as well as our TTS model (Cartesia). By setting 'transcription\_enabled=true' we are using the STT from Daily itself. This is where the Pipecat framework helps convert audio data to text and vice versa.

We then put this all together as a PipelineTask which is what Pipecat runs all together. The make up of a task is completely customizable and has support for Image and Vision use cases. Lastly, we have some event handlers for when a user joins/leaves the room.

### Deploy bot

Deploy your application to Cerebrium:

```bash theme={null}
cerebrium deploy
```

You will then see that an endpoint is created for your bot at `POST \<BASE_URL\>/main` that you can call with your room\_url and token. Let us test it.

### Test it out

```python theme={null}
def create_room():
    url = "https://api.daily.co/v1/rooms/"
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {os.environ.get('DAILY_TOKEN')}",
    }
    data = {
        "properties": {
            "exp": int(time.time()) + 60 * 5,  ##5 mins
            "eject_at_room_exp": True,
        }
    }

    response = requests.post(url, headers=headers, json=data)
    if response.status_code == 200:
        room_info = response.json()
        token = create_token(room_info["name"])
        if token and "token" in token:
            room_info["token"] = token["token"]
        else:
            logger.error("Failed to create token")
            return {
                "message": "There was an error creating your room",
                "status_code": 500,
            }
        return room_info
    else:
        logger.error(f"Failed to create room: {response.status_code}")
        return {"message": "There was an error creating your room", "status_code": 500}


def create_token(room_name: str):
    url = "https://api.daily.co/v1/meeting-tokens"
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {os.environ.get('DAILY_TOKEN')}",
    }
    data = {"properties": {"room_name": room_name, "is_owner": True}}

    response = requests.post(url, headers=headers, json=data)
    if response.status_code == 200:
        token_info = response.json()
        return token_info
    else:
        logger.error(f"Failed to create token: {response.status_code}")
        return None

if __name__ == "__main__":
    room_info = create_room()
    print(f"Join room: {room_info["url"]}")
    asyncio.run(main(room_info["url"], room_info["token"]))

```

## Future Considerations

Since Cerebrium supports both CPU and GPU workloads if you would like to lower the latency of your application then the best would be to get model weights from various providers and run them locally.
You can do this for:

* LLM: Run any OpenSource model using a framework such as vLLM
* TTS: Both PlayHt and Deepgram offer TTS models that can be run locally
* STT: Deepgram offers a local model that can be run locally

If you implement all three models locally, you should have much better performance. We have been able to get \~300ms voice-to-voice responses.

## Examples

* [Fastest voice agent](https://docs.cerebrium.ai/v4/examples/realtime-voice-agents): Local only implementation
* [RAG voice agent](https://www.cerebrium.ai/blog/creating-a-realtime-rag-voice-agent): Create a voice agent that can do RAG using Cerebrium + OpenAI + Pinecone
* [Twilio voice agent](https://docs.cerebrium.ai/v4/examples/twilio-voice-agent): Create a voice agent that can receive phone calls via Twilio
* [OpenAI Realtime API implementation](https://www.cerebrium.ai/blog/an-alternative-to-openai-realtime-api-for-voice-capabilities): Create a voice agent that can receive phone calls via OpenAI Realtime API


# Fly.io
Source: https://docs.pipecat.ai/deployment/platforms/fly

Deploy Pipecat bots to Fly.io machines

## Project setup

Let's explore how we can use [fly.io](https://fly.io/) to make our app scalable for production by spawning our Pipecat bots on virtual machines with their own resources.

We mentioned [before](../pattern) that you would ideally containerize the `bot_runner.py` web service and the `bot.py` separately. To keep this example simple, we'll use the same container image for both services.

<Frame>
  <img alt="Fly.io Pipecat deployment" />
</Frame>

### Install the Fly CLI

You can find instructions for creating and setting up your fly account [here](https://fly.io/docs/getting-started/).

## Creating the Pipecat project

<Note>
  We have created a template project
  [here](https://github.com/pipecat-ai/pipecat-examples/tree/main/deployment/flyio-example)
  which you can clone. Since we're targeting production use-cases, this example
  uses Daily (WebRTC) as a transport, but you can configure your bot however you
  like.
</Note>

### Adding a fly.toml

Add a `fly.toml` to the root of your project directory. Here is a basic example:

```toml fly.toml theme={null}
app = 'some-unique-app-name'
primary_region = 'sjc'

[build]

[env]
FLY_APP_NAME = 'some-unique-app-name'

[http_service]
internal_port = 7860
force_https = true
auto_stop_machines = true
auto_start_machines = true
min_machines_running = 0
processes = ['app']

[[vm]]
memory = 512
cpu_kind = 'shared'
cpus = 1
```

For apps with lots of users, consider what resources your HTTP service will require to meet load. We'll define our `bot.py` resources later, so you can set and scale these as you like (`fly scale ...`)

### Environment setup

Our bot requires some API keys and configuration, so create a `.env` in your project root:

```shell .env theme={null}
DAILY_API_KEY=
OPENAI_API_KEY=
ELEVENLABS_API_KEY=
ELEVENLABS_VOICE_ID=
FLY_API_KEY=
FLY_APP_NAME=
```

Of course, the exact keys you need will depend on which services you are using within your `bot.py`.

<Warning>
  **Important:** your `FLY_APP_NAME` should match the name of your fly instance,
  such as that declared in your fly.toml.
</Warning>

The `.env` will allow us to test in local development, but is not included in the deployment. You'll need to set them as Fly app secrets, which you can do via the Fly dashboard or cli.

`fly secrets set ...`

## Containerize our app

Our Fly deployment will need a container image; let's create a simple `Dockerfile` in the root of the project:

<CodeGroup>
  ```shell Dockerfile theme={null}
  FROM python:3.11-slim-bookworm

  # Open port 7860 for http service

  ENV FAST_API_PORT=7860
  EXPOSE 7860

  # Install Python dependencies

  COPY \*.py .
  COPY ./requirements.txt requirements.txt
  RUN pip3 install --no-cache-dir --upgrade -r requirements.txt

  # Install models

  RUN python3 install_deps.py

  # Start the FastAPI server

  CMD python3 bot_runner.py --port ${FAST_API_PORT}

  ```

  ```shell .dockerignore theme={null}
  **/.DS_Store
  .env
  .env.*
  fly.toml
  ```
</CodeGroup>

<Note>You can use any base image as long as Python is available</Note>

Our container does the following:

* Opens port `7860` to serve our `bot_runner.py` FastAPI service.
* Downloads the necessary python dependencies.
* Download / cache the model dependencies the `bot.py` requires.
* Runs the `bot_runner.py` and listens for web requests.

### What models are we downloading?

To support voice activity detection, we're using Silero VAD. Whilst the filesize is not huge, having each new machine download the Silero model at runtime will impact bootup time. Instead, we include the model as part of the Docker image so it's cached and available.

You could, of course, also attach a network volume to each instance if you plan to include larger files as part of your deployment and don't want to bloat the size of your image.

## Launching new machines in `bot_runner.py`

When a user starts a session with our Pipecat bot, we want to launch a new machine on fly.io with it's own system resources.

Let's grab the bot\_runner.py from the example repo [here](https://github.com/pipecat-ai/pipecat/blob/jpt/deployment-examples/examples/deployment/flyio-example/bot_runner.py).

This runner differs from others in the Pipecat repo; we've added a new method that sends a REST request to Fly to provision a new machine for the session.

This method is invoked as part of the `/start_bot` endpoint:

```python bot_runner.py theme={null}
FLY_API_HOST = os.getenv("FLY_API_HOST", "https://api.machines.dev/v1")
FLY_APP_NAME = os.getenv("FLY_APP_NAME", "your-fly-app-name")
FLY_API_KEY = os.getenv("FLY_API_KEY", "")
FLY_HEADERS = {
    'Authorization': f"Bearer {FLY_API_KEY}",
    'Content-Type': 'application/json'
}

def spawn_fly_machine(room_url: str, token: str):
    # Use the same image as the bot runner
    res = requests.get(f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines", headers=FLY_HEADERS)
    if res.status_code != 200:
        raise Exception(f"Unable to get machine info from Fly: {res.text}")
    image = res.json()[0]['config']['image']

    # Machine configuration
    cmd = f"python3 bot.py -u {room_url} -t {token}"
    cmd = cmd.split()
    worker_props = {
        "config": {
            "image": image,
            "auto_destroy": True,
            "init": {
                "cmd": cmd
            },
            "restart": {
                "policy": "no"
            },
            "guest": {
                "cpu_kind": "shared",
                "cpus": 1,
                "memory_mb": 1024 # Note: 512 is just enough to run VAD, but 1gb is better
            }
        },

    }

    # Spawn a new machine instance
    res = requests.post(
        f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines",
        headers=FLY_HEADERS,
        json=worker_props)

    if res.status_code != 200:
        raise Exception(f"Problem starting a bot worker: {res.text}")

    # Wait for the machine to enter the started state
    vm_id = res.json()['id']

    res = requests.get(
        f"{FLY_API_HOST}/apps/{FLY_APP_NAME}/machines/{vm_id}/wait?state=started",
        headers=FLY_HEADERS)

    if res.status_code != 200:
        raise Exception(f"Bot was unable to enter started state: {res.text}")
```

We want to make sure the machine started ok before returning any data to the user. Fly launches machines pretty fast, but will timeout if things take longer than they should. Depending on your transport method, you may want to optimistically return a response to the user, so they can join the room and poll for the status of their bot.

## Launch the Fly project

Getting your bot on Fly is as simple as:

`fly launch` or `fly launch --org orgname` if you're part of a team.

This will step you through some configuration, and build and deploy your Docker image.

Be sure to configure your app secrets with the necessary environment variables once the deployment has complete.

Assuming all goes well, you can update with any changes with `fly deploy`.

### Test it out

Start a new bot instance by sending a `POST` request to `https://your-fly-url.fly.dev/start_bot`. All being well, this will return a room URL and token.

A nice feature of Fly is the ability to monitor your machines (with live logs) via their dashboard:

[https://fly.io/apps/YOUR-APP\_NAME/machines](https://fly.io/apps/YOUR-APP_NAME/machines)

This is really helpful for monitoring the status of your spawned machine, and debugging if things do not work as expected.

<Note>
  This example is configured to expire after 5 minutes. The bot process is also
  configured to exit after the user leaves the room. This is a good way to
  ensure we don't have any hanging VMs, although you'll likely need to configure
  this behaviour this to meet your own needs.

  You'll also notice that we set `restart policy` to `no`. This prevents the machine attempting to restart after the session has concluded and the process exits.
</Note>

***

## Important considerations

This example does little in the way of load balancing or app security. Indeed, a user can spawn a new machine on your account simply by sending a `POST` request to the `bot_runner.py`. Be sure to configure a maximum number of instances, or authenticate requests to avoid costs getting out of control.

We also deployed our `bot.py` on a machine with the same image as our `bot_runner.py`. To optimize container file sizes and increase security, consider individual images that only deploy resources they require.


# Modal
Source: https://docs.pipecat.ai/deployment/platforms/modal

Deploy Pipecat applications to Modal

[Modal](https://www.modal.com) is well-suited for Pipecat deployments because it handles container orchestration, scaling, and cold starts efficiently. This makes it a good choice for production Pipecat bots that need reliable performance.

This guide walks through the Modal example included in the Pipecat repository, which follows the same [deployment pattern](../pattern).

<Card title="Modal example" icon="github" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/deployment/modal-example">
  View the complete Modal deployment example in our GitHub repository
</Card>

<Frame>
  <img
    alt="Diagram of the deployment architecture of Modal and Pipecat
example"
  />
</Frame>

## Install the Modal CLI

<Card title="Set up Modal" icon="rocket" href="https://modal.com/docs/guide#getting-started">
  Follow Modal's official instructions for creating an account and setting up
  the CLI
</Card>

## Deploy a self-serve LLM

1. Deploy Modal's OpenAI-compatible LLM service:

   ```bash theme={null}
   git clone https://github.com/modal-labs/modal-examples
   cd modal-examples
   modal deploy 06_gpu_and_ml/llm-serving/vllm_inference.py
   ```

<Info>
  Refer to Modal's guide and example for [Deploying an OpenAI-compatible LLM
  service with vLLM](https://modal.com/docs/examples/vllm_inference) for more
  details.
</Info>

2. Take note of the endpoint URL from the previous step, which will look like:
   ```
   https://{your-workspace}--example-vllm-openai-compatible-serve.modal.run
   ```
   You'll need this for the `bot_vllm.py` file in the next section.

<Note>
  The default Modal LLM example uses Llama-3.1 and will shut down after 15
  minutes of inactivity. Cold starts take 5-10 minutes. To prepare the service,
  we recommend visiting the `/docs` endpoint (`https://<Modal workspace>--example-vllm-openai-compatible-serve.modal.run/docs`)
  for your deployed LLM and wait for it to fully load before connecting your client.
</Note>

## Deploy FastAPI App and Pipecat pipeline to Modal

1. Setup environment variables:

```bash theme={null}
cd server
cp env.example .env
# Modify .env to provide your service API Keys
```

<Info>
  Alternatively, you can configure your Modal app to use
  [secrets](https://modal.com/docs/guide/secrets).
</Info>

2. Update the `modal_url` in `server/src/bot_vllm.py` to point to the URL you received from the self-serve LLM deployment in the previous step.

3. From within the `server` directory, test the app locally:

```bash theme={null}
modal serve app.py
```

4. Deploy to production:

```bash theme={null}
modal deploy app.py
```

5. Note the endpoint URL produced from this deployment. It will look like:

```bash theme={null}
https://{your-workspace}--pipecat-modal-fastapi-app.modal.run
```

You'll need this URL for the client's `app.js` configuration mentioned in its README.

## Launch your bots on Modal

### Option 1: Direct Link

Simply click on the URL displayed after running the server or deploy step to launch an agent and be redirected to a Daily room to talk with the launched bot. This will use the OpenAI pipeline.

### Option 2: Connect via an RTVI Client

Follow the instructions provided in the [client folder's README](https://github.com/pipecat-ai/pipecat-examples/tree/main/deployment/modal-example/client/javascript) for building and running a custom client that connects to your Modal endpoint. The provided client includes a dropdown for choosing which bot pipeline to run.

## Navigating your LLM, server, and Pipecat logs

On your [Modal dashboard](https://modal.com/apps), you should have two Apps listed under Live Apps:

1. `example-vllm-openai-compatible`: This App contains the containers and logs used to run your self-hosted LLM. There will be just one App Function listed: `serve`. Click on this function to view logs for your LLM.
2. `pipecat-modal`: This App contains the containers and logs used to run your `connect` endpoints and Pipecat pipelines. It will list two App Functions:
   1. `fastapi_app`: This function is running the endpoints that your client will interact with and initiate starting a new pipeline (`/`, `/connect`, `/status`). Click on this function to see logs for each endpoint hit.
   2. `bot_runner`: This function handles launching and running a bot pipeline. Click on this function to get a list of all pipeline runs and access each run's logs.

## Modal & Pipecat Tips

* In most other Pipecat examples, we use `Popen` to launch the pipeline process from the `/connect` endpoint. In this example, we use a Modal function instead. This allows us to run the pipelines using a separately defined Modal image as well as run each pipeline in an isolated container.
* For the FastAPI and most common Pipecat Pipeline containers, a default `debian_slim` CPU-only should be all that's required to run. GPU containers are needed for self-hosted services.
* To minimize cold starts of the pipeline and reduce latency for users, set `min_containers=1` on the Modal Function that launches the pipeline to ensure at least one warm instance of your function is always available.

## Next steps

<Card title="Explore Modal's LLM Examples" icon="layer-group" href="https://modal.com/docs/examples/vllm_inference">
  For next steps on running a self-hosted LLM and reducing latency, check out
  all of Modal's LLM examples
</Card>


# Pipecat Cloud
Source: https://docs.pipecat.ai/deployment/platforms/pipecat-cloud

Deploy Pipecat agents with managed infrastructure

[Pipecat Cloud](https://pipecat.daily.co) is a managed platform for hosting and scaling Pipecat agents in production.

<Card title="Get Started with Pipecat Cloud" icon="cloud" href="../pipecat-cloud/introduction">
  Visit the complete Pipecat Cloud documentation to deploy your first agent
</Card>


# Examples
Source: https://docs.pipecat.ai/examples

Complete applications and quickstart demos to accelerate your Pipecat development

<Update label={<><Icon icon="rocket" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Quickstart</span></>}>
  Quickstart demo for a basic voice bot.

  [View Example →](https://github.com/pipecat-ai/pipecat-quickstart)
</Update>

<Update label={<><Icon icon="laptop-mobile" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Simple Chatbot</span></>}>
  A full client-server example, showing how to connect different clients with a Pipecat bot. Client SDK examples include JavaScript, React, React Native, iOS, and Android.

  [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/simple-chatbot)
</Update>

<Update label={<><Icon icon="phone" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Twilio Chatbot</span></>}>
  Twilio websocket + Media streams. Inbound and outbound calling.

  [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/twilio-chatbot)
</Update>

<Update label={<><Icon icon="phone" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Telnyx Chatbot</span></>}>
  Telnyx websocket + Media streams. Inbound and outbound calling.

  [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/telnyx-chatbot)
</Update>

<Update label={<><Icon icon="phone" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Plivo Chatbot</span></>}>
  Plivo websocket + Media streams. Inbound and outbound calling.

  [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/plivo-chatbot)
</Update>

<Update label={<><Icon icon="phone" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Exotel Chatbot</span></>}>
  Exotel websocket + Media streams. Inbound and outbound calling.

  [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/exotel-chatbot)
</Update>

<Update label={<><Icon icon="phone" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Daily PSTN & SIP Calling</span></>}>
  Add PSTN & SIP calling to your Pipecat bot. Examples include Daily PSTN and Daily + Twilio SIP. Learn how to place inbound and outbound calls, as well as how to cold transfer a call.

  [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot)
</Update>

<Update label={<><Icon icon="phone" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>PSTN Warm Transfer</span></>}>
  Warm transfer a PSTN call to a human agent using Daily PSTN.

  [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-pstn-warm-transfer)
</Update>

<Update label={<><Icon icon="whatsapp" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>WhatsApp Chatbot</span></>}>
  Add WhatsApp WebRTC calling to your Pipecat bot.

  [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/whatsapp)
</Update>

<Update label={<><Icon icon="route" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>OpenTelemetry</span></>}>
  Add OpenTelemetry to your Pipecat bot. Examples include Langfuse and Jaeger.

  [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/open-telemetry)
</Update>

<Update label={<><Icon icon="arrows-turn-to-dots" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>IVR Navigation</span></>}>
  Automatically navigate an IVR call tree by providing Pipecat with a goal.

  [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/ivr-navigation)
</Update>

<Update label={<><Icon icon="voicemail" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Voicemail Detection</span></>}>
  Detect voicemail in a phone call and leave a message.

  [View Example →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/44-voicemail-detection.py)
</Update>

<Update label={<><Icon icon="video" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Tavus</span></>}>
  Use Tavus to create a video avatar for your Pipecat bot.

  [View Example →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/21a-tavus-video-service.py)
</Update>

<Update label={<><Icon icon="video" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>HeyGen</span></>}>
  Use HeyGen to create a video avatar for your Pipecat bot.

  [View Example →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/43a-heygen-video-service.py)
</Update>

<Update label={<><Icon icon="video" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Simli</span></>}>
  Use Simli to create a video avatar for your Pipecat bot.

  [View Example →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/27-simli-layer.py)
</Update>

<Update label={<><Icon icon="tower-broadcast" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Push to Talk</span></>}>
  A push-to-talk client interface, allowing users to press a button to start talking to the bot.

  [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/push-to-talk)
</Update>

<Update label={<><Icon icon="docker" size={24} color="currentColor" /><span style={{marginLeft: '8px'}}>SmallWebRTC + Docker</span></>}>
  Learn how to deploy the SmallWebRTCTransport in a Docker container.

  [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/p2p-webrtc/docker)
</Update>

<Update label={<><Icon icon="list-check" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Structured Conversation</span></>}>
  Learn how to use Pipecat Flows to create a structured navigation flow. This example shows a simple restaurant reservation flow.

  [View Example →](https://github.com/pipecat-ai/pipecat-flows/blob/main/examples/restaurant_reservation.py)
</Update>

<Update label={<><Icon icon="chart-mixed" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Sentry Metrics</span></>}>
  Learn how to use Sentry to track metrics and errors in your Pipecat bot.

  [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/sentry-metrics)
</Update>

<Update label={<><Icon icon="video" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Vonage Audio Connector</span></>}>
  Stream real-time audio from an active Vonage Video API session into a Pipecat pipeline using the Vonage Audio Connector.

  [View Example →](https://github.com/pipecat-ai/pipecat-examples/tree/main/vonage-audio-bot)
</Update>

***

## 💡 Need Help Getting Started?

<CardGroup>
  <Card title="More Examples" icon="github" href="https://github.com/pipecat-ai/pipecat-examples">
    Browse all examples on GitHub
  </Card>

  <Card title="Recipes" icon="book" href="/recipes">
    Browse code snippets & techniques
  </Card>

  <Card title="Discord Community" icon="discord" href="https://discord.gg/pipecat">
    Get help from the community
  </Card>
</CardGroup>


# AI-Assisted Development
Source: https://docs.pipecat.ai/getting-started/ai-tools

Use AI coding tools with Pipecat documentation and source code.

Pipecat offers several ways to give AI coding tools access to documentation and source code context, helping you build voice and multimodal applications faster.

## MCP Server

The Pipecat docs MCP server lets AI coding tools search across Pipecat documentation directly. This gives your AI assistant up-to-date knowledge about Pipecat APIs, services, and patterns.

### Claude Code

```bash theme={null}
claude mcp add --transport http pipecat-docs https://daily-docs.mcp.kapa.ai
```

### Codex

```bash theme={null}
codex --mcp-config '{"pipecat-docs":{"url":"https://daily-docs.mcp.kapa.ai"}}'
```

### Cursor

<a href="cursor://anysphere.cursor-deeplink/mcp/install?name=pipecat-docs&config=eyJ1cmwiOiJodHRwczovL2RhaWx5LWRvY3MubWNwLmthcGEuYWkifQ%3D%3D">
  <img alt="Add Pipecat MCP to Cursor" />
</a>

Or manually add to your Cursor MCP config:

```json theme={null}
{
  "mcpServers": {
    "pipecat-docs": {
      "url": "https://daily-docs.mcp.kapa.ai"
    }
  }
}
```

### VS Code

<a href="vscode:mcp/install?%7B%22name%22%3A%22pipecat-docs%22%2C%22type%22%3A%22http%22%2C%22url%22%3A%22https%3A%2F%2Fdaily-docs.mcp.kapa.ai%22%7D">
  <img alt="Add Pipecat MCP to VS Code" />
</a>

Or manually add to your VS Code MCP config (`.vscode/mcp.json`):

```json theme={null}
{
  "servers": {
    "pipecat-docs": {
      "url": "https://daily-docs.mcp.kapa.ai"
    }
  }
}
```

## CLAUDE.md

The Pipecat repository includes a [`CLAUDE.md`](https://github.com/pipecat-ai/pipecat/blob/main/CLAUDE.md) file that provides Claude Code with architecture guidance, coding conventions, and development patterns specific to the Pipecat codebase. When you work in a cloned copy of the Pipecat repo, Claude Code automatically reads this file for context.

## Markdown docs

Every page on this docs site has a **Copy page** button (in the top-right area) that copies the page content as markdown. This is useful for pasting into any AI chat or coding tool to provide context about a specific topic.

## llms.txt

Pipecat docs support the [llms.txt standard](https://llmstxt.org/), which provides a machine-readable index of documentation content. This is useful for tools that can ingest documentation in bulk.

* [`https://docs.pipecat.ai/llms.txt`](https://docs.pipecat.ai/llms.txt) — structured index of all pages
* [`https://docs.pipecat.ai/llms-full.txt`](https://docs.pipecat.ai/llms-full.txt) — full documentation content in a single file


# Build Your Next Bot
Source: https://docs.pipecat.ai/getting-started/build-your-next-bot

Scaffold a new Pipecat project with the CLI

Ready to build your own bot? The Pipecat CLI scaffolds complete projects tailored to your platform and use case.

<Note>
  **New to Pipecat?** We recommend completing the
  [Quickstart](/getting-started/quickstart) first to understand Pipecat basics
  before scaffolding your own project.
</Note>

## Install the Pipecat CLI

Install the CLI globally using [uv](https://docs.astral.sh/uv/) or [pipx](https://pipx.pypa.io/):

```bash theme={null}
uv tool install pipecat-ai-cli
# or
pipx install pipecat-ai-cli
```

Verify installation:

```bash theme={null}
pipecat --version
```

## Create Your Project

Run the interactive setup wizard:

```bash theme={null}
pipecat init
```

The CLI will guide you through selecting your platform (phone or web/mobile), transport provider, AI services (STT, LLM, TTS), and deployment target.

<Card title="CLI Reference" icon="terminal" href="/cli/init">
  See detailed options and generated project structure
</Card>

## Next Steps

Once `pipecat init` completes, the CLI will provide specific instructions for your generated project, including:

* How to configure your API keys
* Installing dependencies
* Running your bot locally
* Customizing the bot logic and behavior

Follow the instructions displayed by the CLI to get your bot up and running!

## Deploy to Production

Ready to deploy your bot? Choose between managed cloud hosting or self-hosted infrastructure.

<CardGroup>
  <Card title="Pipecat Cloud" icon="cloud" href="/deployment/pipecat-cloud/introduction">
    Deploy and manage your agents with the CLI - purpose-built for Pipecat
  </Card>

  <Card title="Self-Hosted Options" icon="server" href="/deployment/overview">
    Deploy to Fly.io, Modal, AWS, or your own infrastructure
  </Card>
</CardGroup>

## Learn More

<CardGroup>
  <Card title="Core Concepts" icon="book-open" href="/guides/learn/overview">
    Understand pipelines, processors, and transports
  </Card>

  <Card title="Browse Examples" icon="code" href="https://github.com/pipecat-ai/pipecat-examples">
    30+ production-ready examples for inspiration
  </Card>
</CardGroup>


# Ecosystem
Source: https://docs.pipecat.ai/getting-started/ecosystem

Client SDKs, tools, and AI services in the Pipecat ecosystem

Pipecat provides a complete ecosystem of client SDKs, developer tools, and AI service integrations to build voice and multimodal applications for any platform.

## Client SDKs

The client SDKs connect users to your bot via mobile or web applications. They handle real-time audio transport (using Daily WebRTC, SmallWebRTC, etc.) and provide built-in messaging and events to build responsive voice AI interfaces—tracking when your bot is speaking, what it's saying, and more.

Build client applications for any platform with our official SDKs:

<CardGroup>
  <Card title="JavaScript" icon="JS" href="/client/js/introduction">
    Build web applications with vanilla JavaScript
  </Card>

  <Card title="React" icon="react" href="/client/react/introduction">
    Create React applications with hooks and components
  </Card>

  <Card title="React Native" icon="react" href="/client/react-native/introduction">
    Build mobile apps with React Native
  </Card>

  <Card title="Swift" icon="swift" href="/client/ios/introduction">
    Native iOS applications
  </Card>

  <Card title="Kotlin" icon="android" href="/client/android/introduction">
    Native Android applications
  </Card>

  <Card title="C++" icon="C" href="/client/c++/introduction">
    High-performance native applications
  </Card>
</CardGroup>

## Tools & Frameworks

### Pipecat CLI

Command-line tool for scaffolding new projects, deploying to Pipecat Cloud, and monitoring live sessions.

<Card title="CLI Overview" icon="terminal" href="/cli/overview">
  Learn about scaffolding, deployment, and monitoring commands
</Card>

### Pipecat Flows

A conversation and context management library designed to help you handle complex, structured conversations with state management.

<Card title="Pipecat Flows" icon="diagram-project" href="https://github.com/pipecat-ai/pipecat-flows">
  Build structured conversation flows
</Card>

### Voice UI Kit

Pre-built React components and custom hooks that accelerate building voice AI applications on top of the JavaScript and React SDKs.

<Card title="Voice UI Kit" icon="palette" href="https://github.com/pipecat-ai/voice-ui-kit">
  Ready-to-use components for voice interfaces
</Card>

### Tail

A terminal dashboard for monitoring your Pipecat sessions in real-time with logs, conversations, metrics, and audio levels.

<Card title="Tail" icon="chart-line" href="https://github.com/pipecat-ai/tail">
  Real-time monitoring dashboard
</Card>

## AI Services

Pipecat is a modular system where you can plug in different AI services without changing your code. This flexibility is one of Pipecat's core strengths—swap Deepgram for Azure Speech, OpenAI for Anthropic, or Cartesia for ElevenLabs with minimal changes.

The framework integrates with 70+ services across:

* **Speech-to-Text (STT)** - Deepgram, Azure, Google, Whisper, and more
* **Language Models (LLM)** - OpenAI, Anthropic, Gemini, Groq, and more
* **Text-to-Speech (TTS)** - Cartesia, ElevenLabs, Azure, Google, and more
* **Vision** - Image and video analysis capabilities
* **Transports** - WebRTC, WebSockets, telephony providers
* **And more** - Image generation, memory systems, analytics

<Card title="View All Services" icon="grid" href="/server/services/supported-services">
  Browse the complete list of 70+ supported AI services
</Card>


# Introduction
Source: https://docs.pipecat.ai/getting-started/introduction

Learn about Pipecat and how to get started.

Pipecat is an open source Python framework for building voice and multimodal AI bots that can see, hear, and speak in real-time.

The framework orchestrates AI services, network transports, and audio processing to enable ultra-low latency conversations that feel natural and responsive. Build everything from simple voice assistants to complex multimodal applications that combine audio, video, images, and text.

Want to dive right in? Check out the Quickstart example to run your first Pipecat application.

<Card title="Quickstart" icon="rocket" href="/getting-started/quickstart">
  Build and run your first Pipecat application
</Card>

## What You Can Build

<CardGroup>
  <Card title="Voice Assistants" icon="microphone" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/simple-chatbot">
    Natural, real-time conversations with AI using speech recognition and
    synthesis
  </Card>

  <Card title="Phone Agents" icon="phone" href="https://github.com/pipecat-ai/pipecat-flows/blob/main/examples/patient_intake.py">
    Connect to your agent via phone for support, intake, and customer service
    interactions
  </Card>

  <Card title="Multimodal Apps" icon="layer-group" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/12-describe-video.py">
    Applications that combine voice, video, images, and text for rich interactions
  </Card>

  <Card title="Creative Experiences" icon="wand-magic-sparkles" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/storytelling-chatbot">
    Storytelling experiences and social companions that engage users
  </Card>

  <Card title="Interactive Games" icon="gamepad" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/word-wrangler-gemini-live">
    Voice-controlled games and interactive experiences with real-time AI responses
  </Card>

  <Card title="Conversation Flows" icon="diagram-project" href="https://github.com/pipecat-ai/pipecat-flows">
    Build structured conversations with Pipecat Flows to complete tasks and improve LLM accuracy
  </Card>
</CardGroup>

## How It Works

Pipecat orchestrates AI services in a **pipeline**, which is a series of processors that handle real-time audio, text, and video frames with ultra-low latency.

Here's what happens in a typical voice conversation:

1. **Transport** receives audio from the user (browser, phone, etc.)
2. **Speech Recognition** converts speech to text in real-time
3. **LLM** generates intelligent responses based on context
4. **Speech Synthesis** converts responses back to natural speech
5. **Transport** streams audio back to the user

In most cases, the entire round-trip interaction happens between 500-800ms, creating a natural conversation experience for the user.

The diagram below shows a typical voice assistant pipeline, where each step happens in real-time:

<img alt="Pipecat Overview" />

## Ready to Build?

The best way to understand Pipecat is to build with it. Start with our 5-minute quickstart to create your first voice AI bot.

<CardGroup>
  <Card title="Quickstart" icon="rocket" href="/getting-started/quickstart">
    Build and run your first Pipecat application
  </Card>

  <Card title="Explore the Ecosystem" icon="grid" href="/getting-started/ecosystem">
    Browse client SDKs, tools, and AI services
  </Card>
</CardGroup>

## Get Involved

<CardGroup>
  <Card title="Discord Community" icon="discord" href="https://discord.gg/pipecat">
    Connect with other developers, share your projects, and get support from the
    Pipecat team.
  </Card>

  <Card title="Pipecat GitHub repo" icon="github" href="https://github.com/pipecat-ai/pipecat">
    Explore the source code, open issues, and contribute to the project.
  </Card>
</CardGroup>


# Continue Learning
Source: https://docs.pipecat.ai/getting-started/next-steps

Continue your Pipecat journey

Now that you've run your first Pipecat bot, here's how to continue learning and build your own applications.

## Choose Your Path

<CardGroup>
  <Card title="Learn Core Concepts" icon="book-open" href="/guides/learn/overview">
    Understand pipelines, processors, transports, and how to build custom AI
    applications from the ground up.
  </Card>

  <Card title="Explore Examples" icon="code" href="https://github.com/pipecat-ai/pipecat-examples">
    Dive into complete examples including multimodal bots, creative
    applications, and enterprise integrations.
  </Card>
</CardGroup>

## What's Your Goal?

Choose your path based on what you want to accomplish:

### 🚀 Build a Production Application

Ready to create your own bot? Use the CLI to scaffold a project tailored to your needs.

<Card title="Build Your Next Bot" icon="wand-magic-sparkles" href="/getting-started/build-your-next-bot">
  Scaffold phone or web/mobile projects with the Pipecat CLI
</Card>

### 🧠 Understand How Pipecat Works

Master the fundamentals to build custom solutions and debug effectively.

<CardGroup>
  <Card title="Core Concepts" icon="book-open" href="/guides/learn/overview">
    Learn about pipelines, processors, transports, and context management
  </Card>

  <Card title="Common Features" icon="puzzle-piece" href="/guides/fundamentals/custom-frame-processor">
    Function calling, audio recording, transcripts, and custom processors
  </Card>
</CardGroup>

### 💡 Get Inspired

Explore real-world examples and recipes for common use cases.

<CardGroup>
  <Card title="Browse Examples" icon="code" href="https://github.com/pipecat-ai/pipecat-examples">
    30+ production-ready examples including multimodal bots and games
  </Card>

  <Card title="Recipes" icon="book-open" href="/recipes">
    Common patterns and solutions for specific use cases
  </Card>
</CardGroup>

### 🚀 Deploy to Production

Scale your bot with managed hosting or self-hosted infrastructure.

<Card title="Deployment Options" icon="rocket" href="/deployment/overview">
  Compare Pipecat Cloud, Fly.io, Modal, and other platforms
</Card>

## Need Help?

<CardGroup>
  <Card title="Discord Community" icon="discord" href="https://discord.gg/pipecat">
    Get support and share projects with other developers
  </Card>

  <Card title="GitHub Issues" icon="github" href="https://github.com/pipecat-ai/pipecat/issues">
    Report bugs or request features
  </Card>
</CardGroup>


# Quickstart
Source: https://docs.pipecat.ai/getting-started/quickstart

Run your first Pipecat bot in under 5 minutes

This quickstart guide will help you build and deploy your first Pipecat voice AI bot. You'll create a simple conversational agent that you can talk to in real-time, then deploy it to production on Pipecat Cloud.

**Two steps**: Local Development (5 min) → Production Deployment (5 min)

## Step 1: Local Development

### Prerequisites

**Environment**

* Python 3.10 or later
* [uv](https://docs.astral.sh/uv/getting-started/installation/) package manager installed

**AI Service API Keys**

This quickstart uses three AI services working together in a pipeline. You'll need API keys from each service:

<CardGroup>
  <Card title="Deepgram (STT)" icon="microphone" href="https://console.deepgram.com/signup">
    Create an account and generate your API key for real-time
    speech recognition.
  </Card>

  <Card title="OpenAI (LLM)" icon="brain" href="https://auth.openai.com/create-account">
    Create an account and generate an API key for intelligent conversation
    responses.
  </Card>

  <Card title="Cartesia (TTS)" icon="volume-high" href="https://play.cartesia.ai/sign-up">
    Sign up and generate your API key for natural voice
    synthesis.
  </Card>
</CardGroup>

Have these API keys ready. You'll add them to your environment file in the next section.

### Setup

1. Clone the quickstart repository

```bash theme={null}
git clone https://github.com/pipecat-ai/pipecat-quickstart.git
cd pipecat-quickstart
```

2. Configure your API keys

Create your environment file:

```bash theme={null}
cp env.example .env
```

Open the `.env` file in your text editor and add your API keys:

```ini theme={null}
DEEPGRAM_API_KEY=your_deepgram_api_key
OPENAI_API_KEY=your_openai_api_key
CARTESIA_API_KEY=your_cartesia_api_key
```

3. Set up virtual environment and install dependencies

```bash theme={null}
uv sync
```

### Run your bot locally

Now you're ready to run your bot! Start it with

```bash theme={null}
uv run bot.py
```

You should see output similar to this:

```
🚀 WebRTC server starting at http://localhost:7860/client
   Open this URL in your browser to connect!
```

Open [http://localhost:7860/client](http://localhost:7860/client) in your browser and click **Connect** to start talking to your bot.

<Note>
  **First run note**: The initial startup may take \~20 seconds as Pipecat
  downloads required models and imports. Subsequent runs will be much faster.
</Note>

🎉 **Success!** Your bot is running locally. Now let's deploy it to production so others can use it.

***

## Step 2: Deploy to Production

Transform your local bot into a production-ready service. Pipecat Cloud handles scaling, monitoring, and global deployment.

### Prerequisites

1. Sign up for Pipecat Cloud

[Create your Pipecat Cloud account](https://pipecat.daily.co/sign-up) to deploy and manage your bots.

2. Set up Docker

* **Install [Docker](https://www.docker.com/)** on your system
* **Create a [Docker Hub](https://hub.docker.com/) account**
* **Login to Docker Hub:**

```bash theme={null}
docker login
```

3. Pipecat CLI

The Pipecat CLI is already installed with your quickstart project.

Log in to Pipecat Cloud using the CLI to get started:

```bash theme={null}
uv run pipecat cloud auth login
```

You'll be presented with a link that you can click to authenticate your client.

### Configure your deployment

The `pcc-deploy.toml` file tells Pipecat Cloud how to run your bot. **Update the image field** with your Docker Hub username:

```toml theme={null}
agent_name = "quickstart"
image = "YOUR_DOCKERHUB_USERNAME/quickstart:0.1"  # 👈 Update this line
secret_set = "quickstart-secrets"

[scaling]
	min_agents = 1
```

**Understanding the configuration:**

* `agent_name`: Your bot's name in Pipecat Cloud
* `image`: The Docker image to deploy (format: `username/image:version`)
* `secret_set`: Where your API keys are stored securely
* `min_agents`: Number of bot instances to keep ready (1 = instant start)

<Tip>
  [Set up
  `image_credentials`](/deployment/pipecat-cloud/fundamentals/secrets#image-pull-secrets)
  in your TOML file for authenticated image pulls.
</Tip>

### Configure secrets

Upload your API keys to Pipecat Cloud's secure storage:

```bash theme={null}
uv run pipecat cloud secrets set quickstart-secrets --file .env
```

This creates a secret set called `quickstart-secrets` (matching your TOML file) and uploads all your API keys from `.env`.

### Build and deploy

1. Build and push your Docker image

```bash theme={null}
uv run pipecat cloud docker build-push
```

This builds your Docker image and pushes it to Docker Hub based on the `image` information in your `pcc-deploy.toml` file.

2. Deploy to Pipecat Cloud

```bash theme={null}
uv run pipecat cloud deploy
```

### Connect to your agent

1. Open your [Pipecat Cloud dashboard](https://pipecat.daily.co/)
2. Select your `quickstart` agent → **Sandbox**
3. Allow microphone access and click **Connect**

🎉 **Your bot is now live in production!**

<Card title="Ready to scale?" icon="rocket" href="/deployment/pipecat-cloud/introduction">
  Explore advanced Pipecat Cloud features like scaling, monitoring, secrets
  management, and production best practices.
</Card>

***

## Understanding the Quickstart Bot

When you speak to your bot, here's the real-time pipeline that processes your conversation:

1. **Audio Capture**: Your browser captures microphone audio and sends it via WebRTC
2. **Voice Activity Detection**: Silero VAD detects when you start and stop speaking
3. **Speech Recognition**: Deepgram converts your speech to text in real-time
4. **Language Processing**: OpenAI's GPT model generates an intelligent response
5. **Speech Synthesis**: Cartesia converts the response text back to natural speech
6. **Audio Playback**: The generated audio streams back to your browser

Each step happens with minimal latency, typically completing the full round-trip in under one second.

### AI Services

Your bot uses three AI services, each configured with API keys from your `.env` file:

```python theme={null}
# Create AI Services
stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    voice_id="71a7ad14-091c-4e8e-a314-022ece01c121",
)
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"))
```

Pipecat supports many different AI services. You can swap out Deepgram for Azure Speech, OpenAI for Anthropic, or Cartesia for ElevenLabs without changing the rest of your code.

See the [supported services documentation](/server/services/supported-services) for all available options.

### Context and Messages

Your bot maintains conversation history using a context object, enabling multi-turn interactions where the bot remembers what was said earlier.

The context is initialized with a system message that defines the bot's personality:

```python theme={null}
# All messages use the OpenAI message format
messages = [
    {
        "role": "system",
        "content": "You are a friendly AI assistant. Respond naturally and keep your answers conversational.",
    },
]

context = LLMContext(messages)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
    ),
)
```

The context aggregator automatically collects user messages (after speech-to-text) and assistant responses (after text-to-speech), maintaining the conversation flow without manual intervention.

### RTVI Protocol

When building web or mobile clients, you can use [Pipecat's client SDKs](/client/introduction) that communicate with your bot via the [RTVI (Real-Time Voice Interaction) protocol](/client/rtvi-standard). Pipecat comes with RTVI enabled by default, allowing your the Pipecat server and client to exchange messages and events in real-time.

### Pipeline Configuration

The core of your bot is a Pipeline that processes data through a series of processors:

```python theme={null}
# Create the pipeline with the processors
pipeline = Pipeline([
    transport.input(),     # Receive audio from browser
    stt,                   # Speech-to-text (Deepgram)
    user_aggregator,       # Add user message to context
    llm,                   # Language model (OpenAI)
    tts,                   # Text-to-speech (Cartesia)
    transport.output(),    # Send audio back to browser
    assistant_aggregator,  # Add bot response to context
])
```

Data flows through the pipeline as "frames", objects containing audio, text, or other data types. The ordering is crucial: audio must be transcribed before it can be processed by the LLM, and text must be synthesized before it can be played back.

The pipeline is managed by a PipelineTask:

```python theme={null}
# Create a PipelineTask to manage the pipeline execution
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        enable_metrics=True,
        enable_usage_metrics=True,
    ),
)
```

The task handles pipeline execution, collects metrics, and manages RTVI events through [observers](/server/utilities/observers/observer-pattern).

### Event Handlers

Event handlers manage the bot's lifecycle and user interactions:

```python theme={null}
# Event handler for when a client connects
@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
    logger.info(f"Client connected")
    # Add a greeting message to the context
    messages.append({"role": "system", "content": "Say hello and briefly introduce yourself."})
    # Prompt the bot to start talking when the client connects
    await task.queue_frames([LLMRunFrame()])

# Event handler for when a client disconnects
@transport.event_handler("on_client_disconnected")
async def on_client_disconnected(transport, client):
    logger.info(f"Client disconnected")
    # Cancel the task when the client disconnects
    # This stops the pipeline and all processors, cleaning up resources
    await task.cancel()
```

When a client connects, the bot adds a greeting instruction and queues a context frame to initiate the conversation. When disconnecting, it properly cancels the task to clean up resources.

### Running the Pipeline

Finally, the pipeline is executed by a PipelineRunner:

```python theme={null}
# Create a PipelineRunner to run the task
runner = PipelineRunner(handle_sigint=False)

# Finally, run the task using the runner
# This will start the pipeline and begin processing frames
await runner.run(task)
```

The runner manages the pipeline's execution lifecycle. Note that `handle_sigint=False` because the main runner handles system signals.

### Bot Entry Point

The quickstart uses Pipecat's runner system:

```python theme={null}
async def bot(runner_args: RunnerArguments):
    """Main bot entry point."""

    # Configure transport parameters for different environments
    transport_params = {
        "daily": lambda: DailyParams(
            audio_in_enabled=True,
            audio_out_enabled=True,
        ),
        "webrtc": lambda: TransportParams(
            audio_in_enabled=True,
            audio_out_enabled=True,
        ),
    }

    transport = await create_transport(runner_args, transport_params)
    await run_bot(transport, runner_args)

if __name__ == "__main__":
    from pipecat.runner.run import main
    main()
```

This runner automatically handles WebRTC connection setup and management, making it easy to get started with minimal configuration. The same code works for both local development and production deployment.

<Tip>
  **Production ready**: This bot pattern is fully compatible with Pipecat Cloud,
  meaning you can deploy your bot without any code changes.
</Tip>

## Troubleshooting

* **Browser permissions**: Make sure to allow microphone access when prompted by your browser.
* **Connection issues**: If the WebRTC connection fails, first try a different browser. If that fails, make sure you don't have a VPN or firewall rules blocking traffic. WebRTC uses UDP to communicate.
* **Audio issues**: Check that your microphone and speakers are working and not muted.

## Next Steps

Congratulations! You've built and deployed your first Pipecat bot. Here's what to do next based on your goals:

### 🚀 Ready to Build Your Own Application?

The quickstart gave you a working example, but the Pipecat CLI helps you scaffold production-ready projects with your choice of platform (phone vs web/mobile), transport providers, and AI services—all tailored to your specific use case.

<Card title="Build Your Next Bot" icon="rocket" href="/getting-started/build-your-next-bot">
  Use the CLI to scaffold phone or web/mobile projects customized for your needs
</Card>

### 🧠 Want to Understand How It Works?

Dive deeper into Pipecat's architecture and learn how to build custom solutions.

<CardGroup>
  <Card title="Learn Core Concepts" icon="book-open" href="/guides/learn/overview">
    Master pipelines, processors, transports, and context management
  </Card>

  <Card title="Browse Examples" icon="code" href="https://github.com/pipecat-ai/pipecat-examples">
    30+ production-ready examples for inspiration
  </Card>
</CardGroup>


# Building with Gemini Live
Source: https://docs.pipecat.ai/guides/features/gemini-live

Create real-time voice AI agents using Google's Gemini Live API and Pipecat

Gemini Live is Google's speech-to-speech API that enables natural, real-time voice conversations with AI. With Pipecat, you can build production-ready voice agents that leverage Gemini Live for telephony, web, and mobile applications.

<CardGroup>
  <Card title="API Reference" icon="book" href="/server/services/s2s/gemini-live">
    Gemini Live service documentation
  </Card>

  <Card title="Pipecat CLI" icon="terminal" href="/cli/overview">
    Scaffold and deploy projects
  </Card>
</CardGroup>

## Capabilities

Pipecat's Gemini Live integration supports multiple modalities and deployment targets:

<CardGroup>
  <Card title="Voice" icon="microphone">
    Real-time speech-to-speech conversations with natural turn-taking and voice activity detection
  </Card>

  <Card title="Vision" icon="eye">
    Process video and screenshare alongside audio for multimodal interactions
  </Card>

  <Card title="Telephony" icon="phone">
    Build phone-based voice agents with Twilio WebSocket integration
  </Card>

  <Card title="Tool Use" icon="wrench">
    Function calling support for external integrations and dynamic responses
  </Card>
</CardGroup>

### Architecture

Pipecat manages connections between your client and Gemini Live:

<Frame>
  <img alt="Gemini Live Architecture" />
</Frame>

The Pipecat server handles media streaming with clients via WebRTC (web/mobile) or WebSockets (telephony), while maintaining a persistent connection to Gemini Live for real-time AI processing.

## Quick Start

The fastest way to start building is with the Pipecat CLI:

```bash theme={null}
# Install the CLI
uv tool install pipecat-ai-cli

# Create a new project
pipecat init
```

The CLI will guide you through selecting:

* **Bot type**: Gemini Live (speech-to-speech)
* **Transport**: Daily WebRTC, Twilio, or others
* **Deployment target**: Local development or Pipecat Cloud

<Tip>
  All CLI commands can use either `pipecat` or the shorter `pc` alias.
</Tip>

## Starter Projects

These complete examples demonstrate Gemini Live in production scenarios. Each includes local development setup and Pipecat Cloud deployment configuration.

### Phone Bot (Twilio)

A telephone-based voice agent using Gemini Live with Twilio WebSockets. The demo plays "Two Truths and a Lie" to showcase natural conversation flow.

<Card title="Phone Bot Starter" icon="phone" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/gemini-live-starters/phone-bot">
  Build a production phone agent with Twilio integration
</Card>

**Try it now**: Call **1-970-LIVE-API** (1-970-548-3274) to talk to a live demo.

**What you'll learn**:

* Twilio WebSocket transport configuration
* Google STT/TTS integration alongside Gemini Live
* TwiML setup for incoming calls
* Pipecat Cloud deployment with telephony

### Web Bot (Vision)

A browser-based agent with screensharing and vision capabilities, built with the Pipecat Voice UI Kit and Daily WebRTC transport.

<Card title="Web Bot Starter" icon="browser" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/gemini-live-starters/web-bot">
  Build a web agent with vision and screensharing
</Card>

**What you'll learn**:

* Daily WebRTC transport for web clients
* Vision/screenshare processing with Gemini Live
* Next.js client with Voice UI Kit components
* Resizable panels and event logging

## Deployment

Both starter projects include configuration for [Pipecat Cloud](/deployment/pipecat-cloud/introduction), which handles scaling, monitoring, and global deployment.

```bash theme={null}
# Authenticate with Pipecat Cloud
pipecat cloud auth login

# Build and push your Docker image
pipecat cloud docker build-push

# Deploy your agent
pipecat cloud deploy
```

Each starter includes a `pcc-deploy.toml` file with sensible defaults for agent configuration and scaling.

<Card title="Pipecat Cloud Deployment" icon="cloud" href="/deployment/pipecat-cloud/fundamentals/deploy">
  Learn more about deploying to production
</Card>

## Next Steps

<CardGroup>
  <Card title="Function Calling" icon="code" href="/guides/learn/function-calling">
    Add external integrations and dynamic responses
  </Card>

  <Card title="React Client SDK" icon="react" href="/client/react/introduction">
    Build custom web interfaces
  </Card>

  <Card title="Telephony Guide" icon="phone-volume" href="/deployment/pipecat-cloud/guides/telephony/twilio-websocket">
    Deep dive into phone integrations
  </Card>

  <Card title="Core Concepts" icon="graduation-cap" href="/guides/learn/overview">
    Understand Pipecat pipelines and processors
  </Card>
</CardGroup>


# Noise cancellation with Krisp VIVA
Source: https://docs.pipecat.ai/guides/features/krisp-viva

Learn how to integrate Krisp's VIVA noise cancellation model into your Pipecat application

## Overview

This guide will walk you through setting up and using Krisp's VIVA noise cancellation models to filter out background noise and voices from the user's audio input audio stream. These models yield clearer audio, meaning fewer false interruptions and better transcription.

<CardGroup>
  <Card title="Krisp VIVA API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.audio.filters.krisp_viva_filter.html">
    Pipecat's API reference for Krisp VIVA noise cancellation
  </Card>

  <Card title="Krisp VIVA Example" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07p-interruptible-krisp-viva.py">
    Complete example with Krisp VIVA noise cancellation
  </Card>
</CardGroup>

## Prerequisites

To complete this setup, you will need access to a Krisp developers account, where you can download the Python SDK and corresponding VIVA models.

<Tip>
  Get started on the [Krisp developers website](https://krisp.ai/developers).
</Tip>

## Setup

### Download the Python SDK and Models

1. Log in to the [Krisp developer portal](https://sdk.krisp.ai/)
2. Navigate to the `Server SDK Version` Tab
3. Find the latest version of the Python SDK:
   * Download the SDK
   * Download the Server Side Noise Cancellation models

### Install the Python wheel file

1. First, unzip the SDK files you downloaded in the previous step. In the unzipped folder, you will find a `dist` folder containing the Python wheel file you will need to install.
2. Install the Python wheel file that corresponds to your platform. For example, a macOS ARM64 platform running Python 3.12 would install the following:

   ```bash theme={null}
   uv pip install /PATH_TO_DOWNLOADED_SDK/krisp-audio-sdk-python-1.4.0/dist/krisp_audio-1.4.0-cp312-cp312-macosx_12_0_arm64.whl
   ```

### Set up the environment variable

1. First, unzip the models you downloaded in the first step. The folder contains a number of models. You will want to use either:

   * `krisp-viva-pro`: Mobile, Desktop, Browser (WebRTC, up to 32kHz)
   * `krisp-viva-tel`: Telephony, Cellular, Landline, Mobile, Desktop, Browser (up to 16kHz)

   Note: the full model name will be in the format of `krisp-viva-pro-v1.kef`.

2. In your .env file, add a `KRISP_VIVA_FILTER_MODEL_PATH` environment variable which points to the path to the VIVA model you wish to use.

```bash theme={null}
KRISP_VIVA_FILTER_MODEL_PATH=/PATH_TO_UNZIPPED_MODELS/krisp-viva-models-9.9/krisp-viva-pro-v1.kef
```

## Test the integration

You're ready to test the integration! Try running the [Krisp VIVA foundation example](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07p-interruptible-krisp-viva.py).

<Tip>
  Learn how to [run foundational
  examples](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/README.md)
  in Pipecat.
</Tip>

## Usage Example

The `KrispVivaFilter()` is the audio filter available in Pipecat that uses the Krisp VIVA noise cancellation model. This filter can be added to any transport via the `audio_in_filter` parameter.

For example, here's how to add the `KrispVivaFilter()` to a Daily transport:

```python theme={null}
from pipecat.audio.filters.krisp_viva_filter import KrispVivaFilter
from pipecat.transports.daily.transport import DailyParams, DailyTransport

# Add to transport configuration
transport = DailyTransport(
    room_url,
    token,
    "Audio Bot",
    DailyParams(
        audio_in_enabled=True,
        audio_in_filter=KrispVivaFilter(),  # Enable Krisp noise reduction
        audio_out_enabled=True,
    ),
)
```


# Building With OpenAI Audio Models and APIs
Source: https://docs.pipecat.ai/guides/features/openai-audio-models-and-apis

Create voice agents with OpenAI audio models and Pipecat

This guide provides an overview of the audio capabilities OpenAI offers via their APIs. We'll also link to Pipecat sample code.

## Two Ways To Build Voice-to-voice

You can build voice-to-voice applications in two ways:

1. The cascaded models approach, using separate models for transcription, the LLM, and voice generation.

<Frame>
  <img alt="OpenAI Cascaded Pipeline" />
</Frame>

A cascaded pipeline looks like this, in Pipecat code. Here's a [single-file example that uses a cascaded pipeline](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07g-interruptible-openai.py). (See below for an overview of Pipecat core concepts.)

```python theme={null}
pipeline = Pipeline(
    [
        transport.input(),
        speech_to_text,
        context_aggregator.user(),
        llm,
        text_to_speech,
        context_aggregator.assistant(),
        transport.output(),
    ]
)
```

2. Using a single, speech-to-speech model. This is conceptually much simpler. Though note that most applications also need to implement things like function calling, retrieval-augmented search, context management, and integration with existing systems. So the core pipeline is only part of an app's complexity.

<Frame>
  <img alt="OpenAI S2S Pipeline" />
</Frame>

Here's a speech-to-speech pipeline in Pipecat code. And here's a [single-file example that uses the OpenAI Realtime API](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/19-openai-realtime-beta.py).

```python theme={null}
pipeline = Pipeline(
    [
        transport.input(),
        context_aggregator.user(),
        speech_to_speech_llm,
        context_aggregator.assistant(),
        transport.output(),
    ]
)
```

Which approach should you choose?

* The cascaded models approach is preferable if you are implementing a complex workflow and need the best possible instruction following performance and function calling reliability. The `gpt-4o` model operating in text-to-text mode has the strongest instruction following and function calling performance.
* The speech-to-speech approach offers better audio understanding and human-like voice output. If your application is primarily free-form, open-ended conversation, these attributes might be more important than instruction following and function calling performance. Note also that `gpt-4o-audio-preview` and the OpenAI Realtime API are currently beta products.

## OpenAI Audio Models and APIs

### Transcription API

* Models: `gpt-4o-transcribe`, `gpt-4o-mini-transcribe`
* Pipecat service: `OpenAISTTService` ([reference docs](/server/services/stt/openai))
* OpenAI endpoint: `/v1/audio/transcriptions` ([docs](https://platform.openai.com/docs/api-reference/audio/createTranscription))

### Chat Completions API

* Models: `gpt-4o`, `gpt-4o-mini`, `gpt-4o-audio-preview`
* Pipecat service: `OpenAILLMService` ([reference docs](/server/services/llm/openai))
* OpenAI endpoint: `/v1/chat/completions` ([docs](https://platform.openai.com/docs/api-reference/chat))

### Realtime API

* Models: `gpt-4o-realtime-preview`, `gpt-4o-mini-realtime-preview`
* Pipecat service: `OpenAIRealtimeBetaLLMService` ([reference docs](/server/services/s2s/openai))
* OpenAI docs ([overview](https://platform.openai.com/docs/guides/realtime))

### Speech API

* Models: `gpt-4o-mini-tts`
* Pipecat service: `OpenAITTSService` ([reference docs](/server/services/tts/openai))
* OpenAI endpoint: `/v1/audio/speech` ([docs](https://platform.openai.com/docs/api-reference/audio/createSpeech))

## Sample code and starter kits

*If you have a code example or starter kit you would like this doc to link to, please let us know. We can add examples that help people get started with the OpenAI audio models and APIs.*

### Single-file examples

<CardGroup>
  <Card title="OpenAI STT → LLM → TTS" icon="code" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07g-interruptible-openai.py">
    A complete implementation demonstrating the cascaded approach with OpenAI services
  </Card>

  <Card title="OpenAI Realtime API" icon="tower-broadcast" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/19-openai-realtime-beta.py">
    A speech-to-speech implementation using OpenAI's Realtime API
  </Card>
</CardGroup>

### OpenAI + Twilio + Pipecat Cloud

[This starter kit](https://github.com/daily-co/pcc-openai-twilio/) is a complete telephone voice agent that can talk about the NCAA March Madness basketball tournaments and look up realtime game information using function calls.

<Frame>
  <img alt="OpenAI Twilio" />
</Frame>

The starter kit includes two bot configurations: cascaded model and speech-to-speech. The code can be packaged for deployment to Pipecat Cloud, a commercial platform for Pipecat agent hosting.


# Pipecat Flows
Source: https://docs.pipecat.ai/guides/features/pipecat-flows

Learn how to create structured conversations using Pipecat's flow system

## What is Pipecat Flows?

Pipecat Flows is an add-on framework for Pipecat that allows you to build structured conversations in your AI applications. It enables you to define conversation paths while handling the complexities of state management and LLM interactions.

Want to dive right in? Check out these examples:

<CardGroup>
  <Card title="Hello World" icon="rocket" href="https://github.com/pipecat-ai/pipecat-flows/tree/main/examples/quickstart">
    A great first Flow to show you the ropes
  </Card>

  <Card title="Restaurant Reservation" icon="utensils" href="https://github.com/pipecat-ai/pipecat-flows/blob/main/examples/restaurant_reservation.py">
    A simple, practical Flow example
  </Card>
</CardGroup>

## How do Pipecat and Pipecat Flows work together?

**Pipecat** defines the core capabilities of your bot. This includes the pipeline and processors which, at a minimum, enable your bot to:

* Receive audio from a user
* Transcribe the user's input
* Run an LLM completion
* Convert the LLM response to audio
* Send audio back to the user

**Pipecat Flows** complements Pipecat's core functionality by providing structure to a conversation, managing context and tools as the conversation progresses from one state to another. This is separate from the core pipeline, allowing you to separate conversation logic from core pipeline mechanics.

## When to Use Pipecat Flows?

Pipecat Flows is best suited for use cases where:

* **You need precise control** over how a conversation progresses through specific steps
* **Your bot handles complex tasks** that can be broken down into smaller, manageable pieces
* **You want to improve LLM accuracy** by focusing the model on one specific task at a time instead of managing multiple responsibilities simultaneously

This approach addresses a common problem: traditional methods often use large, monolithic prompts with many tools available at once, leading to hallucinations and lower accuracy.

Pipecat Flows solves this by:

* **Breaking complex tasks into focused steps** - Each node has a clear, single purpose
* **Providing relevant tools only when needed** - Functions are available only in the appropriate context
* **Giving clear, specific instructions** - Task messages focus the LLM on exactly what to do next

## Selecting a Flows Pattern

Pipecat Flows provides two general patterns for how to build your application: Dynamic Flows and Static Flows.

* **Dynamic Flows (recommended)**: When conversation paths need to be determined at runtime based on user input, external data, or business logic.
* **Static Flows**: When your conversation structure is known upfront and follows predefined paths.

<Note>
  Dynamic Flows can handle both simple and complex use cases. Selecting it
  provides you with an option that can grow with the complexity of your
  application. For these reasons, we strongly recommend it as the API pattern to
  follow.
</Note>

## Technical Overview

Pipecat Flows represents a conversation as a graph, where each step of the conversation is represented by a node. Nodes are of type `NodeConfig`, and may contain the following properties:

* `name`: The name of the node; used as a reference to transition to the node.
* `role_messages`: A list of message `dicts` defining the bot's role/personality. Typically set once in the initial node.
* `task_messages`: A list of message `dicts` defining the current node's objectives.
* `functions`: A list of function call definitions and their corresponding handlers.
* `pre_actions`: Actions to execute before LLM inference. Actions run once upon transitioning to a node.
* `post_actions`: Actions to execute after LLM inference. Actions run once after the node's initial LLM inference.
* `context_strategy`: Strategy for updating context during transitions. The default behavior is to append messages to the context.
* `respond_immediately`: Whether to run LLM inference as soon as the node is set. The default is True.

<Info>
  The only required field is `task_messages`, as your bot always needs a prompt
  to advance the conversation.
</Info>

Now that we've defined the structure of a node, let's look at the components that make up a node.

### Messages

Messages define what your bot should do and how it should behave at each node in your conversation flow.

#### Message Types

There are two types of messages you can configure:

**Role Messages** (Optional)
Define your bot's personality, tone, and overall behavior. These are typically set once at the beginning of your flow and establish the consistent persona your bot maintains throughout the conversation.

**Task Messages** (Required)
Define the specific objective your bot should accomplish in the current node. These messages focus the LLM on the immediate task at hand, such as asking a specific question or processing particular information.

#### Message Format

Messages are specified in OpenAI format as a list of `dicts`:

```python theme={null}
"role_messages": [
    {
        "role": "system",
        "content": "You are an inquisitive child.",
    }
],
"task_messages": [
    {
        "role": "system",
        "content": "Say 'Hello world' and ask what is the user's favorite color.",
    }
],
```

#### Cross-Provider Compatibility

Pipecat's default message format uses the OpenAI message spec. With this message format, messages are automatically translated by Pipecat to work with your chosen LLM provider, making it easy to switch between OpenAI, Anthropic, Google, and other providers without changing your code.

<Warning>
  Some LLMs, like Anthropic and Gemini, can only set the system instruction at
  initialization time. This means you will only be able to set the
  `role_messages` at initialization for those LLMs.
</Warning>

### Functions

Functions in Pipecat Flows serve two key purposes:

1. **Process data** by interfacing with external systems and APIs to read or write information
2. **Progress the conversation** by transitioning between nodes in your flow

#### How Functions Work

When designing your nodes, clearly define the task in the `task_messages` and reference the available functions. The LLM will use these functions to complete the task and signal when it's ready to move forward.

For example, if your node's job is to collect a user's favorite color:

1. The LLM asks the question
2. The user provides their answer
3. The LLM calls the function with the answer
4. The function processes the data and determines the next node

#### Function Definition

Flows provides a universal `FlowsFunctionSchema` that works across all LLM providers:

```python theme={null}
from pipecat_flows import FlowsFunctionSchema

record_favorite_color_func = FlowsFunctionSchema(
    name="record_favorite_color_func",
    description="Record the color the user said is their favorite.",
    required=["color"],
    handler=record_favorite_color_and_set_next_node,
    properties={"color": {"type": "string"}},
)
```

#### Function Handlers

Each function has a corresponding `handler` where you implement your application logic and specify the next node:

```python theme={null}
async def record_favorite_color_and_set_next_node(
    args: FlowArgs, flow_manager: FlowManager
) -> tuple[str, NodeConfig]:
    """Function handler that records the color then sets the next node.

    Here "record" means print to the console, but any logic could go here:
    Write to a database, make an API call, etc.
    """
    print(f"Your favorite color is: {args['color']}")
    return args["color"], create_end_node()
```

#### Handler Return Values

Function handlers must return a tuple containing:

* **Result**: Data provided to the LLM for context in subsequent completions
* **Next Node**: The `NodeConfig` for Flows to transition to next

Some handlers may not want to transition conversational state, in which case you can return `None` for the next node. Other handlers may *only* want to transition conversational state without doing other work, in which case you can return `None` for the result.

#### Direct Functions

For more concise code, you can optionally use Direct Functions where the function definition and handler are combined in a single function. The function signature and docstring are automatically used to generate the function schema:

```python theme={null}
async def record_favorite_color(
    flow_manager: FlowManager,
    color: str
) -> tuple[FlowResult, NodeConfig]:
    """
    Record the color the user said is their favorite.

    Args:
        color (str): The user's favorite color
    """
    print(f"Your favorite color is: {args["color"]}")
    return args["color"], create_end_node()

# Use directly in NodeConfig
node_config = {
    "functions": [record_favorite_color]
}
```

This approach eliminates the need for separate `FlowsFunctionSchema` definitions while maintaining the same functionality.

### Actions

Actions allow you to execute custom functionality at specific points in your conversation flow, giving you precise control over timing and sequencing.

#### Action Types

* `pre_actions` execute immediately when transitioning to a new node, *before* the LLM inference begins.
* `post_actions` execute after the LLM inference completes and any TTS has finished speaking.

#### Built-in Actions

Pipecat Flows includes several ready-to-use actions for common scenarios:

* **`tts_say`**: Speak a phrase immediately (useful for "please wait" messages)

```python theme={null}
"pre_actions": [
    {
        "type": "tts_say",
        "text": "Please hold while I process your request..."
    }
]
```

* **`end_conversation`**: Gracefully terminate the conversation

```python theme={null}
"post_actions": [
    {
        "type": "end_conversation",
        "text": "Thank you for your time!"
    }
]
```

* **`function`**: Execute a custom function at the specified timing

```python theme={null}
"post_actions": [
    {
        "type": "function",
        "handler": end_conversation_handler
    }
]
```

#### Custom Actions

You can define your own actions to handle specific business logic or integrations. In most cases, consider using a **function action** first, as it executes at the expected time in the pipeline.

Custom actions give you complete flexibility to execute any functionality your application needs, but require careful timing considerations.

#### Action Timing

The execution order ensures predictable behavior:

1. **Pre-actions** run first upon node entry (in the order they are defined)
2. **LLM inference** processes the node's messages and functions
3. **TTS** speaks the LLM's response
4. **Post-actions** run after TTS completes (in the order they are defined)

This timing guarantees that actions execute in the correct sequence, such as ensuring the bot finishes speaking before ending the conversation. Note that custom actions may not follow this predictable timing, which is another reason to prefer function actions when possible.

### Context Strategy

Flows provides three built-in ways to manage conversation context as you move between nodes:

#### Strategy Types

1. **APPEND** (Default): New node messages are added to the existing context, preserving the full conversation history. The context grows as the conversation progresses.

2. **RESET**: The context is cleared and replaced with only the new node's messages. Useful when previous conversation history is no longer relevant or to reduce context window size.

3. **RESET\_WITH\_SUMMARY**: The context is cleared but includes an AI-generated summary of the previous conversation along with the new node's messages. Helps reduce context size while preserving key information.

#### When to Use Each Strategy

* Use **APPEND** when full conversation history is important for context
* Use **RESET** when starting a new topic or when previous context might confuse the current task
* Use **RESET\_WITH\_SUMMARY** for long conversations where you need to preserve key points but reduce context size

#### Configuration Examples

Context strategies can be defined globally in the FlowManager constructor:

```python theme={null}
from pipecat_flows import ContextStrategy, ContextStrategyConfig

# Global strategy configuration
flow_manager = FlowManager(
    task=task,
    llm=llm,
    context_aggregator=context_aggregator,
    context_strategy=ContextStrategyConfig(
        strategy=ContextStrategy.APPEND,
    )
)
```

Or on a per-node basis:

```python theme={null}
# Per-node strategy configuration
node_config = {
    "task_messages": [...],
    "functions": [...],
    "context_strategy": ContextStrategyConfig(
        strategy=ContextStrategy.RESET_WITH_SUMMARY,
        summary_prompt="Provide a concise summary of the customer's order details and preferences."
    )
}
```

### Respond Immediately

For each node in the conversation, you can decide whether the LLM should respond immediately upon entering the node (the default behavior) or whether the LLM should wait for the user to speak first before responding. You do this using the `respond_immediately` field.

<Tip>
  `respond_immediately=False` may be particularly useful in the very first node,
  especially in outbound-calling cases where the user has to first answer the
  phone to trigger the conversation.
</Tip>

```python theme={null}
NodeConfig(
    task_messages=[
        {
            "role": "system",
            "content": "Warmly greet the customer and ask how many people are in their party. This is your only job for now; if the customer asks for something else, politely remind them you can't do it.",
        }
    ],
    respond_immediately=False,
    # ... other fields
)
```

<Warning>
  Keep in mind that if you specify `respond_immediately=False`, the user may not
  be aware of the conversational task at hand when entering the node (the bot
  hasn't told them yet). While it's always important to have guardrails in your
  node messages to keep the conversation on topic, letting the user speak first
  makes it even more so.
</Warning>

## Initialization

Initialize your flow by creating a `FlowManager` instance and calling `initialize()` to start the conversation.

### Dynamic Flow

First, create the FlowManager:

```python theme={null}
flow_manager = FlowManager(
    task=task,                              # PipelineTask
    llm=llm,                                # LLMService
    context_aggregator=context_aggregator,  # Context aggregator
    transport=transport,                    # Transport
)
```

Then, initialize by passing the first `NodeConfig` into the `initialize()` method:

```python theme={null}
@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
    logger.info(f"Client connected")
    # Kick off the conversation.
    await flow_manager.initialize(create_initial_node())
```

### Static Flow

First, create the FlowManager:

```python theme={null}
flow_manager = FlowManager(
    task=task,                              # PipelineTask
    llm=llm,                                # LLMService
    context_aggregator=context_aggregator,  # Context aggregator
    flow_config=flow_config,                # FlowConfig
)
```

Then initialize:

```python theme={null}
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
    await transport.capture_participant_transcription(participant["id"])
    logger.debug("Initializing flow")
    await flow_manager.initialize()
```

<Note>For Static Flows, `initialize()` does not take any args.</Note>

## Cross-Node Logic

### State

Pipecat Flows supports cross-node state through the `flow_manager.state` dictionary. This persistent storage lets you share data across nodes throughout the entire conversation:

```python theme={null}
async def record_favorite_color_and_set_next_node(
    args: FlowArgs, flow_manager: FlowManager
) -> tuple[str, NodeConfig]:
    """Function handler that records the color then sets the next node.

    Here "record" means print to the console, but any logic could go here;
    Write to a database, make an API call, etc.
    """
    flow_manager.state["color"] = args["color"] # Cross-node state setting
    print(f"Your favorite color is: {args['color']}")
    return args["color"], create_end_node()
```

### Functions

Pipecat Flows supports defining functions that are available across all nodes in your flow. They're defined in the same way as node-specific functions, but are passed into the FlowManager at initialization:

```python theme={null}
flow_manager = FlowManager(
    task=task,
    llm=llm,
    context_aggregator=context_aggregator,
    transport=transport,
    global_functions=[global_function_1, global_function_2], # Cross-node functions
)
```

## Usage Examples:

### Dynamic Flow (Recommended)

Here's an example that ties together all the concepts we've covered:

```python theme={null}
def create_initial_node() -> NodeConfig:
    """Create the initial node of the flow.

    Define the bot's role and task for the node as well as the function for it to call.
    The function call includes a handler which provides the function call result to
    Pipecat and then transitions to the next node.
    """
    record_favorite_color_func = FlowsFunctionSchema(
        name="record_favorite_color_func",
        description="Record the color the user said is their favorite.",
        required=["color"],
        handler=record_favorite_color_and_set_next_node,
        properties={"color": {"type": "string"}},
    )

    return {
        "name": "initial",
        "role_messages": [
            {
                "role": "system",
                "content": "You are an inquisitive child. Use very simple language. Ask simple questions. You must ALWAYS use one of the available functions to progress the conversation. Your responses will be converted to audio. Avoid outputting special characters and emojis.",
            }
        ],
        "task_messages": [
            {
                "role": "system",
                "content": "Say 'Hello world' and ask what is the user's favorite color.",
            }
        ],
        "functions": [record_favorite_color_func],
    }
```

### Static Flow

For static flows, you define your entire flow structure upfront using a `FlowConfig`. This follows the same rules as the NodeConfig, but is assembled as a single JSON object that defines the entire state of the program.

## Next Steps

Now that you understand the basics of Pipecat Flows, explore the reference docs and more examples:

<CardGroup>
  <Card title="Pipecat Flows Reference" icon="book" href="/server/frameworks/flows/pipecat-flows">
    Complete API reference and technical details
  </Card>

  <Card title="Pipecat Flows Examples" icon="rocket" href="https://github.com/pipecat-ai/pipecat-flows/tree/main/examples">
    Explore more complex examples and use cases
  </Card>
</CardGroup>


# WhatsApp Business Calling API
Source: https://docs.pipecat.ai/guides/features/whatsapp

Learn how to receive WhatsApp voice calls in your Pipecat application

## Overview

This guide explains how to integrate WhatsApp Business voice calling into your Pipecat application using the **WhatsApp Cloud API** and **SmallWebRTC**. Once configured, your Pipecat bot will be able to receive and handle real-time voice calls directly from WhatsApp users.

## Prerequisites

### WhatsApp Business Setup

Before proceeding, ensure you have completed the following steps:

1. **Facebook Account** — Create one at [facebook.com](https://facebook.com)
2. **Facebook Developer Account** — Register at [developers.facebook.com](https://developers.facebook.com)
3. **WhatsApp Business App** — Create a new app via [Meta for Developers](https://developers.facebook.com/apps)
4. **Phone Number** — Add and verify a WhatsApp Business phone number
5. **Business Verification** — Required for production environments
6. **Webhook Configuration** — You’ll need a public webhook URL to receive call events

> **Note**: Voice calling is only available for WhatsApp Business API numbers with calling capability enabled.

For reference:

* [WhatsApp Cloud API Getting Started](https://developers.facebook.com/docs/whatsapp/cloud-api/get-started/)
* [Voice Calling API Documentation](https://developers.facebook.com/docs/whatsapp/cloud-api/calling/)
* [Webhook Setup](https://developers.facebook.com/docs/whatsapp/webhooks/)

## WhatsApp Configuration

> **Note**: For these settings you must be logged into your **Meta Developer Console** and inside your **App**.

### 1. Enable Voice Calls

To enable voice calls for your number:

* Go to **WhatsApp** → **Configuration → Phone Numbers → Manage Phone Numbers**
* Select your WhatsApp number
* Under the **Calls** tab, enable **“Allow voice calls”**
* Save the configuration

> For development, Meta provides a **free test number** valid for **90 days**.

### 2. Configure Webhook

Set up your webhook to receive **WhatsApp call events**.

1. In the **Meta Developer Console**, go to
   **WhatsApp → Configuration → Webhooks**
2. **Set your callback URL** — this is where WhatsApp will send incoming call notifications.

   #### For local development

   * Make sure your local Pipecat server is running (e.g. `http://localhost:7860`).
   * Use **ngrok** (or a similar tunneling tool) to expose your local server:
     ```bash theme={null}
     ngrok http --domain=YOUR_NGROK_DOMAIN http://localhost:7860
     ```
   * Copy the generated HTTPS URL and set your webhook to:
     `https://YOUR_NGROK_DOMAIN/whatsapp`
   * ✅ **Important:** Always include the `/whatsapp` path at the end of your webhook URL.

   #### For Pipecat Cloud

   * Ensure your Pipecat agent is already **deployed and running**.
   * Use the following format for your webhook URL:
     ```
     https://api.pipecat.daily.co/v1/public/webhooks/$ORGANIZATION_ID/$AGENT_NAME/whatsapp
     ```
   * ✅ **Important:** Always include the `/whatsapp` path at the end of your webhook URL.
3. **Enter your Verify Token**
   * For **local development**, set the environment variable
     `WHATSAPP_WEBHOOK_VERIFICATION_TOKEN` to match the verify token you enter in Meta.
   * For **Pipecat Cloud**, set the verify token to your **public API key**.
4. Click **Verify and Save** to confirm your webhook setup.
5. Under **Webhook Fields**, enable the following event type:
   * `calls` → *(required to receive voice call events)*

### 3. Configure Access Token

1. Go to **WhatsApp → API Setup**
2. Click **Generate Access Token**
3. Use the generated token as your `WHATSAPP_TOKEN` environment variable (or secret)
4. Note your **Phone Number ID** and configure it as `PHONE_NUMBER_ID`

> This will create a temporary access token, which usually expires in less than 02 hours.

> To create a permanent access token, you can follow [this document](https://developers.facebook.com/blog/post/2022/12/05/auth-tokens/) from Meta.

### 4. Configure App secret

1. Go to **App settings → Basic**
2. Click to show your **App secret**
3. Use the app secret as your `WHATSAPP_APP_SECRET` environment variable (or secret)

> This secret is used to verify that the webhooks you receive are actually from WhatsApp.

## Pipecat Integration

WhatsApp works with any bot using `SmallWebRTCTransport`.

All you need to do is configure WhatsApp as described above and provide the correct environment variables when starting your Pipecat bot.

Make sure your `.env` file includes:

```bash theme={null}
WHATSAPP_TOKEN=
WHATSAPP_PHONE_NUMBER_ID=
WHATSAPP_APP_SECRET=

# Only needed for local development
WHATSAPP_WEBHOOK_VERIFICATION_TOKEN=pk_**
```

> If you are using Pipecat Cloud, you must configure these values as secrets.

### Example

```python theme={null}
transport = SmallWebRTCTransport(
    webrtc_connection=webrtc_connection,
    params=TransportParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
    ),
)
```

## Testing the Integration

1. From the WhatsApp app, call your registered WhatsApp Business number.
2. The call will be routed to your Pipecat application.
3. Your bot should automatically answer and begin the conversation.

## Full demo

<Card title="Demo" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/p2p-webrtc/pipecat-cloud">
  Pipecat Cloud WhatsApp Demo
</Card>

## Troubleshooting

Common issues and their solutions:

1. **Expired WhatsApp Token**
   ```bash theme={null}
   Error validating access token
   ```
   * Generate a new access token under **WhatsApp → API Setup**, then update your `WHATSAPP_TOKEN` and redeploy your agent.

2. **Webhook Verification Failed**
   * Ensure the verify token in Meta Developer Console matches your environment variable.
   * If using Pipecat Cloud, it must be your public API key.
   * Confirm that your webhook URL is publicly accessible.

3. **Bot Not Answering Calls**
   * Verify your phone number has calling enabled.
   * Check that your environment variables are correctly set.
   * Ensure Pipecat is running and reachable.

## Notes

* Voice calling requires WhatsApp Business API access.
* Test phone numbers are valid for 90 days in development mode.
* Production deployment requires verified business credentials.

## References

* [WhatsApp Cloud API – Getting Started](https://developers.facebook.com/docs/whatsapp/cloud-api/get-started/)
* [Voice Calling API Docs](https://developers.facebook.com/docs/whatsapp/cloud-api/calling/)
* [Webhook Configuration](https://developers.facebook.com/docs/whatsapp/webhooks/)
* [SDP Overview & Examples](https://developers.facebook.com/docs/whatsapp/cloud-api/calling/reference#sdp-overview-and-sample-sdp-structures)


# Context Summarization
Source: https://docs.pipecat.ai/guides/fundamentals/context-summarization

Automatically compress conversation history to manage token usage in long-running conversations

## Overview

In long-running voice AI conversations, context grows with every exchange. This increases token usage, raises costs, and can eventually hit context window limits. Pipecat includes built-in context summarization that automatically compresses older conversation history while preserving recent messages and important context.

## How It Works

Context summarization automatically triggers when **either** condition is met:

* **Token limit reached**: Context size exceeds `max_context_tokens` (estimated using \~4 characters per token)
* **Message count reached**: Number of new messages exceeds `max_unsummarized_messages`

When triggered, the system:

1. Sends a `LLMContextSummaryRequestFrame` to the LLM service
2. The LLM generates a concise summary of older messages
3. Context is reconstructed as: `[system_message] + [summary] + [recent_messages]`
4. Incomplete function call sequences and recent messages are preserved

<Note>
  Context summarization is asynchronous and happens in the background without
  blocking the pipeline. The system uses request IDs to match summary requests
  with results and handles interruptions gracefully.
</Note>

## Enabling Context Summarization

Enable summarization by setting `enable_context_summarization=True` in `LLMAssistantAggregatorParams`:

```python theme={null}
from pipecat.processors.aggregators.llm_response_universal import (
    LLMAssistantAggregatorParams,
    LLMContextAggregatorPair,
)

# Create aggregators with summarization enabled
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    assistant_params=LLMAssistantAggregatorParams(
        enable_context_summarization=True,
    ),
)
```

With the default configuration, summarization triggers at 8000 estimated tokens or after 20 new messages, whichever comes first.

## Customizing Behavior

Use `LLMContextSummarizationConfig` to control when and how summarization occurs:

```python theme={null}
from pipecat.utils.context.llm_context_summarization import LLMContextSummarizationConfig

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    assistant_params=LLMAssistantAggregatorParams(
        enable_context_summarization=True,
        context_summarization_config=LLMContextSummarizationConfig(
            max_context_tokens=8000,           # Trigger at 8000 tokens
            target_context_tokens=6000,        # Target summary size
            max_unsummarized_messages=20,      # Or trigger after 20 new messages
            min_messages_after_summary=4,      # Keep last 4 messages uncompressed
            summarization_prompt=None,         # Custom prompt (optional)
        ),
    ),
)
```

**Configuration parameters:**

| Parameter                    | Default | Description                                                                |
| ---------------------------- | ------- | -------------------------------------------------------------------------- |
| `max_context_tokens`         | 8000    | Maximum context size (in estimated tokens) before triggering summarization |
| `target_context_tokens`      | 6000    | Target token count for the generated summary                               |
| `max_unsummarized_messages`  | 20      | Maximum new messages before triggering summarization                       |
| `min_messages_after_summary` | 4       | Number of recent messages to preserve uncompressed                         |
| `summarization_prompt`       | None    | Custom prompt for summary generation (uses built-in default if None)       |

## What Gets Preserved

Context summarization intelligently preserves:

* **System messages**: The first system message (defining assistant behavior) is always kept
* **Recent messages**: The last N messages stay uncompressed (configured by `min_messages_after_summary`)
* **Function call sequences**: Incomplete function call/result pairs are not split during summarization

## Custom Summarization Prompts

You can override the default summarization prompt to control how the LLM generates summaries:

```python theme={null}
custom_prompt = """Summarize this conversation concisely.
Focus on: key decisions, user preferences, and action items.
Keep the summary under {target_tokens} tokens."""

config = LLMContextSummarizationConfig(
    summarization_prompt=custom_prompt,
)
```

When no custom prompt is provided, Pipecat uses a built-in prompt that instructs the LLM to create a concise summary preserving key information, user preferences, and conversation flow.

## Next Steps

<CardGroup>
  <Card title="Context Management" icon="book" href="/guides/learn/context-management">
    Learn how Pipecat manages conversation context in pipelines.
  </Card>

  <Card title="Saving Transcripts" icon="scroll" href="/guides/fundamentals/saving-transcripts">
    Collect and save conversation transcripts for analysis.
  </Card>
</CardGroup>


# Custom FrameProcessor
Source: https://docs.pipecat.ai/guides/fundamentals/custom-frame-processor

Learn how to write your own custom FrameProcessor

Pipecat's architecture is made up of a Pipeline, FrameProcessors, and Frames. See the [Core Concepts](/guides/learn/pipeline) for a full review. From that architecture, recall that FrameProcessors are the workers in the pipeline that receive frames and complete actions based on the frames received.

Pipecat comes with many FrameProcessors built in. These consist of services, like `OpenAILLMService` or `CartesiaTTSService`, utilities, like `LLMTextProcessor`, and other things. Largely, you can build most of your application with these built-in FrameProcessors, but commonly, your application code may require custom frame processing logic. For example, you may want to perform an action as a result of a frame that's pushed in the pipeline.

## Example: MetricsFrame logger

This custom FrameProcessor format and logs MetricsFrames:

```python theme={null}
class MetricsFrameLogger(FrameProcessor):
    """MetricsFrameLogger formats and logs all MetericsFrames"""

    def __init__(self):
        super().__init__()

    async def process_frame(self, frame: Frame, direction: FrameDirection):
        await super().process_frame(frame, direction)

        if isinstance(frame, MetricsFrame):
            logger.info(f"{frame.name}\n    {format_metrics(frame.data)}")
            await self.push_frame(frame, direction)

        # ALWAYS push all frames
        else:
            # SUPER IMPORTANT: always push every frame!
            await self.push_frame(frame, direction)
```

This frame processor looks for `MetricsFrames`. When it sees one, it formats the data and logs it.

It uses this `format_metrics` function:

```python theme={null}
def format_metrics(metrics, indent=0):
    lines = []
    tab = "\t" * indent

    for metric in metrics:
        lines.append(tab + type(metric).__name__)
        for field, value in vars(metric).items():
            if hasattr(value, "__dict__") and not isinstance(
                value, (str, int, float, bool, type(None))
            ):
                lines.append(f"{tab}\t{field}={type(value).__name__}")
                for k, v in vars(value).items():
                    lines.append(f"{tab}\t\t{k}={repr(v)}")
            else:
                lines.append(f"{tab}\t{field}={repr(value)}")

    return "\n".join(lines)
```

<Note>
  See this [working
  example](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/08-custom-frame-processor.py)
  using the `MetricsFrameLogger` FrameProcessor
</Note>

## Add to a Pipeline

```python theme={null}
# Create and initialize the custom FrameProcessor
metrics_frame_processor = MetricsFrameLogger()

pipeline = Pipeline(
    [
        transport.input(),
        stt,
        context_aggregator.user(),
        llm,
        tts,
        transport.output(),
        context_aggregator.assistant(),
        metrics_frame_processor,  # Our custom FrameProcessor that pretty prints metrics frames
    ]
)
```

With this positioning, the `MetricsFrameLogger` FrameProcessor will receive every MetericsFrame in the pipeline.

## Key Requirements

FrameProcessors must inherit from the base `FrameProcessor` class. This ensures that your custom FrameProcessor will correctly handle frames like `StartFrame`, `EndFrame`, `StartInterruptionFrame` without having to write custom logic for those frames. This inheritance also provides it with the ability to `process_frame()` and `push_frame()`:

* **`process_frame()`** is what allows the FrameProcessor to receive frames and add custom conditional logic based on the frames that are received.
* **`push_frame()`** allows the FrameProcessor to push frames to the pipeline. Normally, frames are pushed DOWNSTREAM, but based on which processors need the output, you can also push UPSTREAM or in both directions.

### Essential Implementation Details

To ensure proper base class inheritance, it's critical to include:

1. **`super().__init__()`** in your `__init__` method
2. **`await super().process_frame(frame, direction)`** in your `process_frame()` method

```python theme={null}
class MyCustomProcessor(FrameProcessor):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)  # ✅ Required
        # Your initialization code here

    async def process_frame(self, frame: Frame, direction: FrameDirection):
        await super().process_frame(frame, direction)  # ✅ Required

        # Your custom frame processing logic here
        if isinstance(frame, SomeSpecificFrame):
            # Handle the frame
            pass

        await self.push_frame(frame, direction)  # ✅ Required - pass frame through
```

## Critical Responsibility: Frame Forwarding

FrameProcessors receive **all** frames that are pushed through the pipeline. This gives them a lot of power, but also a great responsibility. Critically, they must push all frames through the pipeline; if they don't, they block frames from moving through the Pipeline, which will cause issues in how your application functions.

As well as formatting and logging MetricsFrames, `MetricsFrameLogger` also has an `await self.push_frame(frame, direction)` which pushes the frame through to the next processor in the pipeline.

## Frame Direction

When pushing frames, you can specify the direction:

```python theme={null}
# Push downstream (default)
await self.push_frame(frame, FrameDirection.DOWNSTREAM)

# Push upstream
await self.push_frame(frame, FrameDirection.UPSTREAM)
```

Most custom FrameProcessors will push frames downstream, but upstream can be useful for sending control frames or error notifications back up the pipeline.

## Best Practices

1. **Always call the parent methods**: Use `super().__init__()` and `await super().process_frame()`
2. **Forward all frames**: Make sure every frame is pushed through with `await self.push_frame(frame, direction)`
3. **Handle frames conditionally**: Use `isinstance()` checks to handle specific frame types
4. **Use proper error handling**: Wrap risky operations in try/catch blocks
5. **Position carefully in pipeline**: Consider where in the pipeline your processor needs to be to receive the right frames

With these patterns, you can create powerful custom FrameProcessors that extend Pipecat's capabilities for your specific use case.


# Detecting Idle Users
Source: https://docs.pipecat.ai/guides/fundamentals/detecting-user-idle

Learn how to detect and respond when users are inactive in conversations

## Overview

In conversational applications, it's important to handle situations where users go silent or inactive. Pipecat provides built-in idle detection through `LLMUserAggregator` and `UserTurnProcessor`, allowing your bot to respond appropriately when users haven't spoken for a defined period.

## How It Works

Idle detection monitors user activity and:

1. Starts tracking after the first interaction (user or bot speaking)
2. Resets a timer whenever the user or bot speaks
3. Pauses during function calls (which may take longer than the timeout)
4. Emits an `on_user_turn_idle` event when the user is idle for longer than the timeout period
5. Allows you to implement escalating responses or gracefully end the conversation in your application code

<Note>
  Idle detection uses continuous activity frames (`UserSpeakingFrame`,
  `BotSpeakingFrame`) to track real-time conversation activity. It requires an
  active speech-to-text service or a transport with built-in speech detection.
</Note>

## Basic Implementation

### Step 1: Enable Idle Detection

Enable idle detection by setting the `user_idle_timeout` parameter when creating your aggregator:

```python theme={null}
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_idle_timeout=5.0,  # Detect idle after 5 seconds
    ),
)
```

### Step 2: Handle Idle Events

Create an event handler to respond when the user becomes idle:

```python theme={null}
@user_aggregator.event_handler("on_user_turn_idle")
async def on_user_turn_idle(aggregator):
    # Send a reminder to the user
    message = {
        "role": "system",
        "content": "The user has been quiet. Politely ask if they're still there.",
    }
    await aggregator.push_frame(LLMMessagesAppendFrame([message], run_llm=True))
```

### Step 3: Implement Retry Logic (Optional)

For escalating responses, track retry count in your application:

```python theme={null}
class IdleHandler:
    def __init__(self):
        self._retry_count = 0

    def reset(self):
        self._retry_count = 0

    async def handle_idle(self, aggregator):
        self._retry_count += 1

        if self._retry_count == 1:
            # First attempt - gentle reminder
            message = {
                "role": "system",
                "content": "The user has been quiet. Politely ask if they're still there.",
            }
            await aggregator.push_frame(LLMMessagesAppendFrame([message], run_llm=True))
        elif self._retry_count == 2:
            # Second attempt - more direct
            message = {
                "role": "system",
                "content": "The user is still inactive. Ask if they'd like to continue.",
            }
            await aggregator.push_frame(LLMMessagesAppendFrame([message], run_llm=True))
        else:
            # Third attempt - end conversation
            await aggregator.push_frame(
                TTSSpeakFrame("It seems like you're busy. Have a nice day!")
            )
            await aggregator.push_frame(EndTaskFrame(), FrameDirection.UPSTREAM)

# Use the handler
idle_handler = IdleHandler()

@user_aggregator.event_handler("on_user_turn_idle")
async def on_user_turn_idle(aggregator):
    await idle_handler.handle_idle(aggregator)

@user_aggregator.event_handler("on_user_turn_started")
async def on_user_turn_started(aggregator, strategy):
    idle_handler.reset()  # Reset retry count when user speaks
```

## Best Practices

* **Set appropriate timeouts**: Shorter timeouts (5-10 seconds) work well for voice conversations
* **Use escalating responses**: Start with gentle reminders and gradually become more direct
* **Limit retry attempts**: After 2-3 unsuccessful attempts, consider [ending the conversation](/guides/learn/pipeline-termination) gracefully by pushing an `EndTaskFrame`
* **Reset on user activity**: Use the `on_user_turn_started` event to reset your retry counter when the user speaks
* **Let the LLM respond naturally**: Use system messages to prompt the LLM rather than hardcoded TTS responses for more natural interactions

## Next Steps

<CardGroup>
  <Card title="Try the User Idle Example" icon="code" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/17-detect-user-idle.py">
    Explore a complete working example that demonstrates how to detect and
    respond to user inactivity in Pipecat.
  </Card>

  <Card title="Turn Events" icon="book" href="/server/utilities/turn-management/turn-events">
    Learn about all available turn events and their parameters.
  </Card>
</CardGroup>

Implementing idle user detection improves the conversational experience by ensuring your bot can handle periods of user inactivity gracefully, either by prompting for re-engagement or politely ending the conversation when appropriate.


# IVR Navigation
Source: https://docs.pipecat.ai/guides/fundamentals/ivr

Automatically navigate phone system menus using AI-powered decision making

## Overview

The `IVRNavigator` enables your bot to automatically navigate Interactive Voice Response (IVR) phone systems to reach specific goals. Instead of manually programming navigation paths, you provide an end goal and the bot handles the complex decision-making required to traverse phone menus using DTMF tones and conversational responses.

## How IVR Navigation Works

The IVRNavigator combines several intelligent capabilities:

1. **Goal-oriented navigation**: You specify what you want to accomplish (e.g., "reach billing support")
2. **Automatic classification**: Detects whether incoming audio is an IVR system or human conversation
3. **Smart decision making**: Analyzes menu options and selects the best path toward your goal
4. **Multi-modal responses**: Uses both DTMF tones for menu selection, natural language for prompts, and waits when no input is appropriate
5. **Status tracking**: Monitors progress and reports completion, waiting, or stuck states

## Navigation Outcomes

The `IVRNavigator` can reach several outcomes during navigation:

### Completed ✅

The navigator successfully reaches its goal and is ready for the next step. Common actions:

* [Terminate the pipeline](/guides/learn/pipeline-termination) if the goal is complete
* Transfer the call to a human agent
* Allow your bot to start a conversation with the reached department

```python theme={null}
@ivr_navigator.event_handler("on_ivr_status_changed")
async def handle_ivr_status(processor, status):
    if status == IVRStatus.COMPLETED:
        logger.info("Successfully navigated to target department")
        # Start conversation
        # Replace the context with new messages for the upcoming conversation
        messages = [{"role": "system", "content": "You are a helpful customer service assistant."}]
        await task.queue_frame(LLMMessagesUpdateFrame(messages))
        # Adjust VAD stop_secs for conversation
        vad_params = VADParams(stop_secs=0.8)
        await task.queue_frame(VADParamsUpdateFrame(vad_params))
```

### Stuck ⚠️

The navigator cannot find a path forward in the IVR system. This might happen when:

* Required information (account numbers, PINs) isn't available
* The menu options don't align with the stated goal
* The system encounters errors or invalid selections

```python theme={null}
@ivr_navigator.event_handler("on_ivr_status_changed")
async def handle_ivr_status(processor, status):
    if status == IVRStatus.STUCK:
        logger.warning("IVR navigation stuck - terminating call")
        # Log the issue and clean up
        await log_navigation_failure()
        await task.queue_frame(EndFrame())
```

## Flexible Entry Points

One of the `IVRNavigator`'s key features is flexible entry point handling. When you dial a phone number, you might encounter either:

* An IVR system with menu options
* A direct connection to a human

The navigator automatically detects which scenario occurs and emits appropriate events:

### Human Conversation Detected

When a human answers instead of an IVR system, an `on_conversation_detected` event is emitted. You can handle that event to transition to a conversation.

```python theme={null}
@ivr_navigator.event_handler("on_conversation_detected")
async def on_conversation_detected(processor, conversation_history):
    # Set up conversation prompt and preserve conversation history
    messages = [
        {
            "role": "system",
            "content": "You are an assistant calling to check on the prescription for John Smith, date of birth 01/01/1990.",
        }
    ]

    # Add preserved conversation history if available
    if conversation_history:
        messages.extend(conversation_history)

    await task.queue_frame(LLMMessagesUpdateFrame(messages=messages, run_llm=True))
```

Note that the `on_conversation_detected` event also emits a `conversation_history` parameter that contains the previous conversation history. This allows you to build a prompt that includes your conversation system prompt plus any conversation history up to that point in time.

### IVR System Detected

When an IVR system is detected, the IVR Navigator automatically transitions into navigation mode:

* **System prompt**: Updates to use your specified navigation goal
* **VAD timing**: Adjusts to `stop_secs=2.0` (or your custom `ivr_vad_params`) to allow time for complete menu announcements
* **Navigation logic**: Begins analyzing menu options and making decisions toward your goal

No additional code is required. The navigator handles this transition automatically.

**Optional Event Handling**

If you need to log the detection or perform custom actions, you can handle the `on_ivr_status_changed` event:

```python theme={null}
@ivr_navigator.event_handler("on_ivr_status_changed")
async def on_ivr_status_changed(processor, status):
    if status == IVRStatus.DETECTED:
        logger.info("IVR system detected - beginning navigation")
        # Optional: Add analytics tracking, custom setup, etc.
```

## Basic Implementation

### Step 1: Create the IVR Navigator

```python theme={null}
from pipecat.extensions.ivr.ivr_navigator import IVRNavigator
from pipecat.audio.vad.vad_analyzer import VADParams

# Define your navigation goal
ivr_goal = "Navigate to the billing department to discuss my account balance"

# Create navigator with extended response time for IVR systems
ivr_vad_params = VADParams(stop_secs=2.0)  # Longer wait for IVR menus

ivr_navigator = IVRNavigator(
    llm=your_llm_service,
    ivr_prompt=ivr_goal,
    ivr_vad_params=ivr_vad_params
)
```

### Step 2: Set Up Event Handlers

```python theme={null}
from pipecat.frames.frames import LLMMessagesUpdateFrame
from pipecat.extensions.ivr.ivr_navigator import IVRStatus

@ivr_navigator.event_handler("on_conversation_detected")
async def on_conversation_detected(processor, conversation_history):
    """Handle when a human conversation is detected instead of IVR"""
    logger.info("Human conversation detected")

    # Set up conversation context
    messages = [
        {"role": "system", "content": "You are a customer service representative."}
    ]
    if conversation_history:
        messages.extend(conversation_history)

    await task.queue_frame(LLMMessagesUpdateFrame(messages=messages, run_llm=True))

@ivr_navigator.event_handler("on_ivr_status_changed")
async def on_ivr_status_changed(processor, status):
    """Handle IVR navigation status changes"""
    if status == IVRStatus.COMPLETED:
        logger.info("IVR navigation completed successfully")
        # Your success handling logic here

    elif status == IVRStatus.STUCK:
        logger.warning("IVR navigation got stuck")
        # Your error handling logic here
        await handle_navigation_failure()
```

### Step 3: Add to Pipeline

Add the IVR Navigator to your pipeline in the place where you would normally add the LLM. The IVR Navigator contains your LLM and will perform the same functions as an LLM would, but in addition it will navigate the IVR system.

```python theme={null}
from pipecat.pipeline.pipeline import Pipeline

pipeline = Pipeline([
    transport.input(),
    stt_service,
    ivr_navigator,  # Add the navigator to your pipeline
    tts_service,
    transport.output()
])
```

## VAD Parameter Optimization

The IVRNavigator automatically optimizes Voice Activity Detection (VAD) parameters for different scenarios:

### IVR Navigation Mode

* **Default**: `stop_secs=2.0`
* **Purpose**: Allows time to hear complete menu options before responding
* **Result**: Higher navigation success rates

### Conversation Mode

* **Recommended**: `stop_secs=0.8`
* **Purpose**: Enables natural conversation flow with quick responses
* **Implementation**: Push `VADParamsUpdateFrame` when transitioning to conversation

## Next Steps

<CardGroup>
  <Card title="Pipeline Termination" icon="stop" href="/guides/learn/pipeline-termination">
    Learn how to properly terminate pipelines when IVR navigation completes
  </Card>

  <Card title="Pipecat Flows" icon="flow" href="/guides/features/pipecat-flows">
    Integrate structured conversation flows after successful IVR navigation
  </Card>
</CardGroup>

The IVRNavigator provides a powerful foundation for automating phone system interactions, allowing your bots to handle the complex task of menu navigation while you focus on the core conversation logic.


# Metrics
Source: https://docs.pipecat.ai/guides/fundamentals/metrics

Learn how to monitor performance and LLM/TTS usage with Pipecat.

When developing real-time, multimodal AI applications, monitoring two key
factors is crucial: performance (latency) and LLM/TTS usage. Performance impacts
user experience, while usage can affect operational costs. Pipecat offers
built-in metrics for both, which can be enabled with straightforward
configuration options.

## Enabling performance metrics

Set `enable_metrics=True` in `PipelineParams` when creating a task:

```python Example config theme={null}
task = PipelineTask(
            pipeline,
            params=PipelineParams(
                ...
                enable_metrics=True,
                ...
            ),
        )
```

Once enabled, Pipecat logs the following metrics:

| Metric          | Description                                     |
| --------------- | ----------------------------------------------- |
| TTFB            | Time To First Byte in seconds                   |
| Processing Time | Time taken by the service to respond in seconds |

```console Sample output theme={null}
AnthropicLLMService#0 TTFB: 0.8378312587738037
CartesiaTTSService#0 processing time: 0.0005071163177490234
CartesiaTTSService#0 TTFB: 0.17177796363830566
AnthropicLLMService#0 processing time: 2.4927797317504883
```

### Limiting TTFB responses

If you only want the **first** TTFB measurement for each service, you can
optionally pass `report_only_initial_ttfb=True` in `PipelineParams`:

```python Example config theme={null}
task = PipelineTask(
            pipeline,
            params=PipelineParams(
                ...
                enable_metrics=True,
                report_only_initial_ttfb=True,
                ...
            ),
        )
```

> **Note:** `enable_metrics=True` is required for this setting to have an
> effect.

## Enabling LLM/TTS Usage Metrics

Set `enable_usage_metrics=True` in PipelineParams when creating a task:

```python Example config theme={null}
task = PipelineTask(
            pipeline,
            params=PipelineParams(
                ...
                enable_usage_metrics=True,
                ...
            ),
        )
```

Pipecat will log the following as applicable:

| Metric    | Description                                 |
| --------- | ------------------------------------------- |
| LLM Usage | Number of prompt and completion tokens used |
| TTS Usage | Number of characters processed              |

```console Sample output theme={null}
CartesiaTTSService#0 usage characters: 65
AnthropicLLMService#0 prompt tokens: 104, completion tokens: 53
```

> **Note:** Usage metrics are recorded per interaction and do not represent
> running totals.

## Capturing Metrics Data

When metrics are enabled, Pipecat will emit a `MetricsFrame` for each interaction. The `MetricsFrame` contains:

* TTFB data
* Processing time
* Token usage for LLMs
* Character usage for TTS
* Smart Turn prediction data

You can access the metrics data by either adding a custom [FrameProcessor](/guides/fundamentals/custom-frame-processor) to your pipeline or adding an [observer](/server/utilities/observers/observer-pattern) to monitor `MetricsFrame`s.

### Example: Using a Custom FrameProcessor

Create a custom FrameProcessor to handle metrics data. Here's an example Metrics Processor that can be added to your pipeline after the TTS processor.

```python theme={null}
from pipecat.frames.frames import MetricsFrame
from pipecat.metrics.metrics import (
    LLMUsageMetricsData,
    ProcessingMetricsData,
    TTFBMetricsData,
    TTSUsageMetricsData,
)
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor

class MetricsLogger(FrameProcessor):
    async def process_frame(self, frame: Frame, direction: FrameDirection):
        await super().process_frame(frame, direction)

        if isinstance(frame, MetricsFrame):
            for d in frame.data:
                if isinstance(d, TTFBMetricsData):
                    print(f"!!! MetricsFrame: {frame}, ttfb: {d.value}")
                elif isinstance(d, ProcessingMetricsData):
                    print(f"!!! MetricsFrame: {frame}, processing: {d.value}")
                elif isinstance(d, LLMUsageMetricsData):
                    tokens = d.value
                    print(
                        f"!!! MetricsFrame: {frame}, tokens: {tokens.prompt_tokens}, characters: {tokens.completion_tokens}"
                    )
                elif isinstance(d, TTSUsageMetricsData):
                    print(f"!!! MetricsFrame: {frame}, characters: {d.value}")
        await self.push_frame(frame, direction)
```


# Recording Conversation Audio
Source: https://docs.pipecat.ai/guides/fundamentals/recording-audio

Learn how to record and save audio from conversations between users and your bot

## Overview

Recording audio from conversations provides valuable data for analysis, debugging, and quality control. You have two options for how to record with Pipecat:

### Option 1: Record using your transport service provider

Record without writing custom code by using your [transport](/server/services/supported-services#transports) provider's recording capabilities. In addition to saving you development time, some providers offer unique recording capabilities.

<Info>Refer to your service provider's documentation to learn more.</Info>

### Option 2: Create your own recording pipeline

Pipecat's `AudioBufferProcessor` makes it easy to capture high-quality audio recordings of both the user and bot during interactions. Opt for this approach if you want more control over your recording.

This guide focuses on how to recording using the `AudioBufferProcessor`, including high-level guidance for how to set up post-processing jobs for longer recordings.

## How the AudioBufferProcessor Works

The `AudioBufferProcessor` captures audio by:

1. Collecting audio frames from both the user (input) and bot (output)
2. Emitting events with recorded audio data
3. Providing options for composite or separate track recordings

<Note>
  Add the processor to your pipeline after the `transport.output()` to capture
  both the user audio and the bot audio as it's spoken.
</Note>

## Audio Recording Options

The `AudioBufferProcessor` offers several configuration options:

* **Composite recording**: Combined audio from both user and bot
  * `on_audio_data` event handler
* **Track-level recording**: Separate audio files for user and bot
  * `on_track_audio_data` event handler
* **Turn-based recording**: Individual audio clips for each speaking turn
  * `on_user_turn_audio_data` and `on_bot_turn_audio_data` event handlers
* **Mono or stereo output**: Single channel mixing or two-channel separation
  * `num_channels=1` for mono; `num_channels=2` for stereo

## Basic Implementation

### Step 1: Create an Audio Buffer Processor

Initialize the audio buffer processor with your desired configuration:

```python theme={null}
from pipecat.processors.audio.audio_buffer_processor import AudioBufferProcessor

# Create audio buffer processor with default settings
audiobuffer = AudioBufferProcessor(
    num_channels=1,               # 1 for mono, 2 for stereo (user left, bot right)
    enable_turn_audio=False,      # Enable per-turn audio recording
    user_continuous_stream=True,  # User has continuous audio stream
)
```

### Step 2: Add to Your Pipeline

Place the processor in your pipeline after all audio-producing components:

```python theme={null}
pipeline = Pipeline(
    [
        transport.input(),
        stt,
        context_aggregator.user(),
        llm,
        tts,
        transport.output(),
        audiobuffer,          # Add after all audio components
        context_aggregator.assistant(),
    ]
)
```

### Step 3: Start Recording

Explicitly start recording when needed, typically when a session begins:

```python theme={null}
@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
    logger.info(f"Client connected")
    # Important: Start recording explicitly
    await audiobuffer.start_recording()
    # Continue with session initialization...
```

<Warning>
  You must call `start_recording()` explicitly to begin capturing audio. The
  processor won't record automatically when initialized.
</Warning>

### Step 4: Handle Audio Data

Register an event handler to process audio data:

```python theme={null}
@audiobuffer.event_handler("on_audio_data")
async def on_audio_data(buffer, audio, sample_rate, num_channels):
    # Save or process the composite audio
    timestamp = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
    filename = f"recordings/conversation_{timestamp}.wav"

    # Create the WAV file
    with wave.open(filename, "wb") as wf:
        wf.setnchannels(num_channels)
        wf.setsampwidth(2)  # 16-bit audio
        wf.setframerate(sample_rate)
        wf.writeframes(audio)

    logger.info(f"Saved recording to {filename}")
```

<Tip>
  If recording separate tracks, you can use the `on_track_audio_data` event
  handler to save user and bot audio separately.
</Tip>

## Recording Longer Conversations

For conversations that last a few minutes, it may be sufficient to just buffer the audio in memory. However, for longer sessions, storing audio in memory poses two challenges:

1. **Memory Usage**: Long recordings can consume significant memory, leading to potential crashes or performance issues.
2. **Conversation Loss**: If the application crashes or the connection drops, you may lose all recorded audio.

Instead, use the `buffer_size` parameter to record audio in manageable segments. This allows you to periodically save audio data to disk or upload it to cloud storage, reducing memory usage and ensuring data persistence.
See an example of how to upload chunked audio to AWS cloud storage [here]().

### Chunked Recording

Set a reasonable `buffer_size` to trigger periodic uploads:

```python theme={null}
# 30-second chunks (recommended for most use cases)
SAMPLE_RATE = 24000
CHUNK_DURATION = 30  # seconds
audiobuffer = AudioBufferProcessor(
    sample_rate=SAMPLE_RATE,
    buffer_size=SAMPLE_RATE * 2 * CHUNK_DURATION  # 2 bytes per sample (16-bit)
)

chunk_counter = 0

@audiobuffer.event_handler("on_track_audio_data")
async def on_chunk_ready(buffer, user_audio, bot_audio, sample_rate, num_channels):
    global chunk_counter

    # Upload or save individual chunks
    await upload_audio_chunk(f"user_chunk_{chunk_counter:03d}.wav", user_audio, sample_rate, 1)
    await upload_audio_chunk(f"bot_chunk_{chunk_counter:03d}.wav", bot_audio, sample_rate, 1)

    chunk_counter += 1
```

### Multipart Upload Strategy

For cloud storage, use multipart uploads to stream audio chunks. For example AWS cloud storage, use the [s3 multipart upload API](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html).

If you are rolling your own multipart upload code, consider the following:

**Conceptual Approach:**

1. **Initialize multipart upload** when recording starts
2. **Upload chunks as parts** when buffers fill (every \~30 seconds)
3. **Complete multipart upload** when recording ends
4. **Post-process** to create final WAV file(s), concatenate audio chunks

**Benefits:**

* Memory efficient for long sessions
* Fault tolerant (no data loss if connection drops)
* Enables real-time processing and analysis
* Parallel upload of multiple tracks

### \[Optional] Post-Processing Pipeline

If not using a managed multipart upload framework like AWS s3 multipart upload, concatenate audio chunks together to create final audio files.
This can be done with tools like FFmpeg:

**Concatenating Audio Files:**

```bash theme={null}
# Method 1: Simple concatenation (same format)
ffmpeg -i "concat:chunk_001.wav|chunk_002.wav|chunk_003.wav" -acodec copy final.wav

# Method 2: Using file list (recommended for many chunks)
# Create filelist.txt with format:
# file 'chunk_001.wav'
# file 'chunk_002.wav'
# ...
ffmpeg -f concat -safe 0 -i filelist.txt -c copy final_recording.wav
```

**Automation Considerations:**

* Use sequence numbers in chunk filenames for proper ordering
* Include metadata (sample rate, channels, duration) with each chunk
* Implement retry logic for failed uploads
* Consider using cloud functions/lambdas for automatic post-processing

## Next Steps

<CardGroup>
  <Card title="Try the Audio Recording Example" icon="code" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/34-audio-recording.py">
    Explore a complete working example that demonstrates how to record and save
    both composite and track-level audio with Pipecat.
  </Card>

  <Card title="AudioBufferProcessor Reference" icon="book" href="/server/utilities/audio/audio-buffer-processor">
    Read the complete API reference documentation for advanced configuration
    options and event handlers.
  </Card>
</CardGroup>

Consider implementing audio recording in your application for quality assurance, training data collection, or creating conversation archives. The recorded audio can be stored locally, uploaded to cloud storage, or processed in real-time for further analysis.


# Saving Conversation Transcripts
Source: https://docs.pipecat.ai/guides/fundamentals/saving-transcripts

Learn how to collect and save conversation transcripts between users and your bot

## Overview

Recording transcripts of conversations between users and your bot is useful for debugging, analysis, and creating a record of interactions. Pipecat's turn events make it easy to collect both user and assistant messages as they occur.

## How It Works

Transcripts are collected using turn events on the context aggregators:

1. Capturing what the user says via `on_user_turn_stopped`
2. Capturing what the assistant says via `on_assistant_turn_stopped`
3. Each event provides the complete transcript for that turn
4. Allowing you to handle these events with custom logic

<Note>
  Turn events are emitted by the context aggregators (`LLMUserAggregator` and
  `LLMAssistantAggregator`), which are created as part of the
  `LLMContextAggregatorPair`.
</Note>

## Basic Implementation

### Step 1: Create Context Aggregators

First, create the context aggregator pair and get references to both aggregators:

```python theme={null}
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    UserTurnStoppedMessage,
    AssistantTurnStoppedMessage,
)

# Create context aggregator pair
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context)
```

### Step 2: Add to Your Pipeline

Include the aggregators in your pipeline:

```python theme={null}
pipeline = Pipeline(
    [
        transport.input(),
        stt,                              # Speech-to-text
        user_aggregator,
        llm,
        tts,                              # Text-to-speech
        transport.output(),
        assistant_aggregator,
    ]
)
```

### Step 3: Handle Turn Events

Register event handlers to capture transcripts when turns complete:

```python theme={null}
@user_aggregator.event_handler("on_user_turn_stopped")
async def on_user_turn_stopped(aggregator, strategy, message: UserTurnStoppedMessage):
    print(f"[{message.timestamp}] user: {message.content}")

@assistant_aggregator.event_handler("on_assistant_turn_stopped")
async def on_assistant_turn_stopped(aggregator, message: AssistantTurnStoppedMessage):
    print(f"[{message.timestamp}] assistant: {message.content}")
```

<Tip>
  In addition to console logging, you can save transcripts to a database or file
  for later analysis.
</Tip>

## Next Steps

<CardGroup>
  <Card title="Turn Events Reference" icon="book" href="/server/utilities/turn-management/turn-events">
    Learn about all available turn events and their parameters.
  </Card>

  <Card title="Transcriptions Reference" icon="scroll" href="/server/utilities/turn-management/transcriptions">
    See more examples for collecting and processing transcriptions.
  </Card>
</CardGroup>

Consider implementing transcript recording in your application for debugging during development and preserving important conversations in production. The transcript data can also be useful for analyzing conversation patterns and improving your bot's responses over time.


# User Input Muting
Source: https://docs.pipecat.ai/guides/fundamentals/user-input-muting

Learn how to control when user speech is processed in your conversational bot

## Overview

In conversational applications, there are moments when you don't want to process user speech, such as during bot introductions or while executing function calls. Pipecat's user mute strategies let you selectively "mute" user input based on different conversation states.

## When to Use Mute Strategies

Common scenarios for muting user input include:

* **During introductions**: Prevent the bot from being interrupted during its initial greeting
* **While processing functions**: Block input while the bot is retrieving external data
* **During bot speech**: Reduce false transcriptions while the bot is speaking
* **For guided conversations**: Create more structured interactions with clear turn-taking

## How It Works

User mute strategies work by blocking specific user-related frames from flowing through your pipeline. When muted, the following frames are filtered:

* Voice activity detection (VAD) events
* Interruption signals
* Raw audio input frames
* Transcription frames (both interim and final)

This prevents user speech from being processed during muted periods.

<Note>
  Mute strategies are configured on the `LLMUserAggregator` via the
  `user_mute_strategies` parameter.
</Note>

## Mute Strategies

Pipecat provides several built-in strategies for determining when to mute user input:

<CardGroup>
  <Card title="FirstSpeechUserMuteStrategy" icon="microphone-slash">
    Mute only during the bot's first speech utterance. Useful for introductions
    when you want the bot to complete its greeting before the user can speak.
  </Card>

  <Card title="MuteUntilFirstBotCompleteUserMuteStrategy" icon="hourglass-half">
    Start muted and remain muted until the first bot utterance completes.
    Ensures the bot's initial instructions are fully delivered.
  </Card>

  <Card title="FunctionCallUserMuteStrategy" icon="gear">
    Mute during function calls. Prevents users from speaking while the bot is
    processing external data requests.
  </Card>

  <Card title="AlwaysUserMuteStrategy" icon="volume-xmark">
    Mute whenever the bot is speaking. Creates a strict turn-taking conversation
    pattern.
  </Card>
</CardGroup>

<Warning>
  The `FirstSpeechUserMuteStrategy` and
  `MuteUntilFirstBotCompleteUserMuteStrategy` strategies should not be used
  together as they handle the first bot speech differently.
</Warning>

## Basic Implementation

Import and configure the mute strategies you need:

```python theme={null}
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.turns.user_mute import AlwaysUserMuteStrategy

# Configure with one or more strategies
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_mute_strategies=[AlwaysUserMuteStrategy()],
    ),
)
```

## Combining Multiple Strategies

Multiple strategies can be combined. They use OR logic—if **any** strategy indicates the user should be muted, input is suppressed:

```python theme={null}
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)

from pipecat.turns.user_mute import (
    MuteUntilFirstBotCompleteUserMuteStrategy,
    FunctionCallUserMuteStrategy,
)

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_mute_strategies=[
            MuteUntilFirstBotCompleteUserMuteStrategy(),  # Mute until first response
            FunctionCallUserMuteStrategy(),               # Mute during function calls
        ],
    ),
)
```

## Responding to Mute Events

You can register event handlers to be notified when muting starts or stops. This is particularly useful for providing visual feedback to users:

```python theme={null}
@user_aggregator.event_handler("on_user_mute_started")
async def on_user_mute_started(aggregator):
    logger.info("User mute started")
    # Send a visual indicator to your client
    # e.g., show a "Bot is speaking" indicator

@user_aggregator.event_handler("on_user_mute_stopped")
async def on_user_mute_stopped(aggregator):
    logger.info("User mute stopped")
    # Update your client UI
    # e.g., show a "You can speak now" indicator
```

These events fire whenever the mute state changes, allowing you to keep your UI synchronized with the bot's state.

## Best Practices

* **Choose strategies wisely**: Select the minimal set of strategies needed for your use case
* **Test user experience**: Excessive muting can frustrate users; balance control with usability
* **Provide feedback**: Use the mute event handlers to show visual cues when the user is muted to improve the experience

## Next Steps

<CardGroup>
  <Card title="User Mute Strategies Reference" icon="book" href="/server/utilities/turn-management/user-mute-strategies">
    Read the complete API reference documentation for all available mute
    strategies and their behavior.
  </Card>

  <Card title="User Turn Strategies" icon="arrows-turn-to-dots" href="/server/utilities/turn-management/user-turn-strategies">
    Learn how to configure turn detection behavior for more control over
    conversation flow.
  </Card>
</CardGroup>

Experiment with different muting strategies to find the right balance for your application.


# Voicemail Detection
Source: https://docs.pipecat.ai/guides/fundamentals/voicemail

Automatically classify outbound calls as conversation or voicemail and respond appropriately

## Overview

The VoicemailDetector classifies incoming communication as either live conversation or voicemail systems. This module is built primarily for voice AI bots that perform outbound calling, enabling them to respond appropriately based on whether a human answered or the call went to voicemail.

The detector is optimized for fast conversation response times, where TTS output is generated immediately but held in a gate until the classification decision is made. This ensures minimal latency for live conversations while preventing inappropriate responses to voicemail systems.

## How It Works

The VoicemailDetector uses a parallel pipeline architecture to perform real-time classification without interrupting conversation flow. It analyzes the initial response from the called party and determines whether it's a human greeting or an automated voicemail system.

Key features:

* **Real-time classification** - Determines conversation vs voicemail as soon as audio is received
* **TTS gating** - Holds generated audio until classification is complete
* **Event-driven** - Triggers custom handlers when voicemail is detected
* **Configurable timing** - Adjustable delay for voicemail response timing

## Basic Setup

### 1. Initialize the Detector

```python theme={null}
from pipecat.extensions.voicemail.voicemail_detector import VoicemailDetector
from pipecat.services.openai.llm import OpenAILLMService

# Create an LLM for classification
classifier_llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"))

# Initialize the detector
voicemail_detector = VoicemailDetector(
    llm=classifier_llm,
    voicemail_response_delay=2.0  # Default: 2 seconds
)
```

<Note>
  The VoicemailDetector works with two LLMs: your main conversation LLM (which
  must be text-based) and the classifier LLM (which can be either text-based or
  realtime with a `text` output modality). Realtime LLMs are not compatible for
  the main conversation LLM due to output control requirements.
</Note>

### 2. Configure the Pipeline

The VoicemailDetector requires two components in your pipeline:

* `detector()`: between STT and user context aggregator
* `gate()`: immediately after TTS service

```python theme={null}
pipeline = Pipeline([
    transport.input(),
    stt,
    voicemail_detector.detector(),    # Between STT and user context aggregator
    context_aggregator.user(),
    llm,
    tts,
    voicemail_detector.gate(),        # Immediately after TTS service
    transport.output(),
    context_aggregator.assistant(),
])
```

### 3. Handle Voicemail Events

When a voicemail is detected, the `on_voicemail_detected` event is triggered. In your event handler, you have access to `processor`, which is a FrameProcessor instance, allowing you to push frames to the pipeline. For example, you may want to have the bot output a pre-canned message and then end the call.

```python theme={null}
@voicemail_detector.event_handler("on_voicemail_detected")
async def handle_voicemail(processor):
    logger.info("Voicemail detected! Leaving a message...")

    # Leave a voicemail message
    await processor.push_frame(
        TTSSpeakFrame("Hello, this is Jamie calling about your appointment. Please call me back at 555-0123.")
    )

    # Optionally end the call after leaving the message
    await processor.push_frame(EndTaskFrame(), FrameDirection.UPSTREAM)
```

## Detecting a Conversation

When a conversation is detected, no additional processing is required by the VoicemailDetector. Once the VoicemailDetector classifies the call as a conversation, the TTSGate will be flushed, allowing the conversation to continue.

## Configuration Options

### Custom System Prompt

For specialized use cases, you can provide a custom classification prompt. The prompt must instruct the LLM to respond with exactly "CONVERSATION" or "VOICEMAIL":

```python theme={null}
custom_prompt = """
Your custom classification logic here.
Consider factors like business hours, call patterns, etc.
""" + VoicemailDetector.CLASSIFIER_RESPONSE_INSTRUCTION

voicemail_detector = VoicemailDetector(
    llm=classifier_llm,
    custom_system_prompt=custom_prompt
)
```

<Note>
  The `CLASSIFIER_RESPONSE_INSTRUCTION` constant contains the required response
  format: `'Respond with ONLY "CONVERSATION" if a person answered, or
      "VOICEMAIL" if it\'s voicemail/recording.'`
</Note>

For most use cases, the default classifier prompt will work effectively. If you need to customize the behavior, reference the built-in prompt as a starting point and modify from there.

### Response Timing

The `voicemail_response_delay` parameter controls how long to wait after the user stops speaking before triggering the voicemail event:

```python theme={null}
voicemail_detector = VoicemailDetector(
    llm=classifier_llm,
    voicemail_response_delay=3.0  # Wait 3 seconds instead of default 2
)
```

This delay ensures:

* Voicemail greetings have finished playing
* The response occurs during the recording period
* Proper timing for different voicemail system behaviors


# Context Management
Source: https://docs.pipecat.ai/guides/learn/context-management

A guide to working with Pipecat's Context and Context Aggregators

## What is Context in Pipecat?

In Pipecat, **context** refers to the conversation history that the LLM uses to generate responses. The context consists of a list of alternating user/assistant messages that represents the collective history of the entire conversation.

```python theme={null}
# Example context structure
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello!"},
    {"role": "assistant", "content": "Hi there! How can I help you?"},
    # Context aggregators automatically add new messages here
]
```

Since Pipecat is a real-time voice AI framework, context management happens automatically as the conversation flows, but you can also control it manually when needed.

## How Context Updates During Conversations

Context updates happen automatically as frames flow through your pipeline:

**User Messages:**

1. User speaks → `InputAudioRawFrame` → STT Service → `TranscriptionFrame`
2. `context_aggregator.user()` receives `TranscriptionFrame` and adds user message to context

**Assistant Messages:**

1. LLM generates response → `LLMTextFrame` → TTS Service → `TTSTextFrame`
2. `context_aggregator.assistant()` receives `TTSTextFrame` and adds assistant message to context

**Frame types that update context:**

* **`TranscriptionFrame`**: Contains user speech converted to text by STT service
* **`LLMTextFrame`**: Contains LLM-generated responses
* **`TTSTextFrame`**: Contains bot responses converted to text by TTS service (represents what was actually spoken)

<Note>
  The TTS service processes `LLMTextFrame`s but outputs `TTSTextFrame`s, which
  represent the actual spoken text returned by the TTS provider. This ensures
  context matches what users actually hear.
</Note>

## Setting Up Context Management

Pipecat includes a context aggregator that creates and manages context for both user and assistant messages:

### 1. Create the Context and Context Aggregator

```python theme={null}
# Create LLM service
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"))

# Create context with initial messages
messages = [
    {"role": "system", "content": "You are a helpful voice assistant."}
]

# Create context (messages only)
context = LLMContext(messages)

# Create context aggregator instance
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context)
```

<Note>
  The context aggregator also supports configuring [user turn
  strategies](/server/utilities/turn-management/user-turn-strategies) and [user
  mute strategies](/server/utilities/turn-management/user-mute-strategies) via
  `LLMUserAggregatorParams`.
</Note>

**About LLMContext:**

`LLMContext` is Pipecat's universal context container that stores conversation messages, tool definitions, and tool choice settings. It uses an OpenAI-compatible format that works across all LLM services through automatic adapter translation.

Key properties:

* **`messages`**: List of conversation messages (system, user, assistant, tool)
* **`tools`**: Optional `ToolsSchema` defining available functions
* **`tool_choice`**: Optional strategy for tool selection

### 2. Context with Function Calling

Context can also include tools (function definitions) that the LLM can call during conversations:

```python theme={null}
from pipecat.adapters.schemas.function_schema import FunctionSchema
from pipecat.adapters.schemas.tools_schema import ToolsSchema

# Define available functions
weather_function = FunctionSchema(
    name="get_current_weather",
    description="Get the current weather",
    properties={
        "location": {
            "type": "string",
            "description": "The city and state, e.g. San Francisco, CA",
        },
        "format": {
            "type": "string",
            "enum": ["celsius", "fahrenheit"],
            "description": "The temperature unit to use.",
        },
    },
    required=["location", "format"],
)

# Create tools schema
tools = ToolsSchema(standard_tools=[weather_function])

# Create context with both messages and tools
context = LLMContext(messages, tools)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context)
```

Function call results are also automatically stored in the context, maintaining a complete conversation history including tool interactions.

<Note>
  We'll cover function calling in detail in an upcoming section. The context
  aggregator handles function call storage automatically.
</Note>

### 3. Add Context Aggregators to Your Pipeline

```python theme={null}
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),      # User context aggregator
    llm,
    tts,
    transport.output(),
    context_aggregator.assistant(), # Assistant context aggregator
])
```

## Context Aggregator Placement

The placement of context aggregator instances in your pipeline is **crucial** for proper operation:

### User Context Aggregator

Place the user context aggregator **downstream from the STT service**. Since the user's speech results in `TranscriptionFrame` objects pushed by the STT service, the user aggregator needs to be positioned to collect these frames.

### Assistant Context Aggregator

Place the assistant context aggregator **after `transport.output()`**. This positioning is important because:

* The TTS service outputs `TTSTextFrame`s in addition to audio
* The assistant aggregator must be downstream to collect those frames
* It ensures context updates happen word-by-word for specific services (e.g. Cartesia, ElevenLabs, and Rime)
* Your context stays updated at the word level in case an interruption occurs

<Tip>
  Always place the assistant context aggregator **after** `transport.output()`
  to ensure proper word-level context updates during interruptions.
</Tip>

## Manual Context Control

You can programmatically add new messages to the context by pushing or queueing specific frames:

### Adding Messages

* **`LLMMessagesAppendFrame`**: Appends a new message to the existing context
* **`LLMMessagesUpdateFrame`**: Completely replaces the existing context with new messages

```python theme={null}
# Add a new system message to context
new_message = {"role": "system", "content": "You are now in expert mode."}
await task.queue_frames([
    LLMMessagesAppendFrame([new_message], run_llm=True), # Optionally trigger bot response, too
])
```

### Retrieving Current Context

The context aggregator provides a `context` property for getting the current context:

```python theme={null}
context = context_aggregator.user().context
```

## Context Summarization

In long-running conversations, context grows with every exchange, increasing token usage and potentially hitting context window limits. Pipecat includes built-in context summarization that automatically compresses older conversation history while preserving recent messages.

Enable it by setting `enable_context_summarization=True` when creating your context aggregators:

```python theme={null}
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    assistant_params=LLMAssistantAggregatorParams(
        enable_context_summarization=True,
    ),
)
```

<Card title="Context Summarization Guide" icon="arrow-right" href="/guides/fundamentals/context-summarization">
  Learn how to configure summarization triggers, customize behavior, and control what gets preserved.
</Card>

## Triggering Bot Responses

You may want to manually trigger the bot to speak in two scenarios:

1. **Starting a pipeline** where the bot should speak first
2. **After editing the context** using `LLMMessagesAppendFrame` or `LLMMessagesUpdateFrame`

```python theme={null}
# Example: Bot speaks first when pipeline starts
@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
    # Trigger a response
    await task.queue_frames([LLMRunFrame()])
```

```python theme={null}
# Example: Bot speaks after context is edited
new_message = {"role": "user", "content": "Tell me a fun fact."}
await task.queue_frames([
    LLMMessagesAppendFrame([new_message], run_llm=True), # Trigger bot response
])
```

This gives you fine-grained control over when and how the bot responds during the conversation flow.

## Key Takeaways

* **Context is conversation history** - automatically maintained as users and bots exchange messages
* **Frame types matter** - `TranscriptionFrame` for users, `TTSTextFrame` for assistants
* **Placement matters** - user aggregator after STT, assistant aggregator after transport output
* **Tools are included** - function definitions and results are stored in context
* **Manual control available** - use frames to append messages or trigger responses when needed
* **Word-level precision** - proper placement ensures context accuracy during interruptions
* **Automatic summarization** - enable context summarization to manage long conversations efficiently and reduce token costs

## What's Next

Now that you understand context management, let's explore how to configure the LLM services that process this context to generate intelligent responses.

<Card title="LLM Inference" icon="arrow-right" href="/guides/learn/llm">
  Learn how to configure language models in your voice AI pipeline
</Card>


# Function Calling
Source: https://docs.pipecat.ai/guides/learn/function-calling

Enable LLMs to interact with external services and APIs in your voice AI pipeline

**Function calling** (also known as tool calling) allows LLMs to request information from external services and APIs during conversations. This extends your voice AI bot's capabilities beyond its training data to access real-time information and perform actions.

## Pipeline Integration

Function calling works seamlessly within your existing pipeline structure. The LLM service handles function calls automatically when they're needed:

```python theme={null}
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),     # Collects user transcriptions
    llm,                          # Processes context, calls functions when needed
    tts,
    transport.output(),
    context_aggregator.assistant(), # Collects function results and responses
])
```

**Function call flow:**

1. User asks a question requiring external data
2. LLM recognizes the need and calls appropriate function
3. Your function handler executes and returns results
4. LLM incorporates results into its response
5. Response flows to TTS and user as normal

**Context integration:** Function calls and their results are automatically stored in conversation context by the context aggregators, maintaining complete conversation history.

## Understanding Function Calling

Function calling allows your bot to access real-time data and perform actions that aren't part of its training data.

For example, you could give your bot the ability to:

* Check current weather conditions
* Look up stock prices
* Query a database
* Control smart home devices
* Schedule appointments

Here's how it works:

1. You define functions the LLM can use and register them to the LLM service used in your pipeline
2. When needed, the LLM requests a function call
3. Your application executes any corresponding functions
4. The result is sent back to the LLM
5. The LLM uses this information in its response

## Implementation

### 1. Define Functions

Pipecat provides a standardized `FunctionSchema` that works across all supported LLM providers. This makes it easy to define functions once and use them with any provider.

As a shorthand, you could also bypass specifying a function configuration at all and instead use "direct" functions. Under the hood, these are converted to `FunctionSchema`s.

#### Using the Standard Schema (Recommended)

```python theme={null}
from pipecat.adapters.schemas.function_schema import FunctionSchema
from pipecat.adapters.schemas.tools_schema import ToolsSchema

# Define a function using the standard schema
weather_function = FunctionSchema(
    name="get_current_weather",
    description="Get the current weather in a location",
    properties={
        "location": {
            "type": "string",
            "description": "The city and state, e.g. San Francisco, CA",
        },
        "format": {
            "type": "string",
            "enum": ["celsius", "fahrenheit"],
            "description": "The temperature unit to use.",
        },
    },
    required=["location", "format"]
)

# Create a tools schema with your functions
tools = ToolsSchema(standard_tools=[weather_function])

# Pass this to your LLM context
context = LLMContext(
    messages=[{"role": "system", "content": "You are a helpful assistant."}],
    tools=tools
)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context)
```

The `ToolsSchema` will be automatically converted to the correct format for your LLM provider through adapters.

#### Using Direct Functions (Shorthand)

You can bypass specifying a function configuration (as a `FunctionSchema` or in a provider-specific format) and instead pass the function directly to your `ToolsSchema`. Pipecat will auto-configure the function, gathering relevant metadata from its signature and docstring. Metadata includes:

* name
* description
* properties (including individual property descriptions)
* list of required properties

Note that the function signature is a bit different when using direct functions. The first parameter is `FunctionCallParams`, followed by any others necessary for the function.

```python theme={null}
from pipecat.adapters.schemas.tools_schema import ToolsSchema
from pipecat.services.llm_service import FunctionCallParams

# Define a direct function
async def get_current_weather(params: FunctionCallParams, location: str, format: str):
    """Get the current weather.

    Args:
        location: The city and state, e.g. "San Francisco, CA".
        format: The temperature unit to use. Must be either "celsius" or "fahrenheit".
    """
    weather_data = {"conditions": "sunny", "temperature": "75"}
    await params.result_callback(weather_data)

# Create a tools schema, passing your function directly to it
tools = ToolsSchema(standard_tools=[get_current_weather])

# Pass this to your LLM context
context = LLMContext(
    messages=[{"role": "system", "content": "You are a helpful assistant."}],
    tools=tools
)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context)
```

#### Using Provider-Specific Formats (Alternative)

You can also define functions in the provider-specific format if needed:

<CodeGroup>
  ```python OpenAI theme={null}
  from openai.types.chat import ChatCompletionToolParam

  # OpenAI native format
  tools = [
      ChatCompletionToolParam(
          type="function",
          function={
              "name": "get_current_weather",
              "description": "Get the current weather",
              "parameters": {
                  "type": "object",
                  "properties": {
                      "location": {
                          "type": "string",
                          "description": "The city and state, e.g. San Francisco, CA",
                      },
                      "format": {
                          "type": "string",
                          "enum": ["celsius", "fahrenheit"],
                          "description": "The temperature unit to use.",
                      },
                  },
                  "required": ["location", "format"],
              },
          },
      )
  ]
  ```

  ```python Anthropic theme={null}
  # Anthropic native format
  tools = [
      {
          "name": "get_weather",
          "description": "Get the weather in a given location",
          "input_schema": {
              "type": "object",
              "properties": {
                  "location": {
                      "type": "string",
                      "description": "The city, e.g. San Francisco",
                  },
                  "format": {
                      "type": "string",
                      "enum": ["celsius", "fahrenheit"],
                      "description": "The temperature unit to use.",
                  },
              },
              "required": ["location", "format"],
          },
      }
  ]
  ```

  ```python Gemini theme={null}
  # Gemini native format
  tools = [
      {
          "function_declarations": [
              {
                  "name": "get_weather_current",
                  "description": "Get the current weather",
                  "parameters": {
                      "type": "object",
                      "properties": {
                          "location": {
                              "type": "string",
                              "description": "Location as 'city,state,country'",
                          },
                          "format": {
                              "type": "string",
                              "enum": ["celsius", "fahrenheit"],
                              "description": "The temperature unit to use.",
                          },
                      },
                      "required": ["location", "format"],
                  },
              }
          ]
      }
  ]
  ```
</CodeGroup>

#### Provider-Specific Custom Tools

Some providers support unique tools that don't fit the standard function schema. For these cases, you can add custom tools:

```python theme={null}
from pipecat.adapters.schemas.tools_schema import AdapterType, ToolsSchema

# Standard functions
weather_function = FunctionSchema(
    name="get_current_weather",
    description="Get the current weather",
    properties={"location": {"type": "string"}},
    required=["location"]
)

# Custom Gemini search tool
gemini_search_tool = {
    "web_search": {
        "description": "Search the web for information"
    }
}

# Create a tools schema with both standard and custom tools
tools = ToolsSchema(
    standard_tools=[weather_function],
    custom_tools={
        AdapterType.GEMINI: [gemini_search_tool]
    }
)
```

<Tip>
  See the provider-specific documentation for details on custom tools and their
  formats.
</Tip>

### 2. Register Function Handlers

Register handlers for your functions using one of these [LLM service methods](https://reference-server.pipecat.ai/en/latest/api/pipecat.services.llm_service.html#llm-service):

* `register_function`
* `register_direct_function`

Which one you use depends on whether your function is a ["direct" function](#using-direct-functions-shorthand).

<CodeGroup>
  ```python Non-Direct Function theme={null}
  from pipecat.services.llm_service import FunctionCallParams

  llm = OpenAILLMService(api_key="your-api-key")

  # Main function handler - called to execute the function
  async def fetch_weather_from_api(params: FunctionCallParams):
      # Fetch weather data from your API
      weather_data = {"conditions": "sunny", "temperature": "75"}
      await params.result_callback(weather_data)

  # Register the function
  llm.register_function(
      "get_current_weather",
      fetch_weather_from_api,
      cancel_on_interruption=True,  # Cancel if user interrupts (default: True)
  )
  ```

  ```python Direct Function theme={null}
  from pipecat.services.llm_service import FunctionCallParams

  llm = OpenAILLMService(api_key="your-api-key")

  # Direct function
  async def get_current_weather(params: FunctionCallParams, location: str, format: str):
      """Get the current weather.

      Args:
          location: The city and state, e.g. "San Francisco, CA".
          format: The temperature unit to use. Must be either "celsius" or "fahrenheit".
      """
      weather_data = {"conditions": "sunny", "temperature": "75"}
      await params.result_callback(weather_data)

  # Register direct function
  llm.register_direct_function(
      get_current_weather,
      cancel_on_interruption=False,  # Don't cancel on interruption
  )
  ```
</CodeGroup>

**Key registration options:**

* **`cancel_on_interruption=True`** (default): Function call is cancelled if user interrupts
* **`cancel_on_interruption=False`**: Function call continues even if user interrupts

Use `cancel_on_interruption=False` for critical operations that should complete even if the user starts speaking. Function calls are async, so you can continue the conversation while the function executes. Once the result returns, the LLM will automatically incorporate it into the conversation context. LLMs vary in terms of how well they incorporate changes to *previous* messages, so you may need to experiment with your LLM provider to see how it handles this.

### 3. Create the Pipeline

Include your LLM service in your pipeline with the registered functions:

```python theme={null}
# Initialize the LLM context with your function schemas
context = LLMContext(
    messages=[{"role": "system", "content": "You are a helpful assistant."}],
    tools=tools
)

# Create the context aggregator to collect the user and assistant context
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context)

# Create the pipeline
pipeline = Pipeline([
    transport.input(),     # Input from the transport
    stt,                   # STT processing
    user_aggregator,       # User context aggregation
    llm,                   # LLM processing
    tts,                   # TTS processing
    transport.output(),    # Output to the transport
    assistant_aggregator,  # Assistant context aggregation
])
```

## Function Handler Details

### FunctionCallParams

Every function handler receives a `FunctionCallParams` object containing all the information needed for execution:

```python theme={null}
@dataclass
class FunctionCallParams:
    function_name: str                          # Name of the called function
    tool_call_id: str                           # Unique identifier for this call
    arguments: Mapping[str, Any]                # Arguments from the LLM
    llm: LLMService                             # Reference to the LLM service
    context: LLMContext                         # Current conversation context
    result_callback: FunctionCallResultCallback # Return results here
```

**Using the parameters:**

```python theme={null}
async def example_function_handler(params: FunctionCallParams):
    # Access function details
    print(f"Called function: {params.function_name}")
    print(f"Call ID: {params.tool_call_id}")

    # Extract arguments
    location = params.arguments["location"]

    # Access LLM context for conversation history
    messages = params.context.messages

    # Use LLM service for additional operations
    await params.llm.push_frame(TTSSpeakFrame("Looking up weather data..."))

    # Return results
    await params.result_callback({"conditions": "nice", "temperature": "75"})
```

See the [API reference](https://reference-server.pipecat.ai/en/latest/api/pipecat.services.llm_service.html#pipecat.services.llm_service.FunctionCallParams) for complete details.

### Handler Structure

Your function handler should:

1. Receive necessary arguments, either:
   * From `params.arguments`
   * Directly from function arguments, if using [direct functions](#using-direct-functions-shorthand)
2. Process data or call external services
3. Return results via `params.result_callback(result)`

<CodeGroup>
  ```python Non-Direct Function theme={null}
  async def fetch_weather_from_api(params: FunctionCallParams):
      try:
          # Extract arguments
          location = params.arguments.get("location")
          format_type = params.arguments.get("format", "celsius")

          # Call external API
          api_result = await weather_api.get_weather(location, format_type)

          # Return formatted result
          await params.result_callback({
              "location": location,
              "temperature": api_result["temp"],
              "conditions": api_result["conditions"],
              "unit": format_type
          })
      except Exception as e:
          # Handle errors
          await params.result_callback({
              "error": f"Failed to get weather: {str(e)}"
          })
  ```

  ```python Direct Function theme={null}
  async def get_current_weather(params: FunctionCallParams, location: str, format: str):
      """Get the current weather.

      Args:
          location: The city and state, e.g. "San Francisco, CA".
          format: The temperature unit to use. Must be either "celsius" or "fahrenheit".
      """
      try:
          # Call external API
          api_result = await weather_api.get_weather(location, format)

          # Return formatted result
          await params.result_callback({
              "location": location,
              "temperature": api_result["temp"],
              "conditions": api_result["conditions"],
              "unit": format
          })
      except Exception as e:
          # Handle errors
          await params.result_callback({
              "error": f"Failed to get weather: {str(e)}"
          })
  ```
</CodeGroup>

## Controlling Function Call Behavior (Advanced)

When returning results from a function handler, you can control how the LLM processes those results using a `FunctionCallResultProperties` object passed to the result callback.

### Properties

`FunctionCallResultProperties` provides fine-grained control over LLM execution:

```python theme={null}
@dataclass
class FunctionCallResultProperties:
    run_llm: Optional[bool] = None                 # Whether to run LLM after this result
    on_context_updated: Optional[Callable] = None  # Callback when context is updated
```

**Property options:**

* **`run_llm=True`**: Run LLM after function call (default behavior)
* **`run_llm=False`**: Don't run LLM after function call (useful for chained calls)
* **`on_context_updated`**: Async callback executed after the function result is added to context

<Tip>
  Skip LLM execution (`run_llm=False`) when you have back-to-back function
  calls. If you skip a completion, you must manually trigger one from the
  context aggregator.
</Tip>

See the [API reference](https://reference-server.pipecat.ai/en/latest/api/pipecat.frames.frames.html#pipecat.frames.frames.FunctionCallResultProperties) for complete details.

### Example Usage

```python theme={null}
from pipecat.frames.frames import FunctionCallResultProperties
from pipecat.services.llm_service import FunctionCallParams

async def fetch_weather_from_api(params: FunctionCallParams):
    # Fetch weather data
    weather_data = {"conditions": "sunny", "temperature": "75"}

    # Don't run LLM after this function call
    properties = FunctionCallResultProperties(run_llm=False)

    await params.result_callback(weather_data, properties=properties)

async def query_database(params: FunctionCallParams):
    # Query database
    results = await db.query(params.arguments["query"])

    async def on_update():
        await notify_system("Database query complete")

    # Run LLM after function call and notify when context is updated
    properties = FunctionCallResultProperties(
        run_llm=True,
        on_context_updated=on_update
    )

    await params.result_callback(results, properties=properties)
```

## Key Takeaways

* **Function calling extends LLM capabilities** beyond training data to real-time information
* **Context integration is automatic** - function calls and results are stored in conversation history
* **Multiple definition approaches** - use standard schema for portability, direct functions for simplicity
* **Pipeline integration is seamless** - functions work within your existing voice AI architecture
* **Advanced control available** - fine-tune LLM execution and monitor function call lifecycle

## What's Next

Now that you understand function calling, let's explore how to configure text-to-speech services to convert your LLM's responses (including function call results) into natural-sounding speech.

<Card title="Text to Speech" icon="arrow-right" href="/guides/learn/text-to-speech">
  Learn how to configure speech synthesis in your voice AI pipeline
</Card>


# LLM Inference
Source: https://docs.pipecat.ai/guides/learn/llm

Learn how to configure language models to generate intelligent responses in your voice AI pipeline

**LLM services** are responsible for chat completions and tool calling based on the provided context (conversation history). The LLM responds by streaming tokens via `LLMTextFrame`s, which are used by subsequent processors to create audio output for the bot.

## Pipeline Placement

The LLM instance should be placed after the user context aggregator and before any downstream services that depend on the LLM's output stream:

```python theme={null}
pipeline = Pipeline([
    transport.input(),
    stt,                           # Creates TranscriptionFrames
    context_aggregator.user(),     # Processes user context → creates LLMContextFrame
    llm,                           # Processes context → streams LLMTextFrames
    tts,                           # Processes LLMTextFrames → creates TTSAudioRawFrames
    transport.output(),
    context_aggregator.assistant(),
])
```

**Frame flow:**

* **Input**: Receives `LLMContextFrame` containing conversation history
* **Processing**:
  * Analyzes context and generates streaming response
  * Handles function calls if tools are available
  * Tracks token usage for metrics
* **Output**:
  * Denotes the start of the streaming response by pushing an `LLMFullResponseStartFrame`
  * Streams `LLMTextFrame`s containing response tokens to downstream processors (enables real-time TTS processing)
  * Ends with an `LLMFullResponseEndFrame` to mark the completion of the response
* **Function calls:**
  * `FunctionCallsStartedFrame`: Indicates function execution beginning
  * `FunctionCallInProgressFrame`: Indicates a function is currently executing
  * `FunctionCallResultFrame`: Contains results from executed functions

## Supported LLM Services

Pipecat supports a wide range of LLM providers to fit different needs, performance requirements, and budgets:

### Text-Based LLMs

Most LLM services are built on the OpenAI chat completion specification for compatibility:

<CardGroup>
  <Card title="OpenAI" icon="openai">
    GPT models with the original chat completion API
  </Card>

  <Card title="Anthropic" icon="anthropic">
    Claude models with advanced reasoning capabilities
  </Card>

  <Card title="Google Gemini" icon="google">
    Multimodal capabilities with competitive performance
  </Card>

  <Card title="AWS Bedrock" icon="aws">
    Enterprise-grade hosting for various foundation models
  </Card>
</CardGroup>

**Compatible APIs**: Any OpenAI-spec compatible service can be used via the `base_url` parameter.

### Speech-to-Speech Models

For lower latency, some providers offer direct speech-to-speech models:

* **OpenAI Realtime**: Direct speech input/output with GPT models
* **Gemini Live**: Real-time speech conversations with Gemini
* **AWS Nova Sonic**: Speech-optimized models on Bedrock

For a complete list of supported LLM services, see the Supported Services page:

<Card title="Supported LLM Services" icon="list" href="/server/services/supported-services#large-language-models">
  View the complete list of supported language model providers
</Card>

## LLM Service Architecture

### BaseOpenAILLMService

Many LLM services use the OpenAI chat completion specification. Pipecat provides a `BaseOpenAILLMService` that most providers extend, enabling easy switching between compatible services:

```python theme={null}
from pipecat.services.openai.llm import OpenAILLMService

# Native OpenAI
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"))

# OpenAI-compatible service via base_url
llm = OpenAILLMService(
    api_key=os.getenv("OTHER_API_KEY"),
    base_url="https://api.other-provider.com/v1"  # Custom endpoint
)
```

This architecture allows you to quickly plug in different LLM services without changing your pipeline code.

## LLM Configuration

### Service-Specific Configuration

Each LLM service has its own configuration options. For example, configuring OpenAI with various parameters:

```python theme={null}
from pipecat.services.openai.base_llm import BaseOpenAILLMService
from pipecat.services.openai.llm import OpenAILLMService

llm = OpenAILLMService(
    api_key=os.getenv("OPENAI_API_KEY"),
    model="gpt-4",
    params=BaseOpenAILLMService.InputParams(
        temperature=0.7,            # Response creativity (0.0-2.0)
        max_completion_tokens=150,  # Maximum response length
        frequency_penalty=0.5,      # Reduce repetition (0.0-2.0)
        presence_penalty=0.5,       # Encourage topic diversity (0.0-2.0)
    ),
)
```

For detailed configuration options specific to each provider:

<Card title="Individual LLM Services" icon="settings" href="/server/services/supported-services#large-language-models">
  Explore configuration options for each supported LLM provider
</Card>

### Base Class Configuration

All LLM services inherit from the `LLMService` base class with shared configuration options:

```python theme={null}
llm = YourLLMService(
    # Service-specific options...
    run_in_parallel=True,  # Whether function calls run in parallel (default: True)
)
```

**Key options:**

* **`run_in_parallel`**: Controls whether function calls execute simultaneously or sequentially
  * `True` (default): Faster execution when multiple functions are called
  * `False`: Sequential execution for dependent function calls

## Event Handlers

LLM services provide event handlers for monitoring completion lifecycle:

```python theme={null}
@llm.event_handler("on_completion_timeout")
async def on_completion_timeout(service):
    logger.warning("LLM completion timed out")
    # Handle timeout (retry, fallback, etc.)

@llm.event_handler("on_function_calls_started")
async def on_function_calls_started(service, function_calls):
    logger.info(f"Starting {len(function_calls)} function calls")
    # Optionally notify user that bot is "thinking"
    await tts.queue_frame(TTSSpeakFrame("Let me check on that."))
```

**Available events:**

* **`on_completion_timeout`**: Triggered when LLM requests timeout
* **`on_function_calls_started`**: Triggered when function calls are initiated

These handlers enable you to provide user feedback and implement error recovery strategies.

## Function Calling

LLMs can call external functions to access real-time data and perform actions beyond their training data. This enables capabilities like checking weather, querying databases, or controlling external APIs.

Function calls and their results are automatically stored in the conversation context by the context aggregator.

<Card title="Function Calling" icon="code" href="/guides/learn/function-calling">
  Learn how to enable LLMs to interact with external services and APIs
</Card>

## Key Takeaways

* **Pipeline placement matters** - LLM goes after user context, before TTS
* **Token streaming enables real-time responses** - no waiting for complete generation
* **OpenAI compatibility** enables easy provider switching
* **Function calling extends capabilities** beyond training data
* **Configuration affects behavior** - tune temperature, penalties, and limits
* **Services are modular** - swap providers without changing pipeline code

## What's Next

Now that you understand LLM configuration, let's explore how function calling enables your bot to interact with external services and real-time data.

<Card title="Function Calling" icon="arrow-right" href="/guides/learn/function-calling">
  Learn how to enable LLMs to interact with external services and APIs
</Card>


# Overview of Pipecat
Source: https://docs.pipecat.ai/guides/learn/overview

Learn the foundational concepts of Pipecat's architecture for building voice AI agents

## What You'll Learn

This comprehensive guide will teach you how to build real-time voice AI agents with Pipecat. By the end, you'll be equipped with the knowledge to create custom applications—from simple voice assistants to complex multimodal bots that can see, hear, and speak.

<Info>
  **Prerequisites**: Basic Python knowledge is recommended. The guide takes
  approximately 45-60 minutes to complete, with hands-on examples throughout.
</Info>

## Why Voice AI is Challenging

Building responsive voice AI applications involves coordinating multiple AI services in real-time:

* **Speech recognition** must transcribe audio as users speak
* **Language models** need to process context and generate responses
* **Speech synthesis** has to convert text back to natural audio
* **Network transports** must handle streaming audio with minimal delay

Doing this manually means managing complex timing, buffering, error handling, and service coordination. Most developers end up rebuilding the same orchestration logic repeatedly.

## Pipecat's Solution

Pipecat solves this orchestration problem with a **pipeline architecture** that handles the complexity for you. Instead of managing individual API calls and timing, you define a flow of processing steps that work together automatically.

Here's what makes Pipecat different:

<CardGroup>
  <Card title="Ultra-Low Latency" icon="bolt">
    Typical voice interactions complete in 500-800ms for natural conversations
  </Card>

  <Card title="Modular Design" icon="puzzle-piece">
    Swap AI providers, add features, or customize behavior without rewriting
    code
  </Card>

  <Card title="Real-time Processing" icon="clock">
    Stream processing eliminates waiting for complete responses at each step
  </Card>

  <Card title="Production Ready" icon="shield-check">
    Built-in error handling, logging, and scaling considerations
  </Card>
</CardGroup>

## Core Architecture Concepts

Before diving into how voice AI works, let's understand Pipecat's three foundational concepts:

### Frames

Think of frames as **data packages** moving through your application. Each frame contains a specific type of information:

* Audio data from a microphone
* Transcribed text from speech recognition
* Generated responses from an LLM
* Synthesized audio for playback

### Frame Processors

Frame processors are **specialized workers** that handle specific tasks:

* A speech-to-text processor converts audio frames into text frames
* An LLM processor takes text frames and produces response frames
* A text-to-speech processor converts response frames into audio frames

### Pipelines

Pipelines **connect processors together**, creating a path for frames to flow through your application. They handle the orchestration automatically.

## Voice AI Processing Flow

Now let's see how these concepts work together in a typical voice AI interaction:

<Steps>
  <Step title="Audio Input">
    User speaks → Transport receives streaming audio → Creates audio frames
  </Step>

  <Step title="Speech Recognition">
    STT processor receives audio frames → Transcribes speech in real-time →
    Outputs text frames
  </Step>

  <Step title="Context Management">
    Context processor aggregates text frames with conversation history → Creates
    formatted input for LLM
  </Step>

  <Step title="Language Processing">
    LLM processor receives context → Generates streaming response → Outputs text
    frames
  </Step>

  <Step title="Speech Synthesis">
    TTS processor receives text frames → Converts to speech → Outputs audio
    frames
  </Step>

  <Step title="Audio Output">
    Transport receives audio frames → Streams to user's device → User hears
    response
  </Step>
</Steps>

The key insight: **everything happens in parallel**. While the LLM is generating later parts of a response, earlier parts are already being converted to speech and played back to the user.

## Pipeline Architecture

Here's how this flow translates into a Pipecat pipeline:

<Frame>
  <img alt="Pipecat Pipeline Architecture" />
</Frame>

Each processor in the pipeline:

1. Receives specific frame types as input
2. Performs its specialized task (transcription, language processing, etc.)
3. Outputs new frames for the next processor
4. Passes through frames it doesn't handle

<Info>
  While frames can flow upstream or downstream, most data flows downstream as
  shown above. We'll discuss pushing frames in later sections.
</Info>

## What's Next

In the following sections, we'll explore each component of this pipeline in detail:

* How to initialize sessions and connect users
* Configuring different transport options (Daily, WebRTC, Twilio, etc.)
* Setting up speech recognition and synthesis services
* Managing conversation context and LLM integration
* Handling the complete pipeline lifecycle
* Building custom processors for your specific needs

Each section includes practical examples and configuration options to help you build production-ready voice AI applications.

<Card title="Ready to Start Building?" icon="arrow-right" href="/guides/learn/session-initialization">
  Let's begin with session initialization to connect users to your voice AI
  agent
</Card>


# Pipeline & Frame Processing
Source: https://docs.pipecat.ai/guides/learn/pipeline

Learn how Pipecat's pipeline architecture orchestrates frame processing for voice AI applications

The **Pipeline** is the core orchestration component in Pipecat that connects frame processors together, creating a structured path for data to flow through your voice AI application.

## Basic Pipeline Structure

A Pipeline takes a list of frame processors and connects them in sequence. Here's a simple voice AI pipeline that matches the voice AI agent architecture we discussed earlier:

```python theme={null}
pipeline = Pipeline([
    transport.input(),              # Receives user audio
    stt,                            # Speech-to-text conversion
    context_aggregator.user(),      # Collect user responses
    llm,                            # Language model processing
    tts,                            # Text-to-speech conversion
    transport.output(),             # Sends audio to user
    context_aggregator.assistant(), # Collect assistant responses
])
```

## Understanding Frames and Frame Processing

Before diving deeper into pipelines, it's important to understand how data moves through them using **frames** and **frame processors**.

### What Are Frames?

**Frames** are data containers that carry information through your pipeline. Think of them as packages on a conveyor belt. Each frame contains a specific type of data that processors can examine and act upon.

Frames automatically receive unique identifiers and names (like `TranscriptionFrame#1`) that help with debugging and tracking data flow through your pipeline.

Common frame types include:

* **Audio frames**: Raw audio data from users or generated by TTS
* **Text frames**: Transcriptions, LLM responses, or other text content
* **Image frames**: Visual data for multimodal applications
* **System frames**: Control signals for pipeline management
* **Context frames**: Conversation history and state information

Frames flow through your pipeline from processor to processor, carrying the data your voice AI application needs to operate.

### What Are Frame Processors?

**Frame Processors** are the workers in your pipeline. Each processor has a specific job - like converting speech to text, generating responses, or playing audio. They:

* **Receive frames** from the previous processor in the pipeline
* **Process the data** (transcribe audio, generate text, etc.)
* **Create new frames** with their output
* **Pass frames along** to the next processor

Frame processors are modular and reusable. You can swap out different STT services or LLM providers without changing the rest of your pipeline.

### Frame Types

Frames in Pipecat have different base classes that determine how they're processed:

```python theme={null}
@dataclass
class SystemFrame(Frame):
    """System frames are processed immediately, bypassing queues."""
    pass

@dataclass
class DataFrame(Frame):
    """Data frames are queued and processed in order."""
    pass

@dataclass
class ControlFrame(Frame):
    """Control frames are queued and processed in order."""
    pass
```

**Key differences:**

* **SystemFrames**: Processed immediately (interruptions, pipeline control, user input)
* **DataFrames & ControlFrames**: Queued and processed in order (audio output, text, images)

**Examples by type:**

```python theme={null}
# SystemFrames (processed immediately)
InputAudioRawFrame         # User audio input
UserStartedSpeakingFrame   # Speech detection events
StartInterruptionFrame     # Interruption control
ErrorFrame                 # Error notifications

# DataFrames (queued and ordered)
OutputAudioRawFrame        # Audio for playback
TextFrame                  # Text content
TranscriptionFrame         # Speech-to-text results
LLMTextFrame               # LLM responses
AggregatedTextFrame        # Text aggregated into a describable unit
TTSTextFrame               # Text-to-speech text output

# ControlFrames (queued and ordered)
EndFrame                   # Pipeline shutdown
TTSStartedFrame            # TTS response boundaries
LLMFullResponseStartFrame  # LLM response boundaries
```

### Frame Processing Order

**Frames are processed in guaranteed order**, even across ParallelPipelines. This enables reliable sequencing. For example, you can push two frames in order and the order will be respected. Additionally, the corresponding processing will finish before allowing the next frame to be processed. Let's look at an example where we push two frames—`TTSSpeakFrame` and `EndFrame`—in order to say goodbye then end the pipeline:

```python theme={null}
from pipecat.frames.frames import EndFrame, TTSSpeakFrame

# These will execute in order: speak first, then end pipeline
await task.queue_frames([
    TTSSpeakFrame("Goodbye!"),
    EndFrame()
])
```

### How Frame Processors Work

Every frame processor follows the same pattern with two key methods:

```python theme={null}
class TranscriptionLogger(FrameProcessor):
    async def process_frame(self, frame: Frame, direction: FrameDirection):
        # Always call parent first
        await super().process_frame(frame, direction)

        # Handle specific frame types
        if isinstance(frame, TranscriptionFrame):
            print(f"Transcription: {frame.text}")

        # Push frame to next processor
        await self.push_frame(frame, direction)
```

**Key methods:**

* **`process_frame()`**: Inspect and handle incoming frames
* **`push_frame()`**: Send frames upstream or downstream

<Card title="Custom Frame Processors" icon="code" href="/guides/fundamentals/custom-frame-processor">
  Learn how to build your own frame processors
</Card>

## How Data Flows Through Pipelines

Understanding data flow is crucial for building effective pipelines:

### Frame Processing Order

**Order matters**: Processors must be arranged so that each receives the frame types it needs:

1. `transport.input()` creates `InputAudioRawFrame`s from user audio
2. `stt` receives audio frames and outputs `TranscriptionFrame`s
3. `llm` processes text and generates `LLMTextFrame`s
4. `tts` converts text frames to `TTSAudioRawFrame`s, `AggregatedTextFrame`s, and `TTSTextFrame`s
5. `transport.output()` creates `OutputAudioRawFrame`s and sends audio back to user

* Note: An `LLMTextProcessor` can sit between the `llm` and `tts` to pre-aggregate `LLMTextFrame`s into `AggregatedTextFrame`s. This simply moves the aggregation step
  out of the TTS.

### Frame Propagation

**Processors always push frames**: Processors don't consume frames, they pass them along:

```python theme={null}
# This pipeline allows multiple processors to use the same audio
pipeline = Pipeline([
    transport.input(),          # Creates InputAudioRawFrame
    stt,                        # Uses audio → creates TranscriptionFrame
    # ...                       # Other processors can use the same audio
    tts,                        # Uses various text frames → creates TTSAudioRawFrame
    transport.output(),         # Uses TTSAudioRawFrame → sends audio to user
    audio_buffer_processor,     # Also uses the same user and assistant audio for recording
])
```

This design allows multiple processors to operate on the same data stream without interfering with each other.

### Parallel Processing Patterns

Use `ParallelPipeline` to create branches where each branch receives all upstream frames. Frames are collected and pushed individually from each branch:

```python theme={null}
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    ParallelPipeline([
        # English branch
        [FunctionFilter(english_filter), english_tts],
        # Spanish branch
        [FunctionFilter(spanish_filter), spanish_tts],
    ]),
    transport.output(),
    context_aggregator.assistant(),
])
```

In this example:

* Both TTS branches receive all LLM output
* Each branch can filter and process frames independently
* Results from both branches flow to `transport.output()`

ParallelPipelines are traditionally paired with filters or gates to control which frames go where, allowing for complex conditional logic.

### Frame Queuing and Processing

Frame processors have internal queues that ensure ordered processing:

* **SystemFrames bypass queues** for immediate processing (interruptions, errors)
* **DataFrames and ControlFrames are queued** and processed in order
* **Queuing is managed automatically** by the pipeline infrastructure
* **Order is guaranteed** even across complex pipeline structures

<Note>
  Learn more about frame flow patterns in the [Custom Frame Processor
  Guide](/guides/fundamentals/custom-frame-processor).
</Note>

## Pipeline Execution

### Creating a Pipeline Task

Wrap your pipeline in a `PipelineTask` to configure execution parameters:

```python theme={null}
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        enable_metrics=True,
        enable_usage_metrics=True,
        audio_in_sample_rate=8000,
        audio_out_sample_rate=8000,
    ),
    observers=[RTVIObserver(rtvi)],
)
```

### Running the Pipeline

Use `PipelineRunner` to execute your pipeline task:

```python theme={null}
# Create the runner
runner = PipelineRunner(handle_sigint=False)

# Run the pipeline
await runner.run(task)
```

**PipelineRunner features:**

* **Lifecycle Management**: Handles startup, execution, and cleanup
* **Signal Handling**: Responds to SIGINT/SIGTERM for graceful shutdown (when `handle_sigint=True`)
* **Resource Cleanup**: Ensures proper disposal of resources when tasks complete

**Common PipelineRunner options:**

```python theme={null}
# Handle system signals automatically
# Useful when running a file directly
runner = PipelineRunner(handle_sigint=True)

# Custom signal handling
# Useful when the PipelineRunner is managed by
# another system like the development runner
runner = PipelineRunner(handle_sigint=False)
```

### Pipeline Lifecycle

The pipeline runs continuously, processing frames as they arrive, until one of these conditions occurs:

1. **Normal Completion**: Bot code returns (session ends)
2. **Signal Termination**: SIGINT or SIGTERM received (if `handle_sigint=True`)
3. **Task Cancellation**: Explicit call to `task.cancel()`
4. **Error Condition**: Unhandled exception in pipeline

**Common lifecycle patterns:**

```python theme={null}
# Graceful shutdown on client disconnect
@transport.event_handler("on_client_disconnected")
async def on_client_disconnected(transport, client):
    logger.info("Client disconnected - stopping pipeline")
    await task.cancel()

# Await the pipeline task
await runner.run(task)
```

**Pipeline states during execution:**

* **Starting**: Initializing processors and connections
* **Running**: Actively processing frames and handling events
* **Stopping**: Gracefully shutting down processors
* **Stopped**: All resources cleaned up

## Configuring Pipeline Behavior

Beyond the basic pipeline structure, you can fine-tune behavior through PipelineParams and add monitoring capabilities:

### Pipeline Parameters

`PipelineParams` lets you configure how your entire pipeline operates:

```python theme={null}
params = PipelineParams(
    # Audio configuration (affects all audio processors)
    audio_in_sample_rate=16000,
    audio_out_sample_rate=24000,

    # Performance monitoring
    enable_metrics=True,
    enable_usage_metrics=True,

    # Debugging and development
    send_initial_empty_metrics=False,
    report_only_initial_ttfb=True,
)
```

### Monitoring and Observability

Add observers to monitor pipeline events and track custom application metrics:

```python theme={null}
task = PipelineTask(
    pipeline,
    params=params,
    observers=[
        RTVIObserver(rtvi),           # RTVI protocol events
        CustomMetricsObserver(),      # Your custom metrics
    ],
)
```

These configuration options enable you to optimize performance, debug issues, and monitor your pipeline in production:

<CardGroup>
  <Card title="Pipeline Parameters" icon="screwdriver-wrench" href="/server/pipeline/pipeline-params">
    Configure execution parameters, audio settings, and debugging options
  </Card>

  <Card title="Metrics Guide" icon="chart-line" href="/guides/fundamentals/metrics">
    Monitor pipeline performance, latency, and API usage in real-time
  </Card>

  <Card title="Observer Pattern" icon="eye" href="/server/utilities/observers/observer-pattern">
    Monitor pipeline events, frame processing, and custom application events
  </Card>
</CardGroup>

## Key Takeaways

* **Order matters** - arrange processors so each gets the frames it needs
* **Processors push frames** - processors pass frames downstream, not consume them
* **Frame types determine processing** - SystemFrames bypass queues, others are ordered
* **Queuing ensures reliability** - frames are processed in guaranteed order
* **Parallel processing** enables conditional logic and multi-modal handling
* **PipelineTask** configures execution parameters and monitoring
* **PipelineRunner** manages the complete pipeline lifecycle
* **PipelineParams** control pipeline-wide behavior like audio settings and metrics

## What's Next

Now that you understand how pipelines orchestrate processing, let's explore the different transport options that connect your pipeline to users.

<Card title="Transports" icon="arrow-right" href="/guides/learn/transports">
  Learn about the different ways users can connect to your voice AI pipeline
</Card>


# Pipeline Termination
Source: https://docs.pipecat.ai/guides/learn/pipeline-termination

Learn how to properly terminate Pipecat pipelines for clean shutdown and resource management

**Pipeline termination** ensures your voice AI applications shut down cleanly without resource leaks or hanging processes. Understanding the different termination methods helps you handle various scenarios from natural conversation endings to unexpected disconnections.

## Pipeline Integration

Pipeline termination works through the same frame-based system as other pipeline operations:

```python theme={null}
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),
    context_aggregator.assistant(),
])
# EndFrame or CancelFrame flows through entire pipeline for shutdown
```

**Termination frames:**

* **`EndFrame`**: A queued ControlFrame that triggers graceful shutdown after processing pending frames
* **`CancelFrame`**: A SystemFrame that triggers immediate shutdown, discarding pending frames

Both frames flow downstream through the pipeline, allowing each processor to clean up resources appropriately.

## Termination Methods

Pipecat provides two primary approaches for pipeline termination, each designed for different scenarios:

### 1. Graceful Termination

Graceful termination allows the bot to complete its current processing before shutting down. This is ideal when you want the bot to properly end a conversation. For example, after completing a specific task or reaching a natural conclusion.

**When to use:**

* Natural conversation endings
* Task completion scenarios
* When the bot should say goodbye

**Implementation options:**

Push an `EndFrame` from outside your pipeline:

```python theme={null}
# From outside the pipeline
from pipecat.frames.frames import EndFrame, TTSSpeakFrame

await task.queue_frame(EndFrame())
```

Push an `EndTaskFrame` upstream from inside your pipeline:

```python theme={null}
# From inside a function call
from pipecat.frames.frames import EndTaskFrame, TTSSpeakFrame
from pipecat.processors.frame_processor import FrameDirection

async def end_conversation(params: FunctionCallParams):
    await params.llm.push_frame(TTSSpeakFrame("Have a nice day!"))

    # Signal that the task should end after processing this frame
    await params.llm.push_frame(EndTaskFrame(), FrameDirection.UPSTREAM)
```

**How graceful termination works:**

1. `EndFrame` is queued and processes after any pending frames (like goodbye messages)
2. All processors shutdown when they receive the `EndFrame`
3. Once the `EndFrame` reaches the end of the pipeline, shutdown is complete
4. Resources are cleaned up and the process terminates

<Tip>
  Graceful termination allows your bot to say goodbye and complete any final
  actions before terminating.
</Tip>

### 2. Immediate Termination

Immediate termination cancels the pipeline without waiting for pending frames to complete. This is appropriate when the user is no longer active in the conversation.

**When to use:**

* User disconnections (browser closed, call ended)
* Error conditions requiring immediate shutdown
* When completing the conversation is no longer necessary

**Implementation:**

Use event handlers to detect disconnections and trigger cancellation:

```python theme={null}
@transport.event_handler("on_client_disconnected")
async def on_client_disconnected(transport, client):
    logger.info("Client disconnected - terminating pipeline")
    await task.cancel()
```

**How immediate termination works:**

1. An event triggers the cancellation (like client disconnection)
2. `task.cancel()` pushes a `CancelFrame` downstream from the PipelineTask
3. `CancelFrame`s are `SystemFrame`s and bypass queues for immediate processing
4. Processors handle the `CancelFrame` and shut down immediately
5. Any pending frames are discarded during shutdown

<Warning>
  Immediate termination will discard any pending frames in the pipeline. Use
  this approach when completing the conversation is no longer necessary.
</Warning>

## Automatic Termination

### Pipeline Idle Detection

Pipecat includes automatic idle detection to prevent hanging pipelines. This feature monitors activity and can automatically cancel tasks when no meaningful bot interactions occur for an extended period.

**How it works:**

* Monitors pipeline activity for meaningful bot interactions
* Automatically triggers termination after configured idle timeout
* Serves as a safety net for anomalous behavior or forgotten sessions

**Configuration:**

```python theme={null}
task = PipelineTask(
    pipeline,
    # Configure idle detection timeout
    cancel_on_idle_timeout=True, # Default is True
    idle_timeout_seconds=600,  # Default is 300 seconds
    idle_timeout_frames=(BotSpeakingFrame,), # Default is (BotSpeakingFrame, LLMFullResponseEndFrame)
)
```

You can further configure the idle detection behavior. To learn more, refer to the Pipeline Idle Detection documentation:

<Card title="Pipeline Idle Detection" icon="clock" href="/server/pipeline/pipeline-idle-detection">
  Learn how to configure and customize idle detection for your use case
</Card>

<Note>
  Pipeline Idle Detection is enabled by default and helps prevent resources from
  being wasted on inactive conversations.
</Note>

## Implementation Patterns

### Event-Driven Termination

Connect termination to transport events for automatic cleanup:

```python theme={null}
@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
    logger.info("Client connected - starting conversation")
    await task.queue_frames([LLMRunFrame()])

@transport.event_handler("on_client_disconnected")
async def on_client_disconnected(transport, client):
    logger.info("Client disconnected - immediate termination")
    await task.cancel()

# Run the pipeline
runner = PipelineRunner(handle_sigint=False)
await runner.run(task)
```

### Conditional Termination

Use function calling or other logic to determine when conversations should end:

```python theme={null}
async def check_conversation_complete(params: FunctionCallParams):
    # Your logic to determine if conversation should end
    conversation_complete = await evaluate_completion_criteria()

    if conversation_complete:
        await params.llm.push_frame(TTSSpeakFrame("Thank you for using our service!"))
        await params.llm.push_frame(EndTaskFrame(), FrameDirection.UPSTREAM)

    await params.result_callback({"status": "complete" if conversation_complete else "continuing"})
```

### Error Handling

Ensure pipelines can terminate properly even when exceptions occur:

```python theme={null}
try:
    runner = PipelineRunner(handle_sigint=False)
    await runner.run(task)
except Exception as e:
    logger.error(f"Pipeline error: {e}")
    # Ensure cleanup happens even on errors
    await task.cancel()
```

## Troubleshooting

If your pipeline isn't shutting down properly, check these common issues:

### Custom Processors Not Propagating Frames

**Problem:** Custom processors that don't call `push_frame()` can block termination frames from reaching the end of the pipeline.

**Solution:** Ensure your custom processors propagate all frames downstream, including `EndFrame` and `CancelFrame`:

```python theme={null}
async def process_frame(self, frame: Frame, direction: FrameDirection):
    await super().process_frame(frame, direction)

    # Your custom processing logic here

    # Always push frames downstream (including termination frames)
    await self.push_frame(frame, direction)
```

### Incorrect Termination Frame Direction

**Problem:** Pushing `EndFrame` or `CancelFrame` from the middle of the pipeline may not reach the pipeline source properly.

**Solution:** Use the appropriate frame type and direction:

```python theme={null}
# From inside the pipeline, push upstream to reach the source
await self.push_frame(EndTaskFrame(), FrameDirection.UPSTREAM)
await self.push_frame(CancelTaskFrame(), FrameDirection.UPSTREAM)

# The pipeline source will then convert these to proper termination frames
# and push them downstream through the entire pipeline
```

<Note>
  The pipeline source automatically converts `EndTaskFrame` to `EndFrame` and
  `CancelTaskFrame` to `CancelFrame` when pushing downstream, ensuring proper
  termination handling throughout the pipeline.
</Note>

## Key Takeaways

* **Frame-based termination** - shutdown uses the same frame system as processing
* **Choose the right method** - graceful for natural endings, immediate for disconnections
* **Event handlers enable automatic termination** - respond to user disconnections cleanly
* **Idle detection provides safety net** - prevents hanging processes and resource waste
* **SystemFrames bypass queues** - CancelFrames process immediately for fast shutdown
* **Resource cleanup is automatic** - proper termination ensures clean resource disposal

## What's Next

You now understand how to build, run, and properly terminate voice AI pipelines! Next, let's explore advanced development tools that can accelerate your Pipecat development process.

<Card title="Next Steps" icon="arrow-right" href="/guides/learn/whats-next">
  You've completed the learning guide! Continue to see recommended next steps.
</Card>


# Session Initialization
Source: https://docs.pipecat.ai/guides/learn/session-initialization

Learn how to set up connections between users and your Pipecat voice AI bot

Before your voice AI bot can start processing audio and generating responses, you need to establish a connection between the user and your bot. This process is called **session initialization** - it's how users and bots find each other and establish a communication channel for real-time audio exchange.

## Understanding the Architecture

Session initialization involves multiple components working together:

* **Runner**: A FastAPI server that handles incoming connection requests and manages session setup
* **Pipecat Bot**: Your voice AI application running as a separate server-side service
* **Client Application**: The user-facing app (web browser, mobile app, etc.)

The runner acts as the coordinator, setting up the necessary resources and starting bot instances, while the Pipecat bot handles the actual voice AI processing.

## Development Runner

For most development and many production use cases, Pipecat provides a **development runner** that handles all the session initialization complexity for you. Instead of building FastAPI servers and managing WebRTC connections yourself, you focus on your bot logic while the runner handles the infrastructure.

### Using the Development Runner

Your bot needs a single entry point function that the runner will call:

```python theme={null}
from pipecat.runner.types import RunnerArguments

async def bot(runner_args: RunnerArguments):
    """Main bot entry point called by the development runner."""

    # Create your transport based on the runner arguments
    transport = SmallWebRTCTransport(
        params=TransportParams(
            audio_in_enabled=True,
            audio_out_enabled=True,
        ),
        webrtc_connection=runner_args.webrtc_connection,
    )

    # Run your bot logic
    await run_bot(transport)

if __name__ == "__main__":
    from pipecat.runner.run import main
    main()
```

Then start your bot with different connection types:

```bash theme={null}
# P2P WebRTC (opens browser interface)
python bot.py -t webrtc

# Daily room-based WebRTC
python bot.py -t daily

# Telephony (requires ngrok or similar)
python bot.py -t twilio -x your_domain.ngrok.io
```

where `-t` specifies the transport type (e.g., `webrtc`, `daily`, `twilio`) and `-x` is the optional proxy domain for telephony.

The development runner automatically:

* Creates the FastAPI server
* Sets up the appropriate endpoints
* Handles connection management
* Starts your bot instances
* Provides a web interface (for WebRTC)

<Tip>
  Learn more about building with the development runner in the [runner
  guide](/server/utilities/runner/guide).
</Tip>

## Connection Types Under the Hood

While the development runner handles the complexity, understanding the three connection patterns helps you choose the right approach and debug issues:

### 1. P2P WebRTC Connections

**What happens:**

1. Runner serves a web interface at `http://localhost:7860/client`
2. When you open the page and connect, browser creates a WebRTC offer
3. Runner receives the offer, establishes connection, starts your bot
4. Browser and bot communicate directly via WebRTC

**When to use:** Direct client connections, embedded applications, local development

### 2. Room-Based WebRTC (Daily)

**What happens:**

<Steps>
  <Step title="Room Request">
    User visits the client application and clicks to start a session
  </Step>

  <Step title="Room Creation">
    Runner calls Daily's API to create a room and tokens using
    `pipecat.runner.daily.configure()`
  </Step>

  <Step title="Parallel Join">
    Both user's browser and your bot join the same Daily room
  </Step>

  <Step title="Media Handshake">
    Once media streams are established, browser sends `client_ready` message
  </Step>

  <Step title="Bot Activation">
    Your bot receives the event and starts the conversation
  </Step>
</Steps>

**When to use:** Video calls, group sessions, production deployments

<Note>
  Room-based WebRTC can also be used for SIP or PSTN connections, which require
  different connection patterns. Refer to the [telephony
  guide](/guides/telephony/overview) for details.
</Note>

### 3. WebSocket Connections (Telephony)

**What happens:**

1. Telephony provider (Twilio, etc.) receives a phone call
2. Provider connects to your runner's webhook endpoint
3. Runner accepts WebSocket connection and parses telephony-specific messages
4. Your bot starts immediately with the parsed connection data

**When to use:** Phone bots, telephony integrations

## Starting Conversations

How and when your bot begins talking depends on the connection type:

### Immediate Start (P2P WebRTC, WebSocket)

These connections are ready immediately, so you can start talking right after connection:

```python theme={null}
@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
    logger.info("Client connected - starting conversation")
    messages.append({
        "role": "system",
        "content": "Say hello and introduce yourself."
    })
    await task.queue_frames([LLMRunFrame()])
```

### Handshake Required (Client/Server Room-based WebRTC)

For client/server applications using room-based WebRTC, a handshake ensures both sides are ready and the client won't miss the opening message:

```python theme={null}
@rtvi.event_handler("on_client_ready")
async def on_client_ready(rtvi):
    await rtvi.set_bot_ready()  # Confirm readiness to client
    # Start the conversation
    await task.queue_frames([LLMRunFrame()])
```

<Note>
  For client/server room-based connections, waiting for `on_client_ready` is
  crucial - starting too early can cause the client to miss part of the initial
  message.
</Note>

## Process Isolation

Each session runs its own dedicated bot instance for:

* **Resource Management**: Dedicated CPU and memory per session
* **Error Isolation**: One session crash doesn't affect others
* **Clean Cleanup**: Resources automatically freed when sessions end

The development runner handles this process management automatically.

## Custom Runners: When You Need More Control

The development runner works for most cases, but sometimes you need custom behavior - specific authentication, custom endpoints, or integration with existing systems. For these cases, you can create your own FastAPI runner.

The development runner source code ([available on GitHub](https://github.com/pipecat-ai/pipecat/blob/main/src/pipecat/runner/run.py)) provides excellent examples for:

**Daily Integration Example:**

```python theme={null}
from pipecat.runner.daily import configure

@app.post("/start")
async def start_bot(background_tasks: BackgroundTasks):
    async with aiohttp.ClientSession() as session:
        room_url, token = await configure(session)

        # Start bot instance
        background_tasks.add_task(run_bot, room_url, token)

        return {"room_url": room_url, "token": token}
```

**WebSocket Telephony Example:**

```python theme={null}
from pipecat.runner.utils import parse_telephony_websocket

@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
    await websocket.accept()

    # Parse provider-specific messages
    transport_type, call_data = await parse_telephony_websocket(websocket)

    # Start bot with parsed data
    await run_telephony_bot(websocket, transport_type, call_data)
```

<Tip>
  Refer to the development runner source code to understand these patterns
  before building custom runners. It handles many edge cases and provides
  battle-tested implementations.
</Tip>

## Key Takeaways

* **Start with the development runner** for fastest development and learning
* **Understand connection types** to choose the right approach for your use case
* **Handle startup timing correctly** - immediate start vs. handshake patterns matter
* **Plan for process isolation** - one bot instance per session is the recommended pattern
* **Reference the source code** when building custom runners for production

## What's Next

Now that you understand session initialization, let's explore the different transport options and how to configure them for your specific needs.

<Card title="Pipeline & Frame Processing" icon="arrow-right" href="/guides/learn/pipeline">
  Learn how Pipecat's pipeline architecture orchestrates frame processing for
  voice AI applications
</Card>


# Speech Input & Turn Detection
Source: https://docs.pipecat.ai/guides/learn/speech-input

Learn how Pipecat detects user turns using VAD, transcriptions, and turn detection models

A key to natural conversations is properly detecting when the user starts and stops speaking. This is more nuanced than simply detecting audio; a brief pause doesn't always mean the user is done talking.

## Overview

Pipecat uses [user turn strategies](/server/utilities/turn-management/user-turn-strategies) to determine when user turns start and end. These strategies can use different techniques:

**For detecting turn start:**

* Voice Activity Detection (VAD): triggers when speech is detected
* Transcription-based (fallback): triggers when transcription is received but VAD didn't detect speech
* Minimum words: waits for a minimum number of spoken words before triggering

**For detecting turn end:**

* Transcription-based: analyzes transcription to determine when the user is done
* Turn detection model: uses AI to understand if the user has finished their thought

Custom strategies can also be implemented for specific use cases. By combining these techniques, you can create responsive yet natural conversations that don't interrupt users mid-sentence or wait too long after they've finished.

## Voice Activity Detection (VAD)

### What VAD Does

VAD is responsible for detecting when a user starts and stops speaking. Pipecat uses the [Silero VAD](https://github.com/snakers4/silero-vad), an open-source model that runs locally on CPU with minimal overhead.

**Performance characteristics:**

* Processes 30+ms audio chunks in less than 1ms
* Runs on a single CPU thread
* Minimal system resource impact

### VAD Configuration

VAD is configured through `VADParams` in the `LLMContextAggregatorPair`:

```python theme={null}
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.audio.vad.vad_analyzer import VADParams
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)

vad_analyzer = SileroVADAnalyzer(
    params=VADParams(
        confidence=0.7,      # Minimum confidence for voice detection
        start_secs=0.2,      # Time to wait before confirming speech start
        stop_secs=0.8,       # Time to wait before confirming speech stop
        min_volume=0.6,      # Minimum volume threshold
    )
)

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(vad_analyzer=vad_analyzer)),
)
```

### Key Parameters

**`start_secs` (default: 0.2)**

* How long a user must speak before VAD confirms speech has started
* Lower values = more responsive, but may trigger on brief sounds
* Higher values = less sensitive, but may miss quick utterances, like "yes", "no", or "ok"

**`stop_secs` (default: 0.8)**

* How much silence must be detected before confirming speech has stopped
* Critical for turn-taking behavior
* Modified automatically when using turn detection

**`confidence` and `min_volume`**

* Generally work well with defaults
* Only adjust after extensive testing with your specific audio conditions

<Warning>
  Changing confidence and min\_volume requires careful profiling to ensure
  optimal performance across different audio environments and use cases.
</Warning>

## User Turn Detection

While VAD detects speech vs. silence, it can't understand linguistic context. A pause doesn't mean the user is done. User turn strategies interpret VAD signals and transcriptions to determine actual turn boundaries.

### How It Works

1. **Turn Start**: When VAD detects speech (or transcription arrives), the start strategy emits `UserStartedSpeakingFrame` and optionally triggers an interruption
2. **Turn End**: When the stop strategy determines the user is done, it emits `UserStoppedSpeakingFrame`

<Note>
  VAD also emits its own frames (`VADUserStartedSpeakingFrame`,
  `VADUserStoppedSpeakingFrame`) which indicate raw speech/silence detection.
  These are inputs to the turn strategies, not the final turn decisions.
</Note>

### Detecting Turn End

Turn end detection uses transcription to understand what the user said:

**Transcription-based (Default)**: Analyzes transcription to determine when the user has finished speaking.

```python theme={null}
from pipecat.turns.user_stop import TranscriptionUserTurnStopStrategy

stop_strategy = TranscriptionUserTurnStopStrategy()
```

**Smart Turn Model**: Uses an AI model to analyze audio and determine if the user has finished their thought.

```python theme={null}
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy

stop_strategy = TurnAnalyzerUserTurnStopStrategy(
    turn_analyzer=LocalSmartTurnAnalyzerV3()
)
```

When using Smart Turn, configure VAD with a low `stop_secs` (0.2) so the model can analyze speech quickly.

<CardGroup>
  <Card title="User Turn Strategies" icon="arrows-turn-to-dots" href="/server/utilities/turn-management/user-turn-strategies">
    Complete reference for start and stop strategies
  </Card>

  <Card title="Smart Turn Overview" icon="brain" href="/server/utilities/smart-turn/smart-turn-overview">
    Smart Turn model implementation guide
  </Card>
</CardGroup>

### Interruptions

Interruptions stop the bot when the user starts speaking. This is controlled by the `enable_interruptions` parameter on start strategies (enabled by default).

When a user turn starts with interruptions enabled:

1. Bot immediately stops speaking
2. Pending audio and text is cleared
3. Pipeline ready for new user input

To disable interruptions:

```python theme={null}
from pipecat.turns.user_start import VADUserTurnStartStrategy

start_strategy = VADUserTurnStartStrategy(enable_interruptions=False)
```

<Note>
  Keep interruptions enabled (default) for natural conversations. This enables
  users to interrupt the bot mid-sentence, just like human conversations.
</Note>

## Best Practices

### Optimal Configuration

**For most voice AI use cases:**

```python theme={null}
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies

# Responsive VAD with turn detection
vad_params = VADParams(
    start_secs=0.2,
    stop_secs=0.2 if using_turn_detection else 0.8,
)

transport = YourTransport(
    params=TransportParams(),
)

# Configure Smart Turn via user turn strategies (if using turn detection)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_turn_strategies=UserTurnStrategies(
            stop=[TurnAnalyzerUserTurnStopStrategy(
                turn_analyzer=LocalSmartTurnAnalyzerV3()
            )]
        ) if using_smart_turn_detection else None,
        vad_analyzer=SileroVADAnalyzer(params=vad_params),
    ),
)

task = PipelineTask(pipeline)
```

### Performance Considerations

* **Use local VAD**: 150-200ms faster than remote VAD services
* **Tune for your use case**: Test with real audio conditions
* **Monitor CPU usage**: VAD adds minimal overhead but monitor in production
* **Consider turn detection**: Improves conversation quality but adds complexity

## Key Takeaways

* **VAD detects speech activity** but turn detection understands conversation context
* **Configuration affects user experience** - tune parameters for your specific use case
* **System frames coordinate behavior** - enable interruptions and natural turn-taking
* **Local processing is faster** - Silero VAD provides low-latency speech detection
* **Turn detection improves quality** - but requires careful VAD configuration

## What's Next

Now that you understand how speech input is detected and processed, let's explore how that audio gets converted to text through speech recognition.

<Card title="Speech to Text" icon="arrow-right" href="/guides/learn/speech-to-text">
  Learn how to configure speech recognition in your voice AI pipeline
</Card>


# Speech to Text
Source: https://docs.pipecat.ai/guides/learn/speech-to-text

Learn how to configure speech recognition to convert user audio into text in your Pipecat pipeline

**Speech to Text (STT)** services are responsible for converting user audio into text transcriptions. They receive audio input from users and provide real-time transcriptions that your bot can process and respond to.

## Pipeline Placement

STT processors must be positioned correctly in your pipeline to receive and process audio frames:

```python theme={null}
pipeline = Pipeline([
    transport.input(),             # Creates InputAudioRawFrames
    stt,                           # Processes audio → creates TranscriptionFrames
    context_aggregator.user(),     # Uses transcriptions for context
    llm,
    tts,
    transport.output(),
])
```

**Placement requirements:**

* **After `transport.input()`**: STT needs `InputAudioRawFrame`s from the transport
* **Before context processing**: Transcriptions must be available for context aggregation
* **Before LLM processing**: Text must be ready for language model input

## STT Service Types

Pipecat provides two types of STT services based on how they process audio:

### 1. STTService (Streaming)

**How it works:**

* Establishes a WebSocket connection to the STT provider
* Continuously streams audio for real-time transcription
* Lower latency due to persistent connection

### 2. SegmentedSTTService (HTTP-based)

**How it works:**

* Uses local VAD (Voice Activity Detection) to chunk speech
* Sends audio segments to STT service as wav files
* Higher latency due to segmentation and HTTP POST requests

<Note>
  STT services are modular and can be swapped out with no additional overhead.
  You can easily switch between streaming and segmented services based on your
  needs.
</Note>

## Supported STT Services

Pipecat supports a wide range of STT providers to fit different needs and budgets:

<Card title="Supported STT Services" icon="list" href="/server/services/supported-services#speech-to-text">
  View the complete list of supported speech-to-text providers
</Card>

Popular options include:

<CardGroup>
  <Card title="Deepgram" icon="microphone" href="/server/services/stt/deepgram">
    Fast, accurate streaming STT with excellent real-time performance
  </Card>

  <Card title="Speechmatics" icon="waveform" href="/server/services/stt/speechmatics">
    Advanced speech recognition with strong accent and dialect handling
  </Card>

  <Card title="AssemblyAI" icon="brain" href="/server/services/stt/assemblyai">
    AI-powered transcription with speaker diarization and sentiment analysis
  </Card>

  <Card title="Gladia" icon="sparkles" href="/server/services/stt/gladia">
    High-performance STT with multilingual support and custom models
  </Card>

  <Card title="Azure Speech" icon="microsoft" href="/server/services/stt/azure">
    Microsoft's enterprise-grade STT service with extensive language support
  </Card>

  <Card title="Google Speech-to-Text" icon="google" href="/server/services/stt/google">
    Reliable transcription with strong language model integration
  </Card>
</CardGroup>

## STT Configuration

### Service-Specific Configuration

Each STT service has its own customization options. Refer to specific service documentation for details:

<Card title="Individual STT Services" icon="screwdriver-wrench" href="/server/services/supported-services#speech-to-text">
  Explore configuration options for each supported STT provider
</Card>

For example, let's look at configuring the **DeepgramSTTService** using the `LiveOptions` class:

```python theme={null}
from deepgram import LiveOptions
from pipecat.services.deepgram.stt import DeepgramSTTService
from pipecat.transcriptions.language import Language

# Configure using LiveOptions for full control
live_options = LiveOptions(
    model="nova-2",
    language=Language.EN_US,
    interim_results=True,        # Enable interim transcripts
    smart_format=True,           # Enable punctuation and formatting
    punctuate=True,              # Add punctuation
    profanity_filter=True,       # Filter profanity
    vad_events=False,            # Use pipeline VAD instead
)

stt = DeepgramSTTService(
    api_key=os.getenv("DEEPGRAM_API_KEY"),
    live_options=live_options,
)
```

### STTService Base Class Configuration

All STT services inherit from the STTService base class. The base class has base configuration options which are set with smart defaults:

```python theme={null}
stt = YourSTTService(
    # Service-specific options...
    audio_passthrough=True,      # Pass audio frames downstream (recommended)
    sample_rate=16000,           # Audio sample rate (better set in PipelineParams)
)
```

**Key options:**

* **`audio_passthrough=True`**: Allows audio frames to continue downstream to other processors (like audio recording)
* **`sample_rate`**: Audio sampling rate - best practice is to **set the `audio_in_sample_rate` in `PipelineParams` for consistency**

<Warning>
  Setting `audio_passthrough=False` will stop audio frames from being passed
  downstream, which may break audio recording or other audio-dependent
  processors.
</Warning>

### Pipeline-Level Audio Configuration

Instead of setting sample rates on individual services, configure them pipeline-wide:

```python theme={null}
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        audio_in_sample_rate=16000,   # All input processors use this rate
        audio_out_sample_rate=24000,  # All output processors use this rate
    ),
)
```

This ensures all audio processors use consistent sample rates without manual configuration.

<Tip>
  Always set audio sample rates in `PipelineParams` to avoid mismatches between
  different audio processors. This simplifies configuration and ensures
  consistent audio quality across your pipeline.
</Tip>

## Best Practices

### Enable Interim Results

When available, enable interim transcripts for better user experience:

```python theme={null}
stt = DeepgramSTTService(
    api_key=os.getenv("DEEPGRAM_API_KEY"),
    live_options=LiveOptions(
      interim_results=True,
    )
)
```

**Benefits:**

* Notifies context aggregation that more text is coming
* Prevents premature LLM completions
* Enables interruption detection
* Improves conversation flow

### Enable Punctuation and Formatting

Use smart formatting when available:

```python theme={null}
stt = DeepgramSTTService(
    api_key=os.getenv("DEEPGRAM_API_KEY"),
    live_options=LiveOptions(
        smart_format=True,     # Adds punctuation and capitalization
        profanity_filter=True, # Optional content filtering
    )
)
```

**Benefits:**

* Professional-looking transcripts
* Better LLM comprehension
* Eliminates post-processing needs
* Improved context understanding

### Use Local VAD

While many STT services provide Voice Activity Detection, use Pipecat's local Silero VAD for better performance:

```python theme={null}
from pipecat.audio.vad.silero import SileroVADAnalyzer

# Configure in context aggregator
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
    ),
)
```

**Advantages:**

* **150-200ms faster** speech detection (no network round trip)
* More responsive conversation flow
* Better interruption handling
* Reduced latency overall

## Key Takeaways

* **Pipeline placement matters** - STT must come after transport input, before context processing
* **Service types differ** - streaming services have lower latency than segmented
* **Services are modular** - easily swap providers without code changes
* **Best practices improve performance** - use interim results, formatting, and local VAD
* **Configuration affects quality** - proper setup significantly impacts transcription accuracy

## What's Next

Now that you understand speech recognition, let's explore how to manage conversation context and memory in your voice AI bot.

<Card title="Context Management" icon="arrow-right" href="/guides/learn/context-management">
  Learn how to handle conversation history and context in your pipeline
</Card>


# Text to Speech
Source: https://docs.pipecat.ai/guides/learn/text-to-speech

Learn how to configure speech synthesis to convert text into natural-sounding audio in your voice AI pipeline

**Text to Speech (TTS)** services are responsible for converting text into natural-sounding speech audio. They receive text input from LLMs and other sources, then generate audio output that users can hear through their connected devices.

## Pipeline Placement

TTS processors must be positioned correctly in your pipeline to receive text and generate audio frames:

```python theme={null}
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,                           # Generates LLMTextFrames
    tts,                           # Processes text → creates TTSAudioRawFrames
    transport.output(),            # Sends audio to user
    context_aggregator.assistant(), # Processes TTSTextFrames for context
])
```

**Placement requirements:**

* **After LLM processing**: TTS needs `LLMTextFrame`s from language model responses
* **Before transport output**: Audio must be generated before sending to user
* **Before assistant context aggregator**: Ensures spoken text is captured in conversation history

### Frame Processing Flow

**TTS generates speech through two primary mechanisms:**

1. **Streamed LLM tokens** via `LLMTextFrame`s:

   * TTS aggregates streaming tokens into complete sentences
   * Sentences are sent to TTS service for audio generation
   * Audio bytes stream back and play immediately
   * End-to-end latency often under 200ms

2. **Direct speech requests** via `TTSSpeakFrame`s:
   * Bypasses LLM and context aggregators
   * Immediate audio generation for specific text
   * Useful for system messages or prompts

**Frame output:**

* `TTSAudioRawFrame`s: Raw audio data for playback
* `TTSTextFrame`s: Text that was actually spoken (for context updates)
* `TTSStartedFrame`/`TTSStoppedFrame`: Speech boundary markers

## Supported TTS Services

Pipecat supports a wide range of TTS providers with different capabilities and performance characteristics:

<Card title="Supported TTS Services" icon="list" href="/server/services/supported-services#text-to-speech">
  View the complete list of supported text-to-speech providers
</Card>

### Service Categories

**WebSocket-Based Services (Recommended):**

* **Cartesia**: Ultra-low latency with word timestamps
* **ElevenLabs**: High-quality voices with emotion control
* **Rime**: Ultra-realistic voices with advanced features

**HTTP-Based Services:**

* **OpenAI TTS**: High-quality synthesis with multiple voices
* **Azure Speech**: Enterprise-grade with extensive language support
* **Google Text-to-Speech**: Reliable with WaveNet voices

**Advanced Features:**

* **Word timestamps**: Enable word-level accuracy for context and subtitles
* **Voice cloning**: Custom voice creation from samples
* **Emotion control**: Dynamic emotional expression
* **SSML support**: Fine-grained pronunciation control

<Note>
  WebSocket services typically provide the lowest latency, while HTTP services
  may have intermittent higher latency due to their request/response nature.
</Note>

## TTS Configuration

### Service-Specific Configuration

Each TTS service has its own configuration options. Here's an example with Cartesia:

```python theme={null}
from pipecat.services.cartesia.tts import CartesiaTTSService
from pipecat.transcriptions.language import Language

tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    voice_id="voice-id-here",
    model="sonic-2",              # TTS model to use
    params=CartesiaTTSService.InputParams(
        language=Language.EN,     # Speech language
        speed="normal",           # Speech rate control
    ),
    # Word timestamps automatically enabled for precise context updates
)
```

**Word timestamps:** Services like Cartesia, ElevenLabs, and Rime provide word-level timestamps that enable precise context updates during interruptions and better synchronization with other pipeline components. For example, if an interruption occurs while the bot is speaking, the word timestamps allow you to accurately capture which words were spoken up to that point, enabling better context management and user experience. Additionally, transcription events streamed from server to client can be done in sync with the audio output, allowing for real-time subtitles or captions.

<Card title="Individual TTS Services" icon="settings" href="/server/services/supported-services#text-to-speech">
  Explore configuration options for each supported TTS provider
</Card>

### Pipeline-Level Audio Configuration

Set consistent audio settings across your entire pipeline:

```python theme={null}
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        audio_in_sample_rate=16000,   # Input audio quality
        audio_out_sample_rate=24000,  # Output audio quality (TTS)
    ),
)
```

<Tip>
  Set the `audio_out_sample_rate` to match your TTS service's requirements for
  optimal quality. This is preferred to setting the sample\_rate directly in the
  TTS service as the PipelineParam ensures that all output sample\_rates match.
</Tip>

## Text Processing and Filtering

### Custom Text Aggregation

By default, TTS services have a built-in text aggregator that collects streaming text into sentences before passing them to the underlying service. However, you can customize this behavior by inserting an [`LLMTextProcessor`](/server/utilities/frame/llm-text-processor) with a different text aggregator before the TTS in your pipeline. This allows the ability to categorize and structure text into logical units beyond simple sentences, such as code blocks, URLs, or custom tags. You can then configure the TTS to handle these different text types appropriately, such as skipping code blocks or transforming them in a just-in-time manner before speaking.

### Skipping Text Aggregations

To skip certain text aggregations (e.g., code snippets or URLs) and keep them from being spoken, use a custom text aggregator like [`PatternPairAggregator`](/server/utilities/text/pattern-pair-aggregator) within an [`LLMTextProcessor`](/server/utilities/frame/llm-text-processor), and configure it to identify and handle specific patterns in the text stream. With this, you can then pass any aggregated types you want to skip (like "code") to the TTS service's `skip_aggregator_types` parameter.

```python theme={null}
# Create pattern aggregator
pattern_aggregator = PatternPairAggregator()

# Add pattern for JSON data
pattern_aggregator.add_pattern(
    type="code",
    start_pattern="<code>",
    end_pattern="</code>",
    action=MatchAction.AGGREGATE
)

# Set the aggregator on an LLMTextProcessor
llm_text_processor = LLMTextProcessor(text_aggregator=pattern_aggregator)

# Initialize TTS service, and don't speak JSON data
tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    skip_aggregator_types=["code"], # The strings here should match the types defined in the PatternPairAggregator
)

# add the llm_text_processor to your pipeline after the llm and before the tts
# llm -> llm_text_processor -> tts
```

### Text Transforms

For TTS-specific text preprocessing, you can provide custom text transforms that modify text in a just-in-time manner before sending the text off to the TTS service. This is useful for handling special text segments that need to be altered for better pronunciation or clarity, such as spelling out phone numbers, removing URLs, or expanding abbreviations. These text transforms can be mapped to a specific text aggregation type, like with `skip_aggregator_types`, or applied globally to all text using `'*'` as the type.

Text transforms are registered directly on the TTS service instance via the `add_text_transformer()` method or during initialization using the `text_transforms` parameter.

<Note>
  The intentions of text transforms are meant to be TTS-specific modifications
  that do not affect the underlying LLM text or context. That said, since the
  context aggregator attempts to base its context on what was actually spoken,
  for services that support word timestamps, like Cartesia, ElevenLabs, and
  Rime,these transforms will modify the context as they modify what is spoken.
</Note>

```python theme={null}
# Create pattern aggregator
pattern_aggregator = PatternPairAggregator()

# Add patterns for different parts of an explanation
pattern_aggregator.add_pattern(
    type="phone_number",
    start_pattern="<pnum>",
    end_pattern="</pnum>",
    action=MatchAction.AGGREGATE
)

# Set the aggregator on an LLMTextProcessor
llm_text_processor = LLMTextProcessor(text_aggregator=pattern_aggregator)

# Text-to-Speech service
tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
)

# Text transformers for TTS
# This will insert Cartesia's spell tags around the provided text.
async def spell_out_text(text: str, type: str) -> str:
    # CartesiaTTSService provides a helper for this along with other common transforms
    return CartesiaTTSService.SPELL(text)

async def replace_acronyms(text: str, type: str) -> str:
    # Replace "SEC" with "Southeastern Conference"
    return text.replace(" SEC ", " Southeastern Conference ")

# Setup the text transformers in TTS to spell out phone numbers and replace
# acronyms. The string below matches the type defined in the PatternPairAggregator
# above so that whenever those segments are encountered, this transform
# is applied
tts.add_text_transformer(spell_out_text, "phone_number")
tts.add_text_transformer(replace_acronyms, "*")  # Apply to all text

# add the llm_text_processor to your pipeline after the llm and before the tts
#   llm -> llm_text_processor -> tts
```

### Text Filters

<Warning>
  Text filters are no longer the preferred method for text preprocessing and
  will be deprecated in future releases. Instead, you should use one of the
  methods described above.
</Warning>

Apply preprocessing to text before synthesis:

```python theme={null}
from pipecat.utils.text.markdown_text_filter import MarkdownTextFilter

tts = YourTTSService(
    # ... other options
    text_filters=[
        MarkdownTextFilter(),     # Remove markdown formatting
        CustomTextFilter(),       # Your custom processing
    ],
)
```

**Common filters:**

* **MarkdownTextFilter**: Strips markdown formatting from LLM responses
* **Custom filters**: Implement your own text preprocessing logic

## Advanced TTS Features

### Direct Speech Commands

Use `TTSSpeakFrame` for immediate speech synthesis:

```python theme={null}
from pipecat.frames.frames import TTSSpeakFrame

# Make bot speak directly
await tts.queue_frame(TTSSpeakFrame("Hello, how can I help you?"))
```

### Dynamic Settings Updates

Update TTS settings during conversation:

```python theme={null}
from pipecat.frames.frames import TTSUpdateSettingsFrame

# Change voice speed during conversation
await task.queue_frames([
    TTSUpdateSettingsFrame({"speed": "fast"}),
    TTSSpeakFrame("I'm speaking faster now!")
])
```

## Key Takeaways

* **Pipeline placement matters** - TTS must come after LLM, before transport output
* **Service types differ** - WebSocket services provide lower latency than HTTP
* **Text processing affects quality** - use aggregation and filters for better results
* **Word timestamps enable precision** - better interruption handling and context accuracy
* **Configuration impacts performance** - balance quality, latency, and bandwidth needs
* **Services are modular** - easily swap providers without changing pipeline code

## What's Next

You've now learned how to build a complete voice AI pipeline! Let's explore some additional topics to enhance your implementation.

<Card title="Pipeline Termination" icon="arrow-right" href="/guides/learn/pipeline-termination">
  Learn how to terminate your voice AI pipeline at the end of a conversation
</Card>


# Transports
Source: https://docs.pipecat.ai/guides/learn/transports

Learn about the different ways users can connect to your Pipecat voice AI bot

**Transports** are the communication layer between users and your Pipecat bot. They handle receiving and sending audio, video, and data, serving as the media interface that enables real-time interaction.

## Available Transport Types

Pipecat supports multiple transport types to fit different use cases and deployment scenarios:

<CardGroup>
  <Card title="DailyTransport" icon="microphone" href="/server/services/transport/daily">
    WebRTC-based transport using Daily's infrastructure for video calls and
    conferencing
  </Card>

  <Card title="FastAPIWebsocketTransport" icon="plug" href="/server/services/transport/fastapi-websocket">
    WebSocket transport for telephony providers and custom WebSocket connections
  </Card>

  <Card title="HeyGenTransport" icon="camera" href="/server/services/transport/heygen">
    Specialized transport for HeyGen LiveAvatar video generation and streaming
  </Card>

  <Card title="LiveKitTransport" icon="microphone" href="/server/services/transport/livekit">
    WebRTC transport using LiveKit's real-time communication platform
  </Card>

  <Card title="SmallWebRTCTransport" icon="link" href="/server/services/transport/small-webrtc">
    Direct peer-to-peer WebRTC connections without cloud infrastructure
  </Card>

  <Card title="TavusTransport" icon="camera" href="/server/services/transport/tavus">
    Specialized transport for Tavus video generation and streaming
  </Card>

  <Card title="WebsocketTransport" icon="globe" href="/server/services/transport/websocket-server">
    General-purpose WebSocket transport for custom implementations
  </Card>
</CardGroup>

## Pipeline Integration

Transports provide two key components for your pipeline: `input()` and `output()` methods. These methods define how the transport interacts with the pipeline:

### Transport Input and Output

```python theme={null}
pipeline = Pipeline([
    transport.input(),              # Receives user audio/video
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),             # Sends bot audio/video
    context_aggregator.assistant(), # Processes after output
])
```

**Key points about transport placement:**

* **`transport.input()`** typically goes first in the pipeline to receive user input
* **`transport.output()`** doesn't always go last - you may want processors after it
* **Post-output processing** enables synchronized actions like:
  * Recording with word-level accuracy
  * Displaying subtitles synchronized to audio
  * Capturing context information precisely timed to output

## Transport Modularity

Transports are modular components in your Pipeline, allowing you to flexibly change how users connect to your bot depending on the context. This modularity enables you to:

* **Switch environments easily**: Use P2P WebRTC for development, Daily for production
* **Support multiple connection types**: Same bot logic works across different transports
* **Optimize for use case**: Choose the best transport for your specific requirements

## Transport Configuration

All transports are configured using `TransportParams`, which provides common settings across transport types:

```python theme={null}
from pipecat.transports.base_transport import TransportParams

params = TransportParams(
    # Audio settings
    audio_in_enabled=True,
    audio_out_enabled=True,

    # Video settings
    video_in_enabled=False,
    video_out_enabled=False,

    # Video stream configuration
    video_out_width=1024,
    video_out_height=576,
    video_out_bitrate=800000,
    video_out_framerate=30,
)
```

<Note>
  Each transport may have its own specialized parameters class that extends
  TransportParams with transport-specific options. Check the individual
  transport documentation for details.
</Note>

<Tip>
  For advanced turn detection (like Smart Turn), configure [User Turn
  Strategies](/server/utilities/turn-management/user-turn-strategies) on the
  context aggregator instead of using the transport's turn\_analyzer parameter.
</Tip>

<Card title="TransportParams Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.transports.base_transport.html#pipecat.transports.base_transport.TransportParams">
  Complete reference for all transport configuration options
</Card>

## Telephony Integration

Telephony services (phone calls) use WebSocket connections with specialized serialization:

### Supported Telephony Providers

<CardGroup>
  <Card title="Twilio" href="/server/services/serializers/twilio">
    Media Streams over WebSocket with TwilioFrameSerializer
  </Card>

  <Card title="Telnyx" href="/server/services/serializers/telnyx">
    Real-time media streaming with TelnyxFrameSerializer
  </Card>

  <Card title="Plivo" href="/server/services/serializers/plivo">
    Voice streaming API with PlivoFrameSerializer
  </Card>

  <Card title="Exotel" href="/server/services/serializers/exotel">
    Voice streaming integration with ExotelFrameSerializer
  </Card>
</CardGroup>

### Telephony Transport Setup

Telephony requires a `FrameSerializer` to handle provider-specific message formats:

```python theme={null}
# Create provider-specific serializer
serializer = TwilioFrameSerializer(
    stream_sid=stream_sid,
    call_sid=call_sid,
    account_sid=os.getenv("TWILIO_ACCOUNT_SID", ""),
    auth_token=os.getenv("TWILIO_AUTH_TOKEN", ""),
)

# Configure transport with serializer
transport = FastAPIWebsocketTransport(
    websocket=websocket_client,
    params=FastAPIWebsocketParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
        add_wav_header=False,
        serializer=serializer,  # Provider-specific serialization
    ),
)
```

<Note>
  The development runner automatically detects and configures the appropriate
  serializer when using `parse_telephony_websocket()`.
</Note>

## Conditional Transport Selection

The development runner provides a pattern for conditionally selecting transports based on the environment:

```python theme={null}
async def bot(runner_args: RunnerArguments):
    """Main bot entry point compatible with Pipecat Cloud."""

    transport = None

    if isinstance(runner_args, DailyRunnerArguments):
        from pipecat.transports.daily.transport import DailyParams, DailyTransport

        transport = DailyTransport(
            runner_args.room_url,
            runner_args.token,
            "Pipecat Bot",
            params=DailyParams(
                audio_in_enabled=True,
                audio_out_enabled=True,
            ),
        )

    elif isinstance(runner_args, SmallWebRTCRunnerArguments):
        from pipecat.transports.base_transport import TransportParams
        from pipecat.transports.network.small_webrtc import SmallWebRTCTransport

        transport = SmallWebRTCTransport(
            params=TransportParams(
                audio_in_enabled=True,
                audio_out_enabled=True,
            ),
            webrtc_connection=runner_args.webrtc_connection,
        )
    else:
        logger.error(f"Unsupported runner arguments type: {type(runner_args)}")
        return

    if transport is None:
        logger.error("Failed to create transport")
        return

    await run_bot(transport)
```

This pattern allows you to run the same bot code across different environments with different connection types.

## WebRTC vs WebSocket Considerations

Understanding when to use each connection type is crucial for building effective voice AI applications:

### WebRTC (Recommended for Client Applications)

**Best for:** Browser apps, mobile apps, real-time conversations

**Advantages:**

* **Low latency**: Optimized for real-time media with minimal delay
* **Built-in resilience**: Handles packet loss and network variations
* **Advanced audio processing**: Echo cancellation, noise reduction, automatic gain control
* **Quality monitoring**: Detailed performance and media quality statistics
* **Automatic timestamping**: Simplifies interruption and playout logic
* **Robust reconnection**: Built-in connection management

**Use WebRTC when:**

* Building client-facing applications (web, mobile)
* Conversational latency is critical
* Users are on potentially unreliable networks
* You need built-in audio processing features

### WebSocket (Good for Server-to-Server)

**Best for:** Telephony integration, server-to-server communication, prototyping

**Limitations for real-time media:**

* **TCP-based**: Subject to head-of-line blocking
* **Network sensitivity**: Less resilient to packet loss and jitter
* **Manual implementation**: Requires custom logic for reconnection, timestamping
* **Limited observability**: Harder to monitor connection quality

**Use WebSocket when:**

* Integrating with telephony providers (Twilio, Telnyx, etc.)
* Building server-to-server connections
* Prototyping or latency isn't critical
* Working within existing WebSocket infrastructure

## Key Takeaways

* **Transports are modular** - swap them without changing bot logic
* **Choose based on use case** - WebRTC for clients, WebSocket for telephony
* **Configuration is standardized** - TransportParams work across transport types
* **Pipeline placement matters** - consider what processing happens after output
* **Development runner helps** - provides patterns for multi-transport bots

## What's Next

Now that you understand how transports connect users to your bot, let's explore how to configure speech recognition to convert user audio into text.

<Card title="Speech Input & Turn Detection" icon="arrow-right" href="/guides/learn/speech-input">
  Learn how to configure speech recognition in your voice AI pipeline
</Card>


# What's Next
Source: https://docs.pipecat.ai/guides/learn/whats-next

Continue your Pipecat journey with advanced features, examples, and production deployment

## You've Mastered Voice AI Pipelines! 🎉

Congratulations! You've learned how to build complete voice AI applications with Pipecat. You're now well on your way to understanding pipelines, processors, transports, and all the components needed to create sophisticated conversational AI.

## Choose Your Path

<CardGroup>
  <Card title="Explore Examples" icon="code" href="https://github.com/pipecat-ai/pipecat-examples">
    **Production-ready applications** - Dive into 30+ complete examples including
    multimodal bots, creative applications, and enterprise integrations.
  </Card>

  <Card title="Advanced Guides" icon="book-open" href="/guides/fundamentals/custom-frame-processor">
    **Master advanced features** - Explore specialized topics like telephony,
    deployment, custom processors, and production optimization.
  </Card>
</CardGroup>

## 💬 Join the Community

Get involved with the Pipecat community to share your projects, get support, and contribute to the ecosystem:

* **[Discord Community](https://discord.gg/pipecat)**: Get support, share projects, and connect with other developers
* **[GitHub](https://github.com/pipecat-ai/pipecat)**: Contribute to the project and report issues
* **[Examples Repository](https://github.com/pipecat-ai/pipecat-examples)**: Community-contributed applications


# Daily Phone Numbers
Source: https://docs.pipecat.ai/guides/telephony/daily-phone-numbers

Purchase and manage phone numbers for Daily's PSTN services

Use Daily's [REST API](https://docs.daily.co/reference/rest-api/phone-numbers) to find, buy, list, and release phone numbers programmatically.

<Tip>
  You can also purchase phone numbers through the Pipecat Cloud Dashboard by
  going to `Settings` > `Telephony` and following the UI.
</Tip>

### Search for available phone numbers

We offer several [search filters](https://docs.daily.co/reference/rest-api/phone-numbers/list-available-numbers), such as, `region`, `city`, `areacode`, etc that can help narrow down the search space for a phone number. The response contains phone numbers that meet the chosen criteria. For simplicty, in the below examples, we will use `+19499870006` as an example `phone_number`.

<CodeGroup>
  ```bash Request theme={null}
  curl --request GET \
    --url 'https://api.daily.co/v1/list-available-numbers?region=CA' \
    --header 'Authorization: Bearer $DAILY_API_KEY'
  ```

  ```js Success theme={null}
  {
      "total_count": 1,
      "data": [{
          "number": "+19499800001",
          "region": "CA"
      }]
  }
  ```
</CodeGroup>

### Buy a phone number

To purchase the chosen number, make a POST request to the [buy-phone-number](https://docs.daily.co/reference/rest-api/phone-numbers/buy-phone-number) and pass the chosen phone number in the data field. The response returns the phone number and a unique id. The unique id is needed when releasing the chosen phone number, for example.

<CodeGroup>
  ```bash Request theme={null}
  curl --request POST \
    --url 'https://api.daily.co/v1/buy-phone-number' \
    --header 'Authorization: Bearer $DAILY_API_KEY' \
    --header 'Content-Type: application/json' \
    --data '{
          "number": "+19499800001"
  }'
  ```

  ```js Success theme={null}
  {
      "id": "5ccc01ca-f448-4f17-b7b4-cf354b7b8256",
      "number": "+19499800001"
  }
  ```
</CodeGroup>

<Info>
  Use the id to select the desired phone number for dial-out. Pass it as
  `callerId` in the `start_dialout` API or include it in the `dialout_settings`
  JSON.
</Info>

### Release a phone number

To release a phone number associated with your account. Successfully deleting the number takes the phone number out of operation.

<Warning>
  For compliance reasons, a phone number cannot be released until 14 days after
  purchase. A successful release operation cannot be undone, and the phone
  number will be taken out of operation.
</Warning>

<CodeGroup>
  ```bash Request theme={null}
  curl -H "Content-Type: application/json" \
    -H "Authorization: Bearer $DAILY_API_KEY" \
    -XDELETE \
    https://api.daily.co/v1/release-phone-number/5ccc01ca-f448-4f17-b7b4-cf354b7b8256
  ```

  ```js Success theme={null}
  {
    "deleted": true,
    "id": "0cb313e1-211f-4be0-833d-8c7305b19902",
  }
  ```

  ```js Failure theme={null}
  {
    ("Failed to release new number, try again in 14 days");
  }
  ```
</CodeGroup>

### List all purchased phone numbers

You can also list all your purchased phone numbers using the following [API](https://docs.daily.co/reference/rest-api/phone-numbers/purchased-phone-numbers)

<CodeGroup>
  ```bash Request theme={null}
  curl --request GET \
    --url 'https://api.daily.co/v1/purchased-phone-numbers' \
    --header 'Authorization: Bearer $DAILY_API_KEY'
  ```

  ```js Success theme={null}
  {
      "total_count": 1,
      "data": [{
          "name": null,
          "id": "5ccc01ca-f448-4f17-b7b4-cf354b7b8256",
          "created_date": "Mon, 19 Aug 2024 18:50:34 +0000",
          "type": "purchased_phone_number",
          "number": "+19499800001",
          "status": "verified",
          "verified": true
      }]
  }
  ```
</CodeGroup>

### CNAM Registration

Please contact `help@daily.co` for adding Caller Name (CNAM) to the purchased phone numbers. Daily will need your Name or Company Name to register with the United States CNAM Registry. Once registered, the Caller ID will display the appropriate name on the calling party's device.

We need the following information:

* All phone numbers and the corresponding CNAM entry. A CNAM entry can only be 14 letter long, including spaces.
* Company documents
* An employee's ID/driver's license

<Info>
  It can take up to 48 hours for the CNAM to reflect on the national caller ID
  name database and can take longer to update for major carriers, who check the
  national database periodically.
</Info>


# Daily PSTN
Source: https://docs.pipecat.ai/guides/telephony/daily-pstn

Complete guide to Daily's PSTN capabilities including dial-in, dial-out, and call transfers

## Things you'll need

* An active [Daily](https://www.daily.co) developer account with API key
* At least one phone number purchased through Daily (covered below)
* For local development: ngrok to expose your bot to the Internet

<CardGroup>
  <Card title="Dial-in Example" icon="phone-arrow-down-left" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-pstn-dial-in">
    Complete dial-in implementation with the Pipecat development runner
  </Card>

  <Card title="Dial-out Example" icon="phone-arrow-up-right" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-pstn-dial-out">
    Outbound calling with Daily PSTN integration
  </Card>

  <Card title="Cold Transfer Example" icon="arrow-right-arrow-left" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-pstn-cold-transfer">
    Cold transfer implementation for advanced call routing
  </Card>
</CardGroup>

## Phone Number Management

Before setting up dial-in or dial-out, you'll need to purchase phone numbers through Daily.

<Card title="Daily Phone Numbers Guide" icon="phone" href="./daily-phone-numbers">
  Complete guide to searching, purchasing, and managing phone numbers with
  Daily's REST API
</Card>

## Dial-in

Dial-in allows users to call your phone number and connect directly to your Pipecat bot.

### How It Works

Here's the sequence of events when someone calls your Daily phone number:

1. **Daily receives an incoming call** to your phone number
2. **Daily calls your webhook endpoint**
3. **The webhook creates a Daily room** with SIP configuration
4. **The webhook starts your bot** with the room details and caller information
5. **The caller is put on hold** with music
6. **The bot joins the Daily room** and signals readiness
7. **Daily forwards the call** to the Daily room
8. **The caller and bot are connected** for the conversation

### Local Development

The Pipecat development runner provides built-in webhook handling for Daily PSTN dial-in, eliminating the need for a separate webhook server.

#### 1. Run your bot with dial-in support

```bash theme={null}
uv run bot.py -t daily --dialin
```

This starts a FastAPI server on port 7860 with a `/daily-dialin-webhook` endpoint.

#### 2. Expose your bot to the internet

```bash theme={null}
ngrok http 7860
```

Copy the ngrok URL (e.g., `https://abc123.ngrok.io`).

<Tip>Use `ngrok http 7860 --subdomain your-subdomain` for a reusable URL.</Tip>

#### 3. Configure your Daily phone number

Set your phone number's `room_creation_api` webhook to:

```
https://your-ngrok-url.ngrok.io/daily-dialin-webhook
```

<Card title="Daily Dial-in Config API" icon="link" href="https://docs.daily.co/reference/rest-api/domainDialinConfig">
  Instructions for configuring the webhook URL for your phone number
</Card>

```shell Create pinless dial-in config theme={null}
curl --location 'https://api.daily.co/v1' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer YOUR_DAILY_API_KEY' \
--data '{
    "properties": {
        "pinless_dialin": [
            {
                "phone_number": "DAILY_PROVISIONED_NUMBER_HERE",
                "room_creation_api": "https://your-ngrok-url.ngrok.io/daily-dialin-webhook"
            }
        ]
    }
}'
```

<Warning>
  Ensure your `DAILY_API_KEY` has the phone number associated with it. This is
  required for the `pinlessCallUpdate` API call that connects the caller to the
  room.
</Warning>

### Configure your Pipecat bot for dial-in

The bot receives dial-in information through `RunnerArguments` containing:

* `room_url`: Daily room URL for the call
* `token`: Daily room token for authentication
* `body`: Contains `DailyDialinRequest` with call details

**Bot Entry Point**:

```python bot.py theme={null}
from pipecat.runner.types import DailyDialinRequest, RunnerArguments
from pipecat.transports.daily import DailyTransport, DailyParams, DailyDialinSettings

async def bot(runner_args: RunnerArguments):
    # Parse dial-in request from runner
    request = DailyDialinRequest.model_validate(runner_args.body)

    # Configure dial-in settings with webhook data
    daily_dialin_settings = DailyDialinSettings(
        call_id=request.dialin_settings.call_id,
        call_domain=request.dialin_settings.call_domain
    )

    # Create transport with dial-in configuration
    transport = DailyTransport(
        runner_args.room_url,
        runner_args.token,
        "Voice Bot",
        DailyParams(
            api_key=request.daily_api_key,
            api_url=request.daily_api_url,
            dialin_settings=daily_dialin_settings,
            audio_in_enabled=True,
            audio_out_enabled=True,
        )
    )

    # Your bot setup and pipeline creation here...

if __name__ == "__main__":
    from pipecat.runner.run import main
    main()
```

<Card title="Complete Bot Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/phone-chatbot/daily-pstn-dial-in/bot.py">
  See the full bot.py with argument handling, transport setup, and pipeline
  configuration
</Card>

### Customize Your Bot with Caller Information

Use the caller's phone number to personalize the conversation:

```python theme={null}
async def bot(runner_args: RunnerArguments):
    # Parse dial-in request
    request = DailyDialinRequest.model_validate(runner_args.body)

    # Get caller's phone number
    caller_phone = request.dialin_settings.From

    # Look up customer information from your database
    customer = await get_customer_by_phone(caller_phone)

    # Customize the system prompt
    messages = [
        {
            "role": "system",
            "content": f"You are a helpful assistant for {customer.name}. "
                      f"Their account status is {customer.status}. "
                      "Keep responses concise and conversational."
        }
    ]

    # Use the customized context in your bot...
```

### Run the Example

<Card title="Complete Setup Instructions" icon="book-open" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-pstn-dial-in">
  See the full README with step-by-step setup, environment variables, and
  troubleshooting tips
</Card>

## Dial-out

Dial-out allows your bot to initiate calls to phone numbers. Unlike dial-in, your bot starts the call rather than waiting for incoming calls.

<Warning>
  You must contact Daily to enable dial-out for your account. Submit a support
  request
  [here](https://docs.google.com/forms/d/1EHy2wsX20a2HjljAsJkyPqvATIM4WCvm7JzYKypGYis/edit).
</Warning>

### How It Works

Here's the sequence of events for dial-out calls:

1. **Your application triggers a dial-out** (via API call or user action)
2. **Server creates a Daily room** with dial-out capabilities enabled
3. **Bot joins the Daily room** and sets up the pipeline
4. **Bot initiates the dial-out call** to the target phone number
5. **Daily connects the call** to the phone number
6. **The recipient answers** and is connected to your bot
7. **The bot handles the conversation** with the called party

### Set up your server for dial-out

The dial-out server is simpler than dial-in since you're initiating calls rather than receiving webhooks. Your server needs to:

1. **Create Daily rooms** with dial-out enabled
2. **Start bot processes** with the target phone number
3. **Handle API requests** to trigger outbound calls

<Card title="Complete Server Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/phone-chatbot/daily-pstn-dial-out/server.py">
  See the full FastAPI server code with room creation and dial-out triggering
</Card>

### Configure your Pipecat bot for dial-out

The dial-out bot receives the target phone number and creates a transport without dial-in settings:

**Bot Entry Point**: The `bot()` method receives `RunnerArguments` containing:

* `room_url`: Daily room URL for the call
* `token`: Daily room token for authentication
* `phone_number`: Target phone number to call

**Transport Creation**: These arguments are used to configure the `DailyTransport`:

```python bot.py theme={null}
from pipecat.transports.daily import DailyTransport, DailyParams

async def bot(args: RunnerArguments):
    # Create transport for dial-out (no dial-in settings needed)
    transport = DailyTransport(
        args.room_url,
        args.token,
        "Voice Bot",
        DailyParams(
            api_url=os.getenv("DAILY_API_URL", "https://api.daily.co/v1"),
            api_key=os.getenv("DAILY_API_KEY"),
            audio_in_enabled=True,
            audio_out_enabled=True,
            video_out_enabled=False,
            transcription_enabled=True,
        )
    )

    # Start the dial-out call
    await transport.start_dialout(args.phone_number)

    # Pass transport to your bot pipeline
    # Your bot setup and pipeline creation here...
```

<Card title="Complete Bot Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/phone-chatbot/daily-pstn-dial-out/bot.py">
  See the full bot.py with dial-out configuration and pipeline setup
</Card>

### Run the Example

To test dial-out functionality:

1. **Start your server**: Run your FastAPI server
2. **Trigger a call**: Make an API request to start a dial-out call
3. **Answer your phone**: The bot will call the specified number
4. **Talk to your bot**: Have a conversation with your AI agent

<Card title="Complete Setup Instructions" icon="book-open" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-pstn-dial-out">
  See the full README with step-by-step setup, API usage, and configuration
  details
</Card>

## Call Transfers

Daily supports cold transfers, allowing you to transfer an active call to another phone number. The bot can initiate a transfer and then leave the call, connecting the caller directly to the transfer destination.

### How Call Transfers Work

1. **Bot receives transfer request** (via function call or user input)
2. **Bot informs the caller** about the transfer
3. **Bot initiates SIP call transfer** to the destination number
4. **Daily connects the transfer** to the destination
5. **Bot leaves the call** (cold transfer)
6. **Caller and destination** continue the conversation

### Implementing Call Transfers

Call transfers are typically implemented as LLM function calls that your bot can invoke:

```python bot.py theme={null}
async def dial_operator(transport: BaseTransport, params: FunctionCallParams):
    """Function the bot can call to transfer to an operator."""
    operator_number = os.getenv("OPERATOR_NUMBER", "+1234567890")

    # Inform the user about the transfer
    content = "I'm transferring you to a supervisor now. Please hold while I connect you."
    message = {"role": "system", "content": content}
    await params.llm.push_frame(LLMMessagesAppendFrame([message], run_llm=True))

    # Execute the SIP call transfer
    transfer_params = {"toEndPoint": operator_number}
    await transport.sip_call_transfer(transfer_params)
```

### Handling Transfer Events

Your bot should handle transfer-related events to manage the call flow:

```python bot.py theme={null}
@transport.event_handler("on_dialout_answered")
async def on_dialout_answered(transport, data):
    logger.info(f"Transfer successful: {data}")
    # Cold transfer: bot leaves, caller and operator continue
    await task.queue_frames([EndFrame()])
```

<Card title="Complete Transfer Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-pstn-cold-transfer">
  See the full call transfer example with LLM function calls, event handling,
  and error management
</Card>

## Deployment

### Pipecat Cloud

For production deployment without managing your own infrastructure, use Pipecat Cloud. Pipecat Cloud handles all webhook infrastructure, room creation, and bot scaling automatically.

<CardGroup>
  <Card title="Daily Dial-in on Pipecat Cloud" icon="phone-arrow-down-left" href="/deployment/pipecat-cloud/guides/telephony/daily-dial-in">
    Complete guide for deploying dial-in bots with automatic webhook handling
  </Card>

  <Card title="Daily Dial-out on Pipecat Cloud" icon="phone-arrow-up-right" href="/deployment/pipecat-cloud/guides/telephony/daily-dial-out">
    Complete guide for deploying dial-out bots with caller ID management
  </Card>
</CardGroup>

### Self-Hosted Deployment

For self-hosted production deployment, ensure your servers are:

* Publicly accessible with HTTPS
* Able to handle concurrent requests
* Properly configured with your Daily API credentials

## Next Steps

* Explore the [complete examples](https://github.com/pipecat-ai/pipecat/tree/main/examples) for full implementations
* Learn about [Daily's SIP integration](./twilio-daily-sip) for more advanced telephony scenarios
* Check out [Daily's REST API documentation](https://docs.daily.co/reference) for additional configuration options


# Exotel WebSocket Integration
Source: https://docs.pipecat.ai/guides/telephony/exotel-websockets

Complete guide to using Exotel Voice Streaming with Pipecat for dial-in and dial-out functionality

## Things you'll need

* An active [Exotel](https://exotel.com) account with voice streaming enabled
* One or more Exotel provisioned phone numbers
* A public-facing server or tunneling service like [ngrok](https://ngrok.com/) (for dial-out only)
* API keys for speech-to-text, text-to-speech, and LLM services

<CardGroup>
  <Card title="Exotel Dial-in Example" icon="phone-arrow-down-left" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/exotel-chatbot/inbound">
    Complete dial-in implementation using Exotel Voice Streaming over WebSocket
  </Card>

  <Card title="Exotel Dial-out Example" icon="phone-arrow-up-right" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/exotel-chatbot/outbound">
    Outbound calling using Exotel Voice Streaming with WebSocket transport
  </Card>
</CardGroup>

## Phone Number Setup

You'll need Exotel phone numbers for both dial-in and dial-out functionality:

* Visit [exotel.com](https://exotel.com) and purchase phone numbers
* Complete KYC verification if required
* Ensure voice streaming is enabled on your account (contact Exotel support if needed)

## Environment Setup

Configure your environment variables for Exotel and AI services:

```shell .env theme={null}
EXOTEL_ACCOUNT_SID=...
EXOTEL_API_KEY=...
EXOTEL_API_TOKEN=...
OPENAI_API_KEY=...
DEEPGRAM_API_KEY=...
CARTESIA_API_KEY=...
```

## Dial-in

Dial-in allows users to call your Exotel number and connect to your Pipecat bot via WebSocket Voice Streaming. Unlike other providers, Exotel automatically provides caller information (to/from numbers) in the WebSocket messages, so no custom server is needed for basic dial-in functionality.

### How It Works

Here's the sequence of events when someone calls your Exotel number:

1. **Exotel receives an incoming call** to your phone number
2. **Exotel executes your App Bazaar flow** which includes a Voicebot applet
3. **Exotel opens a WebSocket** to your server with real-time audio and call metadata
4. **Your bot processes the audio** using the Pipecat pipeline
5. **The bot responds with audio** sent back to Exotel over WebSocket
6. **Exotel plays the audio** to the caller in real-time

### Set up your App Bazaar flow

Exotel uses App Bazaar to configure call flows. For dial-in, you create an app with a Voicebot applet that establishes a WebSocket connection directly to your bot:

1. **Navigate to App Bazaar** in your Exotel dashboard
2. **Create a new app** or edit an existing one
3. **Add a Voicebot applet** (not "Stream" or "Passthru")
4. **Configure the WebSocket URL**: `wss://your-server.com/ws`
5. **Add a Hangup applet** at the end to properly terminate calls

Your flow should look like: `Call Start → [Voicebot Applet] → [Hangup Applet]`

### Personalizing Your Bot with Caller Information

Exotel automatically includes caller information (to/from numbers) in the WebSocket messages. You can use the caller's phone number to:

* Look up customer information in your database
* Personalize greetings and responses based on caller identity
* Route calls to different bot behaviors (e.g., VIP handling, support vs sales)
* Implement custom business logic based on the caller

The example bot demonstrates how to extract caller information from Exotel's WebSocket messages and use it to personalize the conversation.

### Configure your Pipecat bot for dial-in

Your bot receives the WebSocket connection and automatically gets call information from Exotel's WebSocket messages. The key components are:

**WebSocket Parsing**: Pipecat's built-in parser extracts call data from Exotel's WebSocket messages:

```python bot.py theme={null}
from pipecat.runner.utils import parse_telephony_websocket

async def run_bot(websocket: WebSocket):
    # Parse Exotel WebSocket data
    transport_type, call_data = await parse_telephony_websocket(websocket)

    # Extract call information (automatically provided by Exotel)
    # Data format: {
    #     "stream_id": str,
    #     "call_id": str,
    #     "account_sid": str,
    #     "from": str,
    #     "to": str,
    # }
    stream_id = call_data["stream_id"]
    call_id = call_data["call_id"]  # Exotel's CallSid
    account_sid = call_data["account_sid"]
    from_number = call_data["from"]  # Caller's number
    to_number = call_data["to"]      # Your Exotel number
```

**Transport Creation**: Configure the WebSocket transport with Exotel serialization:

```python bot.py theme={null}
# Create Exotel serializer with call details
serializer = ExotelFrameSerializer(
    stream_id=stream_id,
    call_id=call_id,
    account_sid=account_sid,
    api_key=os.getenv("EXOTEL_API_KEY"),
    api_token=os.getenv("EXOTEL_API_TOKEN"),
)

# Configure WebSocket transport
transport = FastAPIWebsocketTransport(
    websocket=websocket,
    params=FastAPIWebsocketParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
        add_wav_header=False,
        serializer=serializer,
    ),
)

# Your bot pipeline setup here...
```

<Card title="Complete Bot Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/exotel-chatbot/inbound/bot.py">
  See the full bot.py with WebSocket parsing, transport setup, and pipeline
  configuration
</Card>

### Set up Exotel phone number

Configure your Exotel phone number to use your App Bazaar flow:

1. Go to the [Exotel Dashboard](https://my.exotel.com)
2. Navigate to ExoPhones
3. Find your phone number and click the edit icon
4. Under "App", select your App Bazaar flow
5. Save the configuration

### Run the Example

For dial-in, you can run your bot directly without a separate server:

```shell theme={null}
# Start your bot directly
python bot.py

# For local development, use ngrok to expose your WebSocket
ngrok http 8000
# Use the ngrok WebSocket URL (wss://abc123.ngrok.io/ws) in your App Bazaar flow
```

<Card title="Complete Setup Instructions" icon="book-open" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/exotel-chatbot/inbound">
  See the full README with step-by-step setup, App Bazaar configuration, and
  testing
</Card>

## Dial-out

Dial-out allows your bot to initiate calls to phone numbers using Exotel's outbound calling capabilities with WebSocket Voice Streaming.

### How It Works

Here's the sequence of events for dial-out calls:

1. **Your application triggers a dial-out** (via API call or user action)
2. **Server initiates an Exotel call** using the Voice API
3. **Exotel establishes the call** and opens a WebSocket connection
4. **Your bot joins the WebSocket** and sets up the pipeline
5. **The recipient answers** and is connected to your bot
6. **The bot handles the conversation** with real-time audio streaming

### Set up your server for dial-out

The dial-out server creates outbound calls and manages WebSocket connections. When you trigger a dial-out call, your server:

1. **Receives the dial-out request** with target phone number and parameters
2. **Creates an Exotel call** using the Voice API with a callback URL
3. **Accepts the WebSocket** connection from Exotel
4. **Parses call data** from the WebSocket messages
5. **Runs the Pipecat bot** with the call information

<Card title="Complete Server Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/exotel-chatbot/outbound/server.py">
  See the full FastAPI server code with outbound call creation and WebSocket
  handling
</Card>

### Configure your Pipecat bot for dial-out

The dial-out bot configuration is similar to dial-in, with Exotel automatically providing call information.

**Call Information**: Extract call details from Exotel's WebSocket messages:

```python bot.py theme={null}
# Parse WebSocket data (same as dial-in)
transport_type, call_data = await parse_telephony_websocket(websocket)

# Extract call information
stream_id = call_data["stream_id"]
call_id = call_data["call_id"]
account_sid = call_data["account_sid"]
from_number = call_data["from"]  # Your Exotel number
to_number = call_data["to"]      # Target number you're calling

# Customize bot behavior for outbound calls
greeting = f"Hi! This is an automated call from {from_number}. How are you today?"
```

**Custom Data Limitations for Dial-out**:

<Warning>
  Currently, there are limitations with passing custom data through query
  parameters in Exotel's WebSocket URLs. While the platform theoretically
  supports query parameters, they may be stripped during call processing. This
  is an ongoing limitation that may be addressed in future Exotel updates.
</Warning>

If you need to pass custom data for dial-out, consider these alternatives:

* Use different WebSocket endpoints for different call types
* Configure multiple App Bazaar flows for different scenarios
* Handle customization based on the called number (available in WebSocket messages)

<Card title="Complete Bot Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/exotel-chatbot/outbound/bot.py">
  See the full bot.py with outbound call handling and call information usage
</Card>

### Run the Example

To test dial-out functionality:

1. **Start your server**: Run your FastAPI server
2. **Trigger a call**: Make an API request to start an outbound call
3. **Answer your phone**: The bot will call the specified number
4. **Talk to your bot**: Have a conversation with your AI agent

<Card title="Complete Setup Instructions" icon="book-open" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/exotel-chatbot/outbound">
  See the full README with step-by-step setup, API usage, and outbound call
  configuration
</Card>

## Key Features

### Audio Format and Sample Rate

Exotel Voice Streaming uses 8kHz mono audio with 16-bit PCM encoding. Configure your pipeline accordingly:

```python theme={null}
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        audio_in_sample_rate=8000,
        audio_out_sample_rate=8000,
    ),
)
```

### Built-in Call Information

Like Telnyx, Exotel automatically includes comprehensive call information (to/from numbers, account details) in the WebSocket messages, eliminating the need for custom webhook servers in basic dial-in scenarios.

## Deployment

### Local Development

Use [ngrok](https://ngrok.com/) to expose your local server for testing:

```shell theme={null}
python server.py  # For dial-out
# OR
python bot.py     # For dial-in

ngrok http 8000
# Use the ngrok URL in your App Bazaar Voicebot applet
```

### Pipecat Cloud Deployment

For production deployment without managing your own infrastructure, use Pipecat Cloud:

<Card title="Exotel WebSocket on Pipecat Cloud" icon="cloud" href="/deployment/pipecat-cloud/guides/telephony/exotel-websocket">
  Deploy Exotel WebSocket bots with automatic scaling and managed infrastructure
</Card>

### Self-Hosted Production Deployment

For fully self-hosted production deployment, ensure your servers are:

* Publicly accessible with HTTPS
* Able to handle concurrent WebSocket connections
* Properly configured with Exotel API credentials
* Implementing proper error handling and logging

## Next Steps

* Explore the [complete examples](https://github.com/pipecat-ai/pipecat-examples/tree/main/exotel-chatbot) for full implementations
* Learn about [Daily + Twilio SIP integration](./twilio-daily-sip) for advanced telephony scenarios
* Check out [Daily PSTN integration](./daily-pstn) for direct phone number provisioning


# Overview
Source: https://docs.pipecat.ai/guides/telephony/overview

How to call your bots (or have them call you)

## Introduction

You can dial-in to your Pipecat bots, and have them dial-out too, across both PSTN and SIP. The technical implementation will depend on your chosen transport and phone number vendor; each will likely have their own methods and events to consider.

## Key Terms

**PSTN (Public Switched Telephone Network)**: The traditional phone network consisting of physical phone lines, cables, and transmission links. PSTN operates on a one-user-per-line basis.

**SIP (Session Initiation Protocol)**: A signaling protocol used for voice and video calls over IP networks. SIP can handle multiple users per line and enables advanced call control features like transfers, forwarding, and multi-party calls.

## Telephony Connection Options

### WebSocket Connections

**Best for:** Simple telephony workflows and quick prototypes

* **How it works:** Real-time audio streaming over WebSocket connections
* **Call control:** Basic, managed by the telephony provider
* **Supported providers:** Twilio, Telnyx, Plivo, Exotel
* **Limitations:** No advanced call center features like transfers or reconnects

**When to use:**

* Simple inbound/outbound calling
* Integration with existing Twilio Studio or Flex workflows
* Quick setup with minimal configuration

### WebRTC Connections

**Best for:** When connecting web clients to telephony systems or for more complex transfer scenarios

* **How it works:** Peer-to-peer real-time communication optimized for varying network conditions
* **Call control:** Advanced, includes track-level control
* **Supported providers:** Daily (with PSTN integration)
* **Benefits:** Designed for scale, handles network variations gracefully

**When to use:**

* When connecting web or mobile clients to telephony systems
* Applications requiring high-quality audio/video
* Users on devices with varying network conditions

### SIP Connections

**Best for:** Enterprise telephony and advanced call control scenarios

* **How it works:** Industry-standard protocol for voice/video calls over IP
* **Call control:** Full control, including transfers, forwarding, multi-party, etc.
* **Integration:** Works with legacy call centers and telephony systems
* **Vendor flexibility:** Supports multiple telephony vendors without platform-specific code

**When to use:**

* Multi-agent or multi-party calls
* Integration with legacy call centers
* Advanced features: warm transfers, agent assist, call forwarding
* Enterprise telephony requirements

<Note>
  **Multiple providers:** You can configure your Pipecat bots to handle multiple
  vendors simultaneously. For example, use both Daily and Twilio as phone number
  providers concurrently.
</Note>

## Telephony Provider Options

### Daily PSTN (WebRTC)

Direct PSTN connectivity through Daily's platform with WebRTC transport.

* **Setup:** Purchase phone numbers directly through Daily
* **Transport:** WebRTC for optimal audio quality
* **Use case:** A range of use cases, from simple to complex including cold and warm transfers

[**Daily PSTN Guide →**](./daily-pstn)

### Daily + SIP Integration

Combine Daily's WebRTC transport with SIP-based telephony providers.

* **Setup:** Use existing Twilio numbers with Daily transport
* **Transport:** WebRTC with SIP forwarding
* **Use case:** Leverage existing Twilio workflows with Daily's transport benefits

[**Daily + Twilio SIP Guide →**](./twilio-daily-sip)

### WebSocket Providers

Direct integration with telephony providers using WebSocket connections.

<CardGroup>
  <Card title="Twilio WebSocket" icon="phone" href="./twilio-websockets">
    Media Streams integration with Twilio's telephony platform
  </Card>

  <Card title="Telnyx WebSocket" icon="phone" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/telnyx-chatbot">
    Real-time media streaming with Telnyx services
  </Card>

  <Card title="Plivo WebSocket" icon="phone" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/plivo-chatbot">
    Voice streaming API integration with Plivo
  </Card>

  <Card title="Exotel WebSocket" icon="phone" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/exotel-chatbot">
    WebSocket integration with Exotel telephony platform
  </Card>
</CardGroup>

## Next Steps

Choose the telephony option that best fits your use case and follow the corresponding guide. Each option provides complete setup instructions and example code to get you started quickly.


# Plivo WebSocket Integration
Source: https://docs.pipecat.ai/guides/telephony/plivo-websockets

Complete guide to using Plivo Media Streaming with Pipecat for dial-in and dial-out functionality

## Things you'll need

* An active [Plivo](https://www.plivo.com) account with API credentials
* One or more Plivo provisioned phone numbers
* A public-facing server or tunneling service like [ngrok](https://ngrok.com/)
* API keys for speech-to-text, text-to-speech, and LLM services

<CardGroup>
  <Card title="Plivo Dial-in Example" icon="phone-arrow-down-left" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/plivo-chatbot/inbound">
    Complete dial-in implementation using Plivo Media Streaming over WebSocket
  </Card>

  <Card title="Plivo Dial-out Example" icon="phone-arrow-up-right" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/plivo-chatbot/outbound">
    Outbound calling using Plivo Media Streaming with WebSocket transport
  </Card>
</CardGroup>

## Phone Number Setup

You'll need Plivo phone numbers for both dial-in and dial-out functionality:

* Visit [plivo.com](https://www.plivo.com) and purchase phone numbers
* Ensure your numbers support Voice capabilities
* Configure XML applications for dial-in numbers (covered below)

## Environment Setup

Configure your environment variables for Plivo and AI services:

```shell .env theme={null}
PLIVO_AUTH_ID=...
PLIVO_AUTH_TOKEN=...
OPENAI_API_KEY=...
DEEPGRAM_API_KEY=...
CARTESIA_API_KEY=...
```

## Dial-in

Dial-in allows users to call your Plivo number and connect to your Pipecat bot via WebSocket Media Streaming.

### How It Works

Here's the sequence of events when someone calls your Plivo number:

1. **Plivo receives an incoming call** to your phone number
2. **Plivo calls your XML server** with call details
3. **Your server responds with XML** that establishes a WebSocket connection
4. **Plivo opens a WebSocket** to your server with real-time audio
5. **Your bot processes the audio** using the Pipecat pipeline
6. **The bot responds with audio** sent back to Plivo over WebSocket
7. **Plivo plays the audio** to the caller in real-time

### Set up your XML server

Your server handles Plivo webhooks and WebSocket connections for real-time audio streaming. When Plivo receives a call, your server:

1. **Receives the webhook** with call details (CallUUID, caller information)
2. **Returns XML** that establishes a WebSocket connection to your server
3. **Accepts the WebSocket** connection from Plivo
4. **Parses call data** from the WebSocket messages
5. **Runs the Pipecat bot** with the parsed call information

The server orchestrates the real-time audio streaming between Plivo's telephony infrastructure and your Pipecat bot pipeline.

### Custom Data with Query Parameters

You can pass custom data to your bot by adding query parameters to the WebSocket URL in your XML response. This works for both dial-in and dial-out scenarios:

```python server.py theme={null}
# Example: Adding custom parameters to the WebSocket URL
xml = f"""<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Stream bidirectional="true" keepCallAlive="true" contentType="audio/x-mulaw;rate=8000">
    wss://your-server.com/ws?user_id={user_id}&amp;session_type=support&amp;campaign_id=summer_sale
  </Stream>
</Response>"""
```

These parameters are accessible in your server code and can be used to customize your bot's behavior based on the specific call context.

<Card title="Complete Server Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/plivo-chatbot/inbound/server.py">
  See the full FastAPI server code with XML generation and WebSocket handling
</Card>

### Configure your Pipecat bot for dial-in

Your bot receives the WebSocket connection and parses the call data to extract call information. The key components are:

**WebSocket Parsing**: Pipecat's built-in parser extracts call data from Plivo's WebSocket messages:

```python bot.py theme={null}
from pipecat.runner.utils import parse_telephony_websocket

async def run_bot(websocket: WebSocket):
    # Parse Plivo WebSocket data
    transport_type, call_data = await parse_telephony_websocket(websocket)

    # Extract call information
    stream_id = call_data["stream_id"]
    call_id = call_data["call_id"]  # Plivo's CallUUID
```

**Transport Creation**: Configure the WebSocket transport with Plivo serialization:

```python bot.py theme={null}
# Create Plivo serializer with call details
serializer = PlivoFrameSerializer(
    stream_id=stream_id,
    call_id=call_id,
    auth_id=os.getenv("PLIVO_AUTH_ID"),
    auth_token=os.getenv("PLIVO_AUTH_TOKEN"),
)

# Configure WebSocket transport
transport = FastAPIWebsocketTransport(
    websocket=websocket,
    params=FastAPIWebsocketParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
        add_wav_header=False,
        serializer=serializer,
    ),
)

# Your bot pipeline setup here...
```

<Card title="Complete Bot Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/plivo-chatbot/inbound/bot.py">
  See the full bot.py with WebSocket parsing, transport setup, and pipeline
  configuration
</Card>

### Set up Plivo XML application

Configure your Plivo phone number to use your XML application:

1. Go to the [Plivo Console](https://console.plivo.com)
2. Navigate to Voice → XML Applications
3. Create a new XML Application with your server's webhook URL
4. Assign the XML Application to your phone number

### Run the Example

For local development, use [ngrok](https://ngrok.com/) to expose your local server:

```shell theme={null}
# Start your server
python server.py

# In another terminal, start ngrok
ngrok http 8000

# Use the ngrok URL in your Plivo XML Application
```

<Card title="Complete Setup Instructions" icon="book-open" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/plivo-chatbot/inbound">
  See the full README with step-by-step setup, XML configuration, and testing
</Card>

## Dial-out

Dial-out allows your bot to initiate calls to phone numbers using Plivo's outbound calling capabilities with WebSocket Media Streaming.

### How It Works

Here's the sequence of events for dial-out calls:

1. **Your application triggers a dial-out** (via API call or user action)
2. **Server initiates a Plivo call** to the target phone number
3. **Plivo establishes the call** and opens a WebSocket connection
4. **Your bot joins the WebSocket** and sets up the pipeline
5. **The recipient answers** and is connected to your bot
6. **The bot handles the conversation** with real-time audio streaming

### Set up your server for dial-out

The dial-out server creates outbound calls and manages WebSocket connections. When you trigger a dial-out call, your server:

1. **Receives the dial-out request** with target phone number and parameters
2. **Creates a Plivo call** using the Voice API with XML that establishes WebSocket
3. **Accepts the WebSocket** connection from Plivo
4. **Parses call data** from the WebSocket messages
5. **Runs the Pipecat bot** with the call information

<Card title="Complete Server Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/plivo-chatbot/outbound/server.py">
  See the full FastAPI server code with outbound call creation and WebSocket
  handling
</Card>

### Passing Custom Data to Your Bot

For dial-out calls, you often need to pass custom data to personalize the conversation—such as user information, campaign details, or context from your application. The dial-out example supports passing a `body` object that becomes available to your bot via `runner_args.body`.

**Triggering a call with custom data:**

```bash theme={null}
curl -X POST https://your-server.com/start \
  -H "Content-Type: application/json" \
  -d '{
    "phone_number": "+1234567890",
    "body": {
      "user": {
        "id": "user123",
        "firstName": "John",
        "lastName": "Doe",
        "accountType": "premium"
      },
      "campaign": "renewal_reminder",
      "context": "subscription expiring in 3 days"
    }
  }'
```

**How it works:**

1. The `/start` endpoint receives the request with `phone_number` and optional `body`
2. The `body` is base64-encoded and passed through the TeXML URL chain
3. When the WebSocket connects, the server decodes the body and passes it to your bot via `runner_args.body`
4. Your bot accesses the custom data to personalize the conversation

**Accessing custom data in your bot:**

```python bot.py theme={null}
async def bot(runner_args: RunnerArguments):
    body_data = runner_args.body or {}
    first_name = body_data.get("user", {}).get("firstName", "there")
    # Use first_name to personalize greetings, prompts, etc.
```

This approach works consistently in both local development and Pipecat Cloud production deployments.

### Configure your Pipecat bot for dial-out

The dial-out bot configuration is similar to dial-in, with Plivo automatically providing call information.

**Call Information**: Extract call details from Plivo's WebSocket messages:

```python bot.py theme={null}
# Parse WebSocket data (same as dial-in)
transport_type, call_data = await parse_telephony_websocket(websocket)

# Extract call information
stream_id = call_data["stream_id"]
call_id = call_data["call_id"]  # Plivo's CallUUID

# Access custom data from runner args
body_data = runner_args.body or {}
user_name = body_data.get("user", {}).get("firstName", "there")

# Customize bot behavior for outbound calls
greeting = f"Hi {user_name}! This is an automated call. How can I help you today?"
```

**Transport Configuration**: Same transport setup as dial-in:

```python bot.py theme={null}
# Create Plivo serializer
serializer = PlivoFrameSerializer(
    stream_id=stream_id,
    call_id=call_id,
    auth_id=os.getenv("PLIVO_AUTH_ID"),
    auth_token=os.getenv("PLIVO_AUTH_TOKEN"),
)

# Configure transport
transport = FastAPIWebsocketTransport(
    websocket=websocket,
    params=FastAPIWebsocketParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
        add_wav_header=False,
        serializer=serializer,
    ),
)
```

<Card title="Complete Bot Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/plivo-chatbot/outbound/bot.py">
  See the full bot.py with outbound call handling and call information usage
</Card>

### Run the Example

To test dial-out functionality:

1. **Start your server**: Run your FastAPI server
2. **Trigger a call**: Make an API request to start an outbound call
3. **Answer your phone**: The bot will call the specified number
4. **Talk to your bot**: Have a conversation with your AI agent

<Card title="Complete Setup Instructions" icon="book-open" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/plivo-chatbot/outbound">
  See the full README with step-by-step setup, API usage, and outbound call
  configuration
</Card>

## Key Features

### Audio Format and Sample Rate

Plivo Media Streaming uses 8kHz mono audio with μ-law encoding. Configure your pipeline accordingly:

```python theme={null}
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        audio_in_sample_rate=8000,
        audio_out_sample_rate=8000,
    ),
)
```

### Automatic Call Termination

When you provide Plivo credentials to the `PlivoFrameSerializer`, it automatically ends calls when your pipeline ends:

```python theme={null}
serializer = PlivoFrameSerializer(
    stream_id=stream_id,
    call_id=call_id,
    auth_id=os.getenv("PLIVO_AUTH_ID"),
    auth_token=os.getenv("PLIVO_AUTH_TOKEN"),  # Enables auto-termination
)
```

## Deployment

### Local Development

Use [ngrok](https://ngrok.com/) to expose your local server for testing:

```shell theme={null}
python server.py
ngrok http 8000
# Use the ngrok URL in your Plivo XML Application
```

### Pipecat Cloud Deployment

For production deployment without managing your own infrastructure, use Pipecat Cloud:

<Card title="Plivo WebSocket on Pipecat Cloud" icon="cloud" href="/deployment/pipecat-cloud/guides/telephony/plivo-websocket">
  Deploy Plivo WebSocket bots with automatic scaling and managed infrastructure
</Card>

### Self-Hosted Production Deployment

For fully self-hosted production deployment, ensure your servers are:

* Publicly accessible with HTTPS
* Able to handle concurrent WebSocket connections
* Properly configured with Plivo API credentials
* Implementing proper error handling and logging

## Next Steps

* Explore the [complete examples](https://github.com/pipecat-ai/pipecat-examples/tree/main/plivo-chatbot) for full implementations
* Learn about [Daily + Twilio SIP integration](./twilio-daily-sip) for advanced telephony scenarios
* Check out [Daily PSTN integration](./daily-pstn) for direct phone number provisioning


# Telnyx WebSocket Integration
Source: https://docs.pipecat.ai/guides/telephony/telnyx-websockets

Complete guide to using Telnyx Media Streaming with Pipecat for dial-in and dial-out functionality

## Things you'll need

* An active [Telnyx](https://telnyx.com) account with API credentials
* One or more Telnyx provisioned phone numbers
* A public-facing server or tunneling service like [ngrok](https://ngrok.com/) (for dial-out only)
* API keys for speech-to-text, text-to-speech, and LLM services

<CardGroup>
  <Card title="Telnyx Dial-in Example" icon="phone-arrow-down-left" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/telnyx-chatbot/inbound">
    Complete dial-in implementation using Telnyx Media Streaming over WebSocket
  </Card>

  <Card title="Telnyx Dial-out Example" icon="phone-arrow-up-right" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/telnyx-chatbot/outbound">
    Outbound calling using Telnyx Media Streaming with WebSocket transport
  </Card>
</CardGroup>

## Phone Number Setup

You'll need Telnyx phone numbers for both dial-in and dial-out functionality:

* Visit [telnyx.com](https://telnyx.com) and purchase phone numbers
* Ensure your numbers support Voice capabilities
* Configure TeXML applications for dial-in numbers (covered below)

## Environment Setup

Configure your environment variables for Telnyx and AI services:

```shell .env theme={null}
TELNYX_API_KEY=...
OPENAI_API_KEY=...
DEEPGRAM_API_KEY=...
CARTESIA_API_KEY=...
```

## Dial-in

Dial-in allows users to call your Telnyx number and connect to your Pipecat bot via WebSocket Media Streaming. Unlike Twilio, Telnyx automatically provides caller information (to/from numbers) in the WebSocket messages, so no custom server is needed for basic dial-in functionality.

### How It Works

Here's the sequence of events when someone calls your Telnyx number:

1. **Telnyx receives an incoming call** to your phone number
2. **Telnyx executes your TeXML application** which establishes a WebSocket connection
3. **Telnyx opens a WebSocket** to your server with real-time audio and call metadata
4. **Your bot processes the audio** using the Pipecat pipeline
5. **The bot responds with audio** sent back to Telnyx over WebSocket
6. **Telnyx plays the audio** to the caller in real-time

### Set up your TeXML application

Telnyx uses TeXML (Telnyx Extensible Markup Language) to control call flow. For dial-in, you create a TeXML application that establishes a WebSocket connection directly to your bot:

```xml TeXML for dial-in theme={null}
<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Connect>
    <Stream url="wss://your-server.com/ws" bidirectionalMode="rtp"></Stream>
  </Connect>
  <Pause length="40"/>
</Response>
```

The `bidirectionalMode="rtp"` parameter enables real-time audio streaming in both directions.

### Personalizing Your Bot with Caller Information

Telnyx automatically includes caller information (to/from numbers) in the WebSocket messages. You can use the caller's phone number to:

* Look up customer information in your database
* Personalize greetings and responses based on caller identity
* Route calls to different bot behaviors (e.g., VIP handling, support vs sales)
* Implement custom business logic based on the caller

The example bot demonstrates how to extract caller information from Telnyx's WebSocket messages and use it to personalize the conversation.

### Configure your Pipecat bot for dial-in

Your bot receives the WebSocket connection and automatically gets call information from Telnyx's WebSocket messages. The key components are:

**WebSocket Parsing**: Pipecat's built-in parser extracts call data from Telnyx's WebSocket messages:

```python bot.py theme={null}
from pipecat.runner.utils import parse_telephony_websocket

async def run_bot(websocket: WebSocket):
    # Parse Telnyx WebSocket data
    transport_type, call_data = await parse_telephony_websocket(websocket)

    # Extract call information (automatically provided by Telnyx)
    # Data format: {
    #     "stream_id": str,
    #     "call_control_id": str,
    #     "outbound_encoding": str,
    #     "from": str,
    #     "to": str,
    # }
    stream_id = call_data["stream_id"]
    call_control_id = call_data["call_control_id"]
    outbound_encoding = call_data["outbound_encoding"]
    from_number = call_data["from"]  # Caller's number
    to_number = call_data["to"]      # Your Telnyx number
```

**Transport Creation**: Configure the WebSocket transport with Telnyx serialization:

```python bot.py theme={null}
# Create Telnyx serializer with call details
serializer = TelnyxFrameSerializer(
    stream_id=stream_id,
    call_control_id=call_control_id,
    api_key=os.getenv("TELNYX_API_KEY"),
)

# Configure WebSocket transport
transport = FastAPIWebsocketTransport(
    websocket=websocket,
    params=FastAPIWebsocketParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
        add_wav_header=False,
        serializer=serializer,
    ),
)

# Your bot pipeline setup here...
```

<Card title="Complete Bot Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/telnyx-chatbot/inbound/bot.py">
  See the full bot.py with WebSocket parsing, transport setup, and pipeline
  configuration
</Card>

### Set up Telnyx TeXML application

Configure your Telnyx phone number to use your TeXML application:

1. Go to the [Telnyx Portal](https://portal.telnyx.com)
2. Navigate to Voice → Programmable Voice → TeXML Applications
3. Create a new TeXML Application with your WebSocket URL
4. Assign the TeXML Application to your phone number

<Card title="Complete Setup Instructions" icon="book-open" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/telnyx-chatbot/inbound">
  See the full README with step-by-step setup, TeXML configuration, and testing
</Card>

## Dial-out

Dial-out allows your bot to initiate calls to phone numbers using Telnyx's outbound calling capabilities with WebSocket Media Streaming.

### How It Works

Here's the sequence of events for dial-out calls:

1. **Your application triggers a dial-out** (via API call or user action)
2. **Server initiates a Telnyx call** using the Call Control API
3. **Telnyx establishes the call** and opens a WebSocket connection
4. **Your bot joins the WebSocket** and sets up the pipeline
5. **The recipient answers** and is connected to your bot
6. **The bot handles the conversation** with real-time audio streaming

### Set up your server for dial-out

The dial-out server creates outbound calls and manages WebSocket connections. When you trigger a dial-out call, your server:

1. **Receives the dial-out request** with target phone number and parameters
2. **Creates a Telnyx call** using the Call Control API with a webhook URL
3. **Accepts the WebSocket** connection from Telnyx
4. **Parses call data** including call control information
5. **Runs the Pipecat bot** with the call information

<Card title="Complete Server Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/telnyx-chatbot/outbound/server.py">
  See the full FastAPI server code with outbound call creation and WebSocket
  handling
</Card>

### Passing Custom Data to Your Bot

For dial-out calls, you often need to pass custom data to personalize the conversation—such as user information, campaign details, or context from your application. The dial-out example supports passing a `body` object that becomes available to your bot via `runner_args.body`.

**Triggering a call with custom data:**

```bash theme={null}
curl -X POST https://your-server.com/start \
  -H "Content-Type: application/json" \
  -d '{
    "phone_number": "+1234567890",
    "body": {
      "user": {
        "id": "user123",
        "firstName": "John",
        "lastName": "Doe",
        "accountType": "premium"
      },
      "campaign": "renewal_reminder",
      "context": "subscription expiring in 3 days"
    }
  }'
```

**How it works:**

1. The `/start` endpoint receives the request with `phone_number` and optional `body`
2. The `body` is base64-encoded and passed through the TeXML URL chain
3. When the WebSocket connects, the server decodes the body and passes it to your bot via `runner_args.body`
4. Your bot accesses the custom data to personalize the conversation

**Accessing custom data in your bot:**

```python bot.py theme={null}
async def bot(runner_args: RunnerArguments):
    body_data = runner_args.body or {}
    first_name = body_data.get("user", {}).get("firstName", "there")
    # Use first_name to personalize greetings, prompts, etc.
```

This approach works consistently in both local development and Pipecat Cloud production deployments.

### Configure your Pipecat bot for dial-out

The dial-out bot configuration is similar to dial-in, with Telnyx automatically providing call information.

**Call Information**: Telnyx provides call details in the WebSocket messages:

```python bot.py theme={null}
# Parse WebSocket data (same as dial-in)
transport_type, call_data = await parse_telephony_websocket(websocket)

# Extract call information
stream_id = call_data["stream_id"]
call_control_id = call_data["call_control_id"]
from_number = call_data["from"]  # Your Telnyx number
to_number = call_data["to"]      # Target number you're calling

# Customize bot behavior for outbound calls
greeting = f"Hi! This is an automated call from {from_number}. How are you today?"
```

**Custom Parameters for Dial-out**: You can pass custom data to your bot by adding query parameters to the WebSocket URL in your TeXML:

```python server.py theme={null}
# When creating the TeXML for outbound calls, include custom parameters
texml = f"""<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Connect>
    <Stream url="wss://your-server.com/ws?campaign_id=summer_sale&user_id={user_id}&priority=high" bidirectionalMode="rtp"></Stream>
  </Connect>
  <Pause length="40"/>
</Response>"""
```

These parameters are accessible in your server code and can be used to customize your bot's behavior based on the specific call context.

<Card title="Complete Bot Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/telnyx-chatbot/outbound/bot.py">
  See the full bot.py with outbound call handling and call information usage
</Card>

### Run the Example

To test dial-out functionality:

1. **Start your server**: Run your FastAPI server
2. **Trigger a call**: Make an API request to start an outbound call
3. **Answer your phone**: The bot will call the specified number
4. **Talk to your bot**: Have a conversation with your AI agent

<Card title="Complete Setup Instructions" icon="book-open" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/telnyx-chatbot/outbound">
  See the full README with step-by-step setup, API usage, and outbound call
  configuration
</Card>

## Key Features

### Audio Format and Sample Rate

Telnyx Media Streaming uses 8kHz mono audio with 16-bit PCM encoding. Configure your pipeline accordingly:

```python theme={null}
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        audio_in_sample_rate=8000,
        audio_out_sample_rate=8000,
    ),
)
```

### Automatic Call Termination

When you provide Telnyx API credentials to the `TelnyxFrameSerializer`, it automatically ends calls when your pipeline ends:

```python theme={null}
serializer = TelnyxFrameSerializer(
    stream_id=stream_id,
    call_control_id=call_control_id,
    api_key=os.getenv("TELNYX_API_KEY"),  # Enables auto-termination
)
```

### Built-in Call Information

Unlike other providers, Telnyx automatically includes caller information (to/from numbers) in the WebSocket messages, eliminating the need for custom webhook servers in basic dial-in scenarios.

## Deployment

### Pipecat Cloud Deployment

For production deployment without managing your own infrastructure, use Pipecat Cloud:

<Card title="Telnyx WebSocket on Pipecat Cloud" icon="cloud" href="/deployment/pipecat-cloud/guides/telephony/telnyx-websocket">
  Deploy Telnyx WebSocket bots with automatic scaling and managed infrastructure
</Card>

### Self-Hosted Production Deployment

For fully self-hosted production deployment, ensure your servers are:

* Publicly accessible with HTTPS
* Able to handle concurrent WebSocket connections
* Properly configured with Telnyx API credentials
* Implementing proper error handling and logging

## Next Steps

* Explore the [complete examples](https://github.com/pipecat-ai/pipecat-examples/tree/main/telnyx-chatbot) for full implementations
* Learn about [Daily + Twilio SIP integration](./twilio-daily-sip) for advanced telephony scenarios
* Check out [Daily PSTN integration](./daily-pstn) for direct phone number provisioning


# Daily + Twilio SIP
Source: https://docs.pipecat.ai/guides/telephony/twilio-daily-sip

Complete guide to using Daily's WebRTC transport with Twilio's SIP services for dial-in and dial-out

## Things you'll need

* An active [Twilio](https://www.twilio.com) developer account with API credentials
* One or more Twilio provisioned phone numbers
* Daily API key for WebRTC transport
* The Twilio Python client library (`pip install twilio`)

<CardGroup>
  <Card title="SIP Dial-in Example" icon="phone-arrow-down-left" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-twilio-sip-dial-in">
    Complete dial-in implementation using Twilio SIP with Daily WebRTC transport
  </Card>

  <Card title="SIP Dial-out Example" icon="phone-arrow-up-right" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-twilio-sip-dial-out">
    Outbound calling using Daily WebRTC transport with Twilio SIP routing
  </Card>
</CardGroup>

## Phone Number Setup

You'll need Twilio phone numbers for both dial-in and dial-out functionality:

* Visit [console.twilio.com](https://console.twilio.com) and purchase phone numbers
* Ensure your numbers support Voice capabilities
* Configure webhook URLs for dial-in numbers (covered below)

## Environment Setup

Configure your environment variables for both Twilio and Daily:

```shell .env theme={null}
DAILY_API_KEY=...
DAILY_API_URL=https://api.daily.co/v1
TWILIO_ACCOUNT_SID=...
TWILIO_AUTH_TOKEN=...
OPENAI_API_KEY=...
CARTESIA_API_KEY=...
```

## Dial-in

Dial-in allows users to call your Twilio number and connect to your Pipecat bot via Daily's WebRTC transport.

### How It Works

Here's the sequence of events when someone calls your Twilio number:

1. **Twilio receives an incoming call** to your phone number
2. **Twilio calls your webhook server** (`/call` endpoint)
3. **Your server creates a Daily room** with SIP capabilities
4. **Your server starts the bot process** with the room details
5. **Your server responds to Twilio** with TwiML that puts the caller on hold with music
6. **Upon receiving the `on_dialin_ready` event**, the bot forwards the call to the Daily SIP endpoint
7. **The caller and bot are connected**, and the bot handles the conversation

### Set up your webhook server

Your server acts as the orchestrator between Twilio's telephony infrastructure and Daily's WebRTC transport. When Twilio receives a call, it sends a webhook to your server. Your server then:

1. **Extracts call information** from Twilio's webhook (CallSid, caller details)
2. **Creates a Daily room** with SIP capabilities enabled
3. **Spawns a bot process** with the room details and call information
4. **Responds to Twilio** with TwiML that puts the caller on hold with music
5. **Coordinates the connection** between Twilio SIP and Daily WebRTC

The server handles the complex orchestration between two different telephony systems, ensuring seamless audio quality through Daily's WebRTC transport while leveraging Twilio's robust SIP infrastructure.

<Card title="Complete Server Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/phone-chatbot/daily-twilio-sip-dial-in/server.py">
  See the full FastAPI server code with Twilio webhook handling and Daily room
  creation
</Card>

### Configure your Pipecat bot for dial-in

The bot receives arguments from the server process and uses them to coordinate between Daily's WebRTC transport and Twilio's SIP infrastructure. The key components are:

**Bot Entry Point**: The bot receives arguments containing:

* `room_url`: Daily room URL for the WebRTC connection
* `token`: Daily room token for authentication
* `call_sid`: Twilio call identifier for SIP forwarding
* `sip_endpoint`: Daily's SIP endpoint URL for call routing

**Transport Creation**: These arguments are used to configure the `DailyTransport` for WebRTC:

```python bot.py theme={null}
from pipecat.transports.daily import DailyTransport, DailyParams

async def run_bot(room_url: str, token: str, call_sid: str, sip_endpoint: str):
    # Create Daily WebRTC transport
    transport = DailyTransport(
        room_url,
        token,
        "Voice Bot",
        DailyParams(
            audio_in_enabled=True,
            audio_out_enabled=True,
            video_out_enabled=False,
            transcription_enabled=True,
        )
    )

    # Handle when Daily SIP endpoint is ready for call forwarding
    @transport.event_handler("on_dialin_ready")
    async def on_dialin_ready(transport, cdata):
        # Forward the Twilio call to Daily's SIP endpoint
        twilio_client.calls(call_sid).update(
            twiml=f"<Response><Dial><Sip>{sip_endpoint}</Sip></Dial></Response>"
        )
        logger.info("Call forwarded from Twilio to Daily SIP endpoint")

    # Your bot pipeline setup here...
```

**Call Forwarding Flow**: The `on_dialin_ready` event is crucial. It signals when Daily's SIP infrastructure is ready to receive the call. At this point, the bot uses Twilio's API to update the call with new TwiML that forwards the audio from Twilio's SIP to Daily's SIP endpoint, completing the connection.

<Card title="Complete Bot Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/phone-chatbot/daily-twilio-sip-dial-in/bot.py">
  See the full bot.py with Daily transport setup and Twilio call forwarding
</Card>

### Set up Twilio webhook

Configure your Twilio phone number to use your server's webhook URL:

1. Go to the [Twilio Console](https://console.twilio.com)
2. Navigate to Phone Numbers → Manage → Active Numbers
3. Click on your phone number
4. Under "Configure", set "A Call Comes In" to:
   * Webhook: `https://your-server.com/call` (your server's URL)
   * HTTP Method: POST

### Run the Example

For local development, you can use [ngrok](https://ngrok.com/) to expose your local server:

```shell theme={null}
# Start your server
python server.py

# In another terminal, start ngrok
ngrok http 8000

# Use the ngrok URL (e.g., https://a1b2c3.ngrok.io/call) as your webhook
```

<Card title="Complete Setup Instructions" icon="book-open" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-twilio-sip-dial-in">
  See the full README with step-by-step setup, webhook configuration, and
  troubleshooting
</Card>

## Dial-out

Dial-out allows your bot to initiate calls through Twilio's SIP infrastructure using Daily's WebRTC transport.

### How It Works

Here's the sequence of events for dial-out calls:

1. **Your application triggers a dial-out** (via API call or user action)
2. **Server creates a Daily room** with SIP capabilities
3. **Bot joins the Daily room** and sets up the WebRTC transport
4. **Bot initiates SIP call** through Twilio to the target number
5. **Twilio routes the call** to the destination phone number
6. **The recipient answers** and is connected to your bot via Daily WebRTC
7. **The bot handles the conversation** with high-quality WebRTC audio

### Set up your server for dial-out

The dial-out server orchestrates outbound calls by coordinating between your application, Daily's WebRTC transport, and Twilio's SIP infrastructure. When you trigger a dial-out call, your server:

1. **Receives the dial-out request** with target phone number and call parameters
2. **Creates a Daily room** with SIP capabilities and dial-out enabled
3. **Spawns a bot process** with the room details and target phone number
4. **Initiates the SIP call** through Twilio to the destination number
5. **Manages the connection** between Daily WebRTC and Twilio SIP routing
6. **Handles call events** like answered, busy, or failed connections

The server simplifies outbound calling by abstracting the complexity of coordinating between WebRTC transport and SIP telephony, providing high-quality audio through Daily while leveraging Twilio's reliable call routing.

<Card title="Complete Server Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/phone-chatbot/daily-twilio-sip-dial-out/server.py">
  See the full FastAPI server code with Daily room creation and SIP dial-out
  management
</Card>

### Configure your Pipecat bot for dial-out

The dial-out bot receives arguments from the server process and uses them to initiate outbound calls through Twilio's SIP infrastructure via Daily's WebRTC transport. The key components are:

**Bot Entry Point**: The bot receives arguments containing:

* `room_url`: Daily room URL for the WebRTC connection
* `token`: Daily room token for authentication
* `target_number`: Phone number to call via Twilio SIP
* `sip_uri`: Twilio SIP URI for routing the outbound call

**Transport Creation**: These arguments are used to configure the `DailyTransport` for dial-out:

```python bot.py theme={null}
from pipecat.transports.daily import DailyTransport, DailyParams

async def run_bot(room_url: str, token: str, target_number: str, sip_uri: str):
    # Create Daily WebRTC transport
    transport = DailyTransport(
        room_url,
        token,
        "Voice Bot",
        DailyParams(
            audio_in_enabled=True,
            audio_out_enabled=True,
            video_out_enabled=False,
            transcription_enabled=True,
        )
    )

    # Initiate the outbound call through Twilio SIP
    await transport.start_dialout(sip_uri)

    # Your bot pipeline setup here...
```

**Dial-out Flow**: The `start_dialout()` method initiates the call by connecting Daily's WebRTC transport to Twilio's SIP infrastructure. The `sip_uri` parameter contains the Twilio SIP endpoint configured with the target phone number, allowing Daily to route the call through Twilio's telephony network to reach the destination.

<Card title="Complete Bot Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/phone-chatbot/daily-twilio-sip-dial-out/bot.py">
  See the full bot.py with WebRTC transport setup and SIP dial-out configuration
</Card>

### Run the Example

To test dial-out functionality:

1. **Start your server**: Run your FastAPI server
2. **Trigger a call**: Make an API request to start a dial-out call
3. **Answer your phone**: The bot will call the specified number via Twilio
4. **Talk to your bot**: Have a conversation with high-quality WebRTC audio

<Card title="Complete Setup Instructions" icon="book-open" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot/daily-twilio-sip-dial-out">
  See the full README with step-by-step setup, API usage, and SIP configuration
</Card>

## Best Practices

### Hold Music for Initialization

Always respond to Twilio's initial webhook with hold music to give your bot time to initialize:

```python theme={null}
resp = VoiceResponse()
resp.play(url="https://your-hold-music.mp3", loop=10)
return str(resp)
```

**Avoid using `<Pause>`** - Twilio's pause duration is limited and may not provide enough time for Daily SIP setup.

### Handle Multiple Events

Use a flag to ensure you only forward calls once when handling multiple `on_dialin_ready` events:

```python theme={null}
call_already_forwarded = False

@transport.event_handler("on_dialin_ready")
async def on_dialin_ready(transport, cdata):
    nonlocal call_already_forwarded
    if call_already_forwarded:
        return
    # Forward the call...
    call_already_forwarded = True
```

## Deployment

### Local Development

Use [ngrok](https://ngrok.com/) to expose your local server for testing:

```shell theme={null}
python server.py
ngrok http 8000
# Use the ngrok URL as your Twilio webhook
```

### Pipecat Cloud Deployment

For production deployment on Pipecat Cloud, your webhook server calls the `/{agent}/start` endpoint instead of spawning local bot processes:

```python theme={null}
# Instead of subprocess.Popen(bot_cmd)
response = await httpx.post(
    f"https://api.pipecat.daily.co/v1/public/{agent_id}/start",
    headers={"Authorization": f"Bearer {pipecat_api_key}"},
    json={
        "createDailyRoom": True,
        "dailyRoomProperties": {
            "sip": {"sip_mode": "dial-in", "num_endpoints": 1}
        },
        "body": {
            "room_url": room_url,
            "token": token,
            "call_sid": call_sid,  # For dial-in
            "sip_endpoint": sip_endpoint,
            # Or for dial-out:
            # "target_number": target_number,
            # "sip_uri": sip_uri
        }
    }
)
```

The room information and call parameters are passed in the `body` field, which your Pipecat Cloud bot receives as arguments.

### Self-Hosted Production Deployment

For self-hosted production deployment, ensure your servers are:

* Publicly accessible with HTTPS
* Able to handle concurrent webhook requests
* Properly configured with both Twilio and Daily API credentials

## Next Steps

* Explore the [complete examples](https://github.com/pipecat-ai/pipecat-examples/tree/main/phone-chatbot) for full implementations
* Learn about [Daily PSTN integration](./daily-pstn) for direct phone number provisioning
* Check out [Twilio WebSocket integration](./twilio-websockets) for simpler telephony workflows


# Twilio WebSocket Integration
Source: https://docs.pipecat.ai/guides/telephony/twilio-websockets

Complete guide to using Twilio Media Streams with Pipecat for dial-in and dial-out functionality

## Things you'll need

* An active [Twilio](https://www.twilio.com) account with API credentials
* One or more Twilio provisioned phone numbers
* A tunneling service like [ngrok](https://ngrok.com/) for local development
* API keys for speech-to-text, text-to-speech, and LLM services

<CardGroup>
  <Card title="Twilio Dial-in Example" icon="phone-arrow-down-left" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/twilio-chatbot/inbound">
    Complete dial-in implementation using Twilio Media Streams over WebSocket
  </Card>

  <Card title="Twilio Dial-out Example" icon="phone-arrow-up-right" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/twilio-chatbot/outbound">
    Outbound calling using Twilio Media Streams with WebSocket transport
  </Card>
</CardGroup>

## Phone Number Setup

You'll need Twilio phone numbers for both dial-in and dial-out functionality:

* Visit [console.twilio.com](https://console.twilio.com) and purchase phone numbers
* Ensure your numbers support Voice capabilities
* Configure webhook URLs for dial-in numbers (covered below)

## Environment Setup

Configure your environment variables for Twilio and AI services:

```shell .env theme={null}
TWILIO_ACCOUNT_SID=...
TWILIO_AUTH_TOKEN=...
OPENAI_API_KEY=...
DEEPGRAM_API_KEY=...
CARTESIA_API_KEY=...
```

## Dial-in

Dial-in allows users to call your Twilio number and connect to your Pipecat bot via WebSocket Media Streams.

### How It Works

Here's the sequence of events when someone calls your Twilio number:

1. **Twilio sends WebSocket messages**: Twilio processes the associated TwiML Bin and starts a WebSocket stream to your bot (local or Pipecat Cloud)
2. **Parse the WebSocket messages**: Your bot parses the WebSocket connection messages to set up the corresponding Pipecat transport
3. **(Optional) Look up the caller**: Optionally, look up the caller using Twilio's REST API to retrieve custom information about the call and personalize your bot's behavior
4. **Bot starts responding**: Once the pipeline is started, your bot will initiate the conversation

### Setting Up Twilio

#### 1. Create a TwiML Bin

A TwiML Bin tells Twilio how to handle incoming calls. You'll create one that establishes a WebSocket connection to your bot.

1. Go to the [Twilio Console](https://console.twilio.com)
2. Navigate to **TwiML Bins** → **My TwiML Bins**
3. Click the **+** to create a new TwiML Bin
4. Name your bin and add the TwiML:

<Tabs>
  <Tab title="Local Development">
    ```xml theme={null}
    <?xml version="1.0" encoding="UTF-8"?>
    <Response>
      <Connect>
        <Stream url="wss://your-url.ngrok.io/ws" />
      </Connect>
    </Response>
    ```

    Replace `your-url.ngrok.io` with your ngrok URL.
  </Tab>

  <Tab title="Pipecat Cloud">
    ```xml theme={null}
    <?xml version="1.0" encoding="UTF-8"?>
    <Response>
      <Connect>
        <Stream url="wss://api.pipecat.daily.co/ws/twilio">
          <Parameter name="_pipecatCloudServiceHost"
            value="AGENT_NAME.ORGANIZATION_NAME"/>
        </Stream>
      </Connect>
    </Response>
    ```

    Replace:

    * `AGENT_NAME` with the name of the agent you deployed to Pipecat Cloud
    * `ORGANIZATION_NAME` with the name of your Pipecat Cloud organization
  </Tab>
</Tabs>

5. Click **Save**

#### 2. Assign TwiML Bin to Your Phone Number

1. Navigate to **Phone Numbers** → **Manage** → **Active Numbers**
2. Click on your Twilio phone number
3. In the "Voice Configuration" section:
   * Set "A call comes in" to **TwiML Bin**
   * Select the TwiML Bin you created
4. Click **Save configuration**

### Running the Example

You'll need two terminal windows to run the example.

1. In your first terminal, start ngrok to expose your local server:

   ```shell theme={null}
   ngrok http 7860
   ```

   > Tip: Use the `--subdomain` flag for a reusable ngrok URL.

2. In your second terminal, install dependencies:

   ```shell theme={null}
   uv sync
   ```

3. Then run the bot:

   ```shell theme={null}
   uv run bot.py -t twilio
   ```

4. Call your bot by placing a call to the number associated with your bot. The bot will answer and start the conversation.

### Personalizing Your Bot with Caller Information

When a call comes in, your bot receives a Call SID from Twilio. You can use this Call SID to fetch caller information from Twilio's REST API, including the caller's phone number. With this information, you can:

* Look up customer information in your database
* Personalize greetings and responses based on caller identity
* Route calls to different bot behaviors (e.g., VIP handling, support vs sales)
* Implement custom business logic based on the caller

The example bot demonstrates how to fetch caller information asynchronously using `aiohttp` to avoid blocking the event loop, ensuring your bot remains responsive even with multiple concurrent calls.

<Card title="Complete Dial-in Example" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/twilio-chatbot/inbound">
  See the full implementation with WebSocket parsing, REST API integration, and
  caller personalization
</Card>

## Dial-out

Dial-out allows your bot to initiate calls to phone numbers using Twilio's outbound calling capabilities with WebSocket Media Streams.

### How It Works

Here's the sequence of events for dial-out calls:

1. **Your application triggers a dial-out** (via API call or user action)
2. **Server initiates a Twilio call** to the target phone number
3. **Twilio establishes the call** and opens a WebSocket connection
4. **Your bot joins the WebSocket** and sets up the pipeline
5. **The recipient answers** and is connected to your bot
6. **The bot handles the conversation** with real-time audio streaming

### Set up your server for dial-out

The dial-out server creates outbound calls and manages WebSocket connections. When you trigger a dial-out call, your server:

1. **Receives the dial-out request** with target phone number and parameters
2. **Creates a Twilio call** using the REST API with TwiML that establishes WebSocket
3. **Accepts the WebSocket** connection from Twilio
4. **Parses call data** including any custom parameters
5. **Runs the Pipecat bot** with the call information

<Card title="Complete Server Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/twilio-chatbot/outbound/server.py">
  See the full FastAPI server code with outbound call creation and WebSocket
  handling
</Card>

### Configure your Pipecat bot for dial-out

The dial-out bot configuration is similar to dial-in, but you can pass custom parameters when creating the outbound call:

**Custom Parameters for Dial-out**: Pass information to your bot through TwiML parameters:

```python server.py theme={null}
# When creating the outbound call, include custom parameters
twiml = f"""
<Response>
  <Connect>
    <Stream url="wss://your-server.com/ws">
      <Parameter name="campaign_id" value="summer_sale" />
      <Parameter name="customer_id" value="{customer_id}" />
      <Parameter name="call_type" value="outbound" />
    </Stream>
  </Connect>
</Response>
"""

# Create the outbound call
call = twilio_client.calls.create(
    to=target_number,
    from_=your_twilio_number,
    twiml=twiml
)
```

**Bot Configuration**: The bot receives and uses these parameters:

```python bot.py theme={null}
# Parse WebSocket data (same as dial-in)
transport_type, call_data = await parse_telephony_websocket(websocket)

# Access custom parameters
custom_params = call_data["body"]
campaign_id = custom_params.get("campaign_id")
customer_id = custom_params.get("customer_id")

# Customize bot behavior for outbound calls
if campaign_id == "summer_sale":
    greeting = f"Hi! I'm calling about our summer sale promotion..."
```

<Card title="Complete Bot Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/twilio-chatbot/outbound/bot.py">
  See the full bot.py with outbound call handling and parameter usage
</Card>

### Run the Example

To test dial-out functionality:

1. **Start your server**: Run your FastAPI server
2. **Trigger a call**: Make an API request to start an outbound call
3. **Answer your phone**: The bot will call the specified number
4. **Talk to your bot**: Have a conversation with your AI agent

<Card title="Complete Setup Instructions" icon="book-open" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/twilio-chatbot/outbound">
  See the full README with step-by-step setup, API usage, and outbound call
  configuration
</Card>

## Key Features

### Audio Format and Sample Rate

Twilio Media Streams uses 8kHz mono audio with 16-bit PCM encoding. To avoid resampling, set the audio input and output sample rate to 8000 Hz:

```python theme={null}
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        audio_in_sample_rate=8000,
        audio_out_sample_rate=8000,
    ),
)
```

### Automatic Call Termination

When you provide Twilio credentials to the `TwilioFrameSerializer`, it automatically ends calls when your pipeline ends:

```python theme={null}
serializer = TwilioFrameSerializer(
    stream_sid=stream_id,
    call_sid=call_id,
    account_sid=os.getenv("TWILIO_ACCOUNT_SID"),
    auth_token=os.getenv("TWILIO_AUTH_TOKEN"),
)
```

## Deployment

### Pipecat Cloud Deployment

For production deployment without managing your own infrastructure, use Pipecat Cloud:

<Card title="Twilio WebSocket on Pipecat Cloud" icon="cloud" href="/deployment/pipecat-cloud/guides/telephony/twilio-websocket">
  Deploy Twilio WebSocket bots with automatic scaling and managed infrastructure
</Card>

### Self-Hosted Production Deployment

For fully self-hosted production deployment, ensure your servers are:

* Publicly accessible with HTTPS
* Able to handle concurrent WebSocket connections
* Properly configured with Twilio API credentials
* Implementing proper error handling and logging

## Next Steps

* Explore the [complete examples](https://github.com/pipecat-ai/pipecat-examples/tree/main/twilio-chatbot) for full implementations
* Learn about [Daily + Twilio SIP integration](./twilio-daily-sip) for advanced telephony scenarios
* Check out [Daily PSTN integration](./daily-pstn) for direct phone number provisioning


# Recipes
Source: https://docs.pipecat.ai/recipes

Code snippets and techniques to enhance your Pipecat applications

<Update label={<><Icon icon="code" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Function Calling</span></>}>
  Add function calling to your Pipecat bot. Examples exist for each LLM provider supported in Pipecat.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14-function-calling.py)
</Update>

<Update label={<><Icon icon="microphone" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Record Audio</span></>}>
  Collect audio frames from the user and bot for later processing or storage.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/34-audio-recording.py)
</Update>

<Update label={<><Icon icon="file-lines" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Capture Transcripts</span></>}>
  Capture user and bot transcripts for later processing or storage.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/28-transcription-processor.py)
</Update>

<Update label={<><Icon icon="volume-high" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Play Background Sound</span></>}>
  Play a background sound in your Pipecat bot. The audio is mixed with the transport audio to create a single integrated audio stream.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/23-bot-background-sound.py)
</Update>

<Update label={<><Icon icon="microphone-slash" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Mute User Input</span></>}>
  Specify a strategy for mute to mute user input, allowing the bot to continue without interruption.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/24-stt-mute-filter.py)
</Update>

<Update label={<><Icon icon="bell" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Wake Phrase</span></>}>
  Use a wake phrase to wake up your Pipecat bot.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/10-wake-phrase.py)
</Update>

<Update label={<><Icon icon="music" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Play Sound Effects</span></>}>
  Play sound effects in your Pipecat bot.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/11-sound-effects.py)
</Update>

<Update label={<><Icon icon="language" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Switch Languages</span></>}>
  A ParallelPipeline example showing how to dynamically switch languages.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/15a-switch-languages.py)
</Update>

<Update label={<><Icon icon="hourglass" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Detect an Idle User</span></>}>
  Detect when a user is idle and automatically respond.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/17-detect-user-idle.py)
</Update>

<Update label={<><Icon icon="bug" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Debug with an Observer</span></>}>
  Learn how to debug your Pipecat bot with an Observer by observing frames flowing through the pipeline.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/30-observer.py)
</Update>

<Update label={<><Icon icon="monitor-waveform" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Live Debugger (Whisker)</span></>}>
  A live graphical debugger for the Pipecat voice and multimodal conversational AI framework. It lets you visualize pipelines and debug frames in real time — so you can see exactly what your bot is thinking and doing.

  [View Recipe →](https://github.com/pipecat-ai/whisker)
</Update>

<Update label={<><Icon icon="envelope" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Collect Emails</span></>}>
  Parse user email from the LLM response.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/36-user-email-gathering.py)
</Update>

<Update label={<><Icon icon="brain" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Smart Turn Detection</span></>}>
  Detect when a user has finished speaking and automatically respond. Learn more about [smart-turn model](https://github.com/pipecat-ai/smart-turn).

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/38b-smart-turn-local.py)
</Update>

<Update label={<><Icon icon="screwdriver-wrench" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>MCP Tools</span></>}>
  Use MCP tools to interact with external services.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/39-mcp-stdio.py)
</Update>

<Update label={<><Icon icon="hand" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Interruption Strategies</span></>}>
  Learn how to configure interruption strategies for your Pipecat bot.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/42-interruption-config.py)
</Update>

<Update label={<><Icon icon="video" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>Describe Video</span></>}>
  Pass a video frame from a live video stream to a model and get a description.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/12b-describe-video-gpt-4o.py)
</Update>

<Update label={<><Icon icon="users" size={16} color="currentColor" /><span style={{marginLeft: '8px'}}>User and Bot Turn Events</span></>}>
  Handle user and bot end of turn events to add custom logic after a turn.

  [View Recipe →](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/29-turn-tracking-observer.py)
</Update>

***

## 💡 Need Help Getting Started?

<CardGroup>
  <Card title="More Recipes" icon="github" href="https://github.com/pipecat-ai/pipecat/tree/main/examples/foundational">
    Browse all recipes on GitHub
  </Card>

  <Card title="Examples" icon="folder" href="/examples">
    Browse complete applications
  </Card>

  <Card title="Discord Community" icon="discord" href="https://discord.gg/pipecat">
    Get help from the community
  </Card>
</CardGroup>


# Pipecat Flows
Source: https://docs.pipecat.ai/server/frameworks/flows/pipecat-flows

Reference docs for Pipecat’s conversation flow system

<Tip>
  New to building conversational flows? Check out our [Pipecat Flows
  guide](/guides/features/pipecat-flows) first.
</Tip>

Pipecat Flows is an add-on framework for Pipecat that allows you to build structured conversations in your AI applications. It enables you to define conversation paths while handling the complexities of state management and LLM interactions.

<CardGroup>
  <Card title="API Reference" icon="code" href="https://reference-flows.pipecat.ai">
    Complete API documentation and method details
  </Card>

  <Card title="GitHub Repository" icon="github" href="https://github.com/pipecat-ai/pipecat-flows">
    Source code, examples, and issue tracking
  </Card>

  <Card title="Hello World Example" icon="play" href="https://github.com/pipecat-ai/pipecat-flows/blob/main/examples/quickstart/hello_world.py">
    Working example with basic conversation flow
  </Card>
</CardGroup>

## Installation

### Pipecat Flows

To use Pipecat Flows, install the required dependency:

```bash theme={null}
pip install pipecat-ai-flows
```

### Pipecat Dependencies

For fresh installations, you'll need to install Pipecat with depdencies for your Transport, STT, LLM, and TTS providers.

For example, to use Daily, OpenAI, Deepgram, Cartesia, and Silero:

```bash theme={null}
pip install "pipecat-ai[daily,openai,deepgram,cartesia,silero]"
```

## Flow Types

### Dynamic Flows (Recommended)

Create conversation paths at runtime based on user input, external data, or business logic. Enables more flexible and adaptive interactions. Additionally, can handle simple scenarios with predefined paths.

<Note>
  Dynamic Flows are the recommended approach for most applications, as they
  allow for greater flexibility and adaptability in conversation design.
</Note>

### Static Flows

Use predefined conversation paths configured upfront. Ideal for predefined interactions where the graph navigation can be predetermined.

## Key Components

### Core Classes

* **FlowManager** - Main orchestration class for managing conversation flows
* **FlowsFunctionSchema** - Standardized function call schema that works with any LLM provider

### Type Definitions

* **FlowArgs** - Function handler arguments
* **FlowResult** - Function return type with status and optional data
* **NodeConfig** - Complete node configuration including messages, functions, and actions
* **FlowConfig** - Static flow configuration with nodes and initial state

### Context Management

* **ContextStrategyConfig** - Configuration for context management during transitions
* **ContextStrategy** - Enum for APPEND, RESET, and RESET\_WITH\_SUMMARY strategies

### Actions

* **ActionConfig** - Configuration for custom pre- and post-actions during conversation flow

## Function Types

### Node Functions

Execute operations within a single conversation state without switching nodes. Return `(FlowResult, None)`.

### Edge Functions

Create transitions between conversation states, optionally processing data first. Return `(FlowResult, NodeConfig)` or `(FlowResult, str)`.

### Direct Functions

Functions passed directly to NodeConfig with automatic metadata extraction from signatures and docstrings.

## LLM Provider Support

Pipecat Flows automatically handles format differences between providers:

| Provider      | Format Support        | Installation                          |
| ------------- | --------------------- | ------------------------------------- |
| OpenAI        | Function calling      | `pip install "pipecat-ai[openai]"`    |
| Anthropic     | Native tools          | `pip install "pipecat-ai[anthropic]"` |
| Google Gemini | Function declarations | `pip install "pipecat-ai[google]"`    |
| AWS Bedrock   | Anthropic-compatible  | `pip install "pipecat-ai[aws]"`       |

## Error Handling

Comprehensive exception hierarchy for flow management:

* **FlowError** - Base exception for all flow-related errors
* **FlowInitializationError** - Flow manager setup failures
* **FlowTransitionError** - Node transition failures
* **InvalidFunctionError** - Function registration or execution errors
* **ActionError** - Action execution failures

## Additional Notes

* **State Management**: Use `flow_manager.state` dictionary for persistent conversation data
* **Automatic Function Call Registration and Validation**: All functions are automatically registered and validated at run-time
* **Provider Compatibility**: Format differences handled automatically via adapter system
* **Deprecation**: Some legacy patterns (`transition_to`, `transition_callback`) are deprecated in favor of consolidated function handlers


# RTVI (Real-Time Voice Interaction)
Source: https://docs.pipecat.ai/server/frameworks/rtvi/introduction

Build real-time voice and multimodal applications with Pipecat’s RTVI protocol

Pipecat's RTVI (Real-Time Voice Interaction) protocol provides a standardized communication layer between clients and servers for building real-time voice and multimodal applications. It handles the synchronization of user and bot interactions, transcriptions, LLM processing, and text-to-speech delivery.

This page provides an overview of RTVI from the server's perspective and how to use it in your bot applications.

<Card title="RTVI Protocol" icon="left-right" href="/client/rtvi-standard">
  A complete specification of the RTVI protocol for client-server communication.
</Card>

## Architecture

RTVI operates with two primary components:

1. [**RTVIProcessor**](./rtvi-processor) - A frame processor residing in the pipeline that serves as the entry point for sending and receiving messages to/from the client.

2. [**RTVIObserver**](./rtvi-observer) - An observer that monitors pipeline events and translates them into client-compatible messages, handling:
   * Speaking state changes
   * Transcription updates
   * LLM responses
   * TTS events
   * Performance metrics

<Note>
  RTVI is enabled by default. When you create a `PipelineTask`, it automatically adds `RTVIProcessor` to the start of your pipeline and registers an `RTVIObserver`. The default `on_client_ready` handler calls `set_bot_ready()` automatically.
</Note>

## Basic Example

With automatic RTVI setup, your pipeline code can focus on core functionality:

```python theme={null}
pipeline = Pipeline(
    [
        transport.input(),
        stt,
        context_aggregator.user(),
        llm,
        tts,
        transport.output(),
        context_aggregator.assistant(),
    ]
)

# Add the RTVIObserver to your pipeline task
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        enable_metrics=True,
        enable_usage_metrics=True,
    ),
)

# Access the RTVI processor via task.rtvi
@task.rtvi.event_handler("on_client_ready")
async def on_client_ready(rtvi):
    # set_bot_ready() is called automatically, add custom logic here
    await task.queue_frames([LLMRunFrame()])

# Handle participant disconnection
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
    await task.cancel()

# Run the pipeline
runner = PipelineRunner()
await runner.run(task)
```

## Customizing RTVI

You can customize RTVI behavior through `PipelineTask` parameters:

```python theme={null}
from pipecat.processors.frameworks.rtvi import RTVIProcessor, RTVIObserverParams

task = PipelineTask(
    pipeline,
    rtvi_processor=RTVIProcessor(),  # Provide your own processor
    rtvi_observer_params=RTVIObserverParams(...),  # Customize observer
)
```

To disable RTVI entirely:

```python theme={null}
task = PipelineTask(pipeline, enable_rtvi=False)
```

## Protocol Flow

1. Client connects and sends a `client-ready` message
2. Server responds with `bot-ready` and initial configuration
3. Client and server exchange real-time events:
   * Speaking state changes (`user/bot-started/stopped-speaking`)
   * Transcriptions (`user-transcription/bot-output`)
   * LLM processing (`bot-llm-started/stopped`, `bot-llm-text`, `llm-function-call`)
   * TTS events (`bot-tts-started/stopped`, `bot-tts-text`, `bot-tts-audio`)

## Key Components

<CardGroup>
  <Card title="RTVIProcessor" icon="gear" href="/server/frameworks/rtvi/rtvi-processor">
    Configure and manage RTVI services, actions, and client communication
  </Card>

  <Card title="RTVIObserver" icon="magnifying-glass" href="/server/frameworks/rtvi/rtvi-observer">
    Translate internal pipeline events to standardized client messages
  </Card>
</CardGroup>

## Client Integration

RTVI is implemented in Pipecat client SDKs, providing a high-level API to interact with the protocol. Visit the Pipecat Client SDKs documentation:

<Card title="Client SDKs" icon="mobile-screen" href="/client/introduction">
  Learn how to implement RTVI on the client-side with our JavaScript, React, and
  mobile SDKs
</Card>


# RTVI Observer
Source: https://docs.pipecat.ai/server/frameworks/rtvi/rtvi-observer

Converting pipeline frames to RTVI protocol messages

The `RTVIObserver` translates Pipecat's internal pipeline events into standardized RTVI protocol messages. It monitors frame flow through the pipeline and generates corresponding client messages based on event types.

## Purpose

The `RTVIObserver` primarily serves to convert internal pipeline frames into to client-compatible RTVI messages. It is required for any application using RTVI as the client protocol to ensure proper communication of events such as speech start/stop, user transcript, bot output, metrics, and server messages.

## Automatic Setup

`RTVIObserver` is automatically created and attached when you create a `PipelineTask`. No manual setup is required for standard usage.

To customize the observer's behavior, pass `RTVIObserverParams` to the task:

```python theme={null}
from pipecat.processors.frameworks.rtvi import RTVIObserverParams

task = PipelineTask(
    pipeline,
    rtvi_observer_params=RTVIObserverParams(
        bot_llm_enabled=False,
        metrics_enabled=False,
    ),
)
```

## Configuration

`RTVIObserverParams` accepts the following fields:

<ParamField>
  Indicates if bot output messages should be sent.
</ParamField>

<ParamField>
  Indicates if the bot's LLM messages should be sent.
</ParamField>

<ParamField>
  Indicates if the bot's TTS messages should be sent.
</ParamField>

<ParamField>
  Indicates if the bot's started/stopped speaking messages should be sent.
</ParamField>

<ParamField>
  Indicates if bot's audio level messages should be sent.
</ParamField>

<ParamField>
  Indicates if the user's LLM input messages should be sent.
</ParamField>

<ParamField>
  Indicates if the user's started/stopped speaking messages should be sent.
</ParamField>

<ParamField>
  Indicates if user's transcription messages should be sent.
</ParamField>

<ParamField>
  Indicates if user's audio level messages should be sent.
</ParamField>

<ParamField>
  Indicates if metrics messages should be sent.
</ParamField>

<ParamField>
  Indicates if system logs should be sent.
</ParamField>

<ParamField>
  ⚠️ **Deprecated**: Indicates if errors messages should be sent.
</ParamField>

<ParamField>
  List of aggregation types to skip sending as tts/output messages.

  <Note>
    If using this to avoid sending secure information, be sure to also disable bot\_llm\_enabled
    to avoid leaking through LLM messages.
  </Note>
</ParamField>

<ParamField>
  A list of tuples to transform text just before sending it to TTS. Each tuple should be of the form `(aggregation_type, transform_function)`, where `aggregation_type` is a string (or `'*'` for all types), and `transform_function` is a callable that takes `(text, aggregation_type)` and returns the transformed text.

  **Example:**

  ```python theme={null}
  def redact_sensitive(text, agg_type):
      # Example: redact numbers
      import re
      return re.sub(r"\d+", "[REDACTED]", text)

  bot_output_transforms = [
      ("credit_card", redact_sensitive),  # Only for 'credit_card' type
      ("*", lambda text, agg_type: text.upper()),  # For all types, make uppercase
  ]

  observer = RTVIObserver(
      rtvi,
      params=RTVIObserverParams(bot_output_transforms=bot_output_transforms),
  )
  ```
</ParamField>

<ParamField>
  How often audio levels should be sent if enabled.
</ParamField>

## Frame Translation

The observer maps Pipecat's internal frames to RTVI protocol messages:

| Pipeline Frame              | RTVI Message                                |
| --------------------------- | ------------------------------------------- |
| **Speech Events**           |                                             |
| `UserStartedSpeakingFrame`  | `RTVIUserStartedSpeakingMessage`            |
| `UserStoppedSpeakingFrame`  | `RTVIUserStoppedSpeakingMessage`            |
| `BotStartedSpeakingFrame`   | `RTVIBotStartedSpeakingMessage`             |
| `BotStoppedSpeakingFrame`   | `RTVIBotStoppedSpeakingMessage`             |
| **Transcription**           |                                             |
| `TranscriptionFrame`        | `RTVIUserTranscriptionMessage(final=true)`  |
| `InterimTranscriptionFrame` | `RTVIUserTranscriptionMessage(final=false)` |
| **Bot Output**              |                                             |
| `AggregatedTextFrame`       | `RTVIBotOutputMessage`                      |
| **LLM Processing**          |                                             |
| `LLMFullResponseStartFrame` | `RTVIBotLLMStartedMessage`                  |
| `LLMFullResponseEndFrame`   | `RTVIBotLLMStoppedMessage`                  |
| `LLMTextFrame`              | `RTVIBotLLMTextMessage`                     |
| **TTS Events**              |                                             |
| `TTSStartedFrame`           | `RTVIBotTTSStartedMessage`                  |
| `TTSStoppedFrame`           | `RTVIBotTTSStoppedMessage`                  |
| `TTSTextFrame`              | `RTVIBotTTSTextMessage`                     |
| **Context/Metrics**         |                                             |
| `LLMContextFrame`           | `RTVIUserLLMTextMessage`                    |
| `MetricsFrame`              | `RTVIMetricsMessage`                        |
| `RTVIServerMessageFrame`    | `RTVIServerMessage`                         |


# RTVIProcessor
Source: https://docs.pipecat.ai/server/frameworks/rtvi/rtvi-processor

Core coordinator for RTVI protocol communication

The `RTVIProcessor` manages bidirectional communication between clients and your Pipecat application. It processes client messages, handles service configuration, executes actions, and coordinates function calls.

## Initialization

`RTVIProcessor` is automatically added to your pipeline when you create a `PipelineTask`. Access it via `task.rtvi`:

```python theme={null}
pipeline = Pipeline([
    transport.input(),
    stt,
    # ... other processors ...
    transport.output()
])

task = PipelineTask(pipeline)

# Access the processor
rtvi = task.rtvi
```

To provide a custom processor (e.g., for [Google RTVI](./google-rtvi-observer)):

```python theme={null}
from pipecat.services.google.rtvi import GoogleRTVIProcessor

task = PipelineTask(pipeline, rtvi_processor=GoogleRTVIProcessor())
```

To disable RTVI entirely, set `enable_rtvi=False`.

## Readiness Protocol

### Client Ready State

Clients indicate readiness by sending a `client-ready` message, triggering the `on_client_ready` event in the processor:

```python theme={null}
@rtvi.event_handler("on_client_ready")
async def on_client_ready(rtvi):
    # Handle client ready state
    await rtvi.set_bot_ready()
    # Initialize conversation
    await task.queue_frames([...])
```

### Bot Ready State

The server must mark the bot as ready before it can process client messages:

```python theme={null}
await rtvi.set_bot_ready()
```

When marked ready, the bot sends a response containing:

* RTVI protocol version
* Current service configuration
* Available actions

## Services

Services represent configurable components of your application that clients can interact with.

### Registering Services

```python theme={null}
# 1. Define option handler
async def handle_voice_option(processor, service, option):
    voice_id = option.value
    # Apply configuration change
    logger.info(f"Voice ID updated to: {voice_id}")

# 2. Create RTVIService
voice_service = RTVIService(
    name="voice",
    options=[
        RTVIServiceOption(
            name="voice_id",
            type="string",
            handler=handle_voice_option
        )
    ]
)

# 3. Register with processor
rtvi.register_service(voice_service)
```

### Option Types

Services support multiple data types for configuration:

```python theme={null}
RTVIServiceOption(
    name="temperature",
    type="number",  # number, string, bool, array, object
    handler=handle_temperature
)
```

Option handlers receive:

* The processor instance
* The service name
* The option configuration with new value

## Actions

Actions are server-side functions that clients can trigger with arguments.

### Registering Actions

```python theme={null}
# 1. Define handler function
async def handle_print_message(processor, service, arguments):
    message = arguments.get("message", "Default message")
    logger.info(f"Print action triggered with message: {message}")
    return True

# 2. Create and register RTVIAction
print_action = RTVIAction(
    service="conversation",
    action="print_message",
    arguments=[
        RTVIActionArgument(name="message", type="string")
    ],
    result="bool",
    handler=handle_print_message
)
rtvi.register_action(print_action)
```

### Action Arguments

Actions can accept typed arguments from clients:

```python theme={null}
search_action = RTVIAction(
    service="knowledge",
    action="search",
    arguments=[
        RTVIActionArgument(name="query", type="string"),
        RTVIActionArgument(name="limit", type="number")
    ],
    result="array",
    handler=handle_search
)
```

## Function Calls

Handle LLM function calls with client interaction:

```python theme={null}

await processor.handle_function_call(
    function_name=function_name,
    tool_call_id=tool_call_id,
    arguments=arguments,
)

await processor.handle_function_call(params)
```

The function call process:

1. LLM requests a function call
2. Processor notifies client with `llm-function-call` message
3. Client executes function and returns result
4. Result is passed back to LLM via `FunctionCallResultFrame`
5. Conversation continues

## Error Handling

Send error messages to clients:

```python theme={null}
# General error
await processor.send_error("Invalid configuration")

# Request-specific error
await processor._send_error_response(request_id, "Invalid action arguments")
```

Error categories:

* Configuration errors
* Action execution errors
* Function call errors
* Protocol errors
* Fatal and non-fatal errors

## Bot Control

Manage bot state and handle interruptions:

```python theme={null}
# Set bot as ready
await processor.set_bot_ready()

# Handle interruptions
await processor.interrupt_bot()
```

## Custom Messaging

Send custom messages from server to client:

```python theme={null}
from pipecat.processors.frameworks.rtvi import RTVIServerMessageFrame

# Send a custom message
frame = RTVIServerMessageFrame(
    data={
        "type": "custom-event",
        "payload": {"key": "value"}
    }
)
await rtvi.push_frame(frame)
```

The client receives these messages through the `onServerMessage` callback or `serverMessage` event.


# Server API Reference
Source: https://docs.pipecat.ai/server/introduction

Complete reference for Pipecat’s Python server APIs and services

## Overview

Pipecat's server-side functionality is organized into three main categories:

<CardGroup>
  <Card title="Services" icon="puzzle-piece">
    AI service integrations for speech, language, vision, and more
  </Card>

  <Card title="Frameworks" icon="puzzle">
    > User RTVI for client/server communication or Pipecat Flows for building structured conversations
  </Card>

  <Card title="Utilities" icon="wrench">
    Helper functions and tools for audio, filtering, and flows
  </Card>
</CardGroup>

## Service Categories

Pipecat integrates with various AI services across different categories:

<CardGroup>
  <Card title="Transport" icon="network-wired">
    WebRTC and WebSocket implementations for real-time communication
  </Card>

  <Card title="Speech Processing" icon="waveform-lines">
    Speech-to-text, text-to-speech, and speech-to-speech services
  </Card>

  <Card title="Language Models" icon="brain">
    Integration with various LLM providers
  </Card>

  <Card title="Vision & Media" icon="image">
    Image generation, video processing, and computer vision
  </Card>
</CardGroup>

## Getting Started

1. Browse our [Supported Services](/server/services/supported-services) to see available integrations
2. Install required dependencies for your chosen services
3. Reference individual service docs for detailed configuration options

## Example Usage

```python theme={null}
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.audio.vad.vad_analyzer import VADParams
from pipecat.pipeline import Pipeline
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.services.cartesia.tts import CartesiaTTSService
from pipecat.services.deepgram.stt import DeepgramSTTService
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.transports.daily.transport import DailyParams, DailyTransport

transport = DailyTransport(
    room_url,
    token,
    "Respond bot",
    DailyParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
    ),
)

# Configure services
stt = DeepgramSTTService(api_key=KEY)
llm = OpenAILLMService(api_key=KEY, model="gpt-4o")
tts = CartesiaTTSService(api_key=KEY, voice_id=ID)

# Create context and aggregators
context = LLMContext(
    messages=[{"role": "system", "content": "You are a helpful assistant."}]
)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
    ),
)

# Create pipeline
pipeline = Pipeline([
    transport.input(),
    stt,
    user_aggregator,
    llm,
    tts,
    transport.output(),
    assistant_aggregator,
])
```


# Reference docs
Source: https://docs.pipecat.ai/server/links/server-reference





# Pipeline Heartbeats
Source: https://docs.pipecat.ai/server/pipeline/heartbeats

Monitor pipeline health with heartbeat frames

## Overview

Pipeline heartbeats provide a way to monitor the health of your pipeline by sending periodic heartbeat frames through the system. When enabled, the pipeline will send heartbeat frames every second and monitor their progress through the pipeline.

## Enabling Heartbeats

Heartbeats can be enabled by setting `enable_heartbeats` to `True` in the `PipelineParams`:

```python theme={null}
from pipecat.pipeline.task import PipelineParams, PipelineTask

pipeline = Pipeline([...])
params = params=PipelineParams(enable_heartbeats=True)
task = PipelineTask(pipeline, params)
```

## How It Works

When heartbeats are enabled:

1. The pipeline sends a `HeartbeatFrame` every second
2. The frame traverses through all processors in the pipeline, from source to sink
3. The pipeline monitors how long it takes for heartbeat frames to complete their journey
4. If a heartbeat frame isn't received within 5 seconds, a warning is logged

## Monitoring Output

The system will log:

* Trace-level logs showing heartbeat processing time
* Warning messages if heartbeats aren't received within the monitoring window

Example warning message:

```
WARNING    PipelineTask#1: heartbeat frame not received for more than 5.0 seconds
```

## Use Cases

Heartbeat monitoring is useful for:

* Detecting pipeline stalls or blockages
* Monitoring processing latency through the pipeline
* Identifying performance issues in specific processors
* Ensuring the pipeline remains responsive

## Configuration

The heartbeat system uses two key timing constants:

* `HEARTBEAT_SECONDS = 1.0` - Interval between heartbeat frames
* `HEARTBEAT_MONITOR_SECONDS = 10.0` - Time before warning if no heartbeat received

<Note>
  These values are currently fixed but may be configurable in future versions.
</Note>


# ParallelPipeline
Source: https://docs.pipecat.ai/server/pipeline/parallel-pipeline

Run multiple pipeline branches in parallel, with synchronized inputs and outputs for complex flows

## Overview

`ParallelPipeline` allows you to create multiple independent processing branches that run simultaneously, sharing input and coordinating output. It's particularly useful for multi-agent systems, parallel stream processing, and creating redundant service paths.

Each branch receives the same downstream frames, processes them independently, and the results are merged back into a single stream. System frames (like `StartFrame` and `EndFrame`) are synchronized across all branches.

## Constructor Parameters

<ParamField type="List[List[FrameProcessor]]">
  Multiple lists of processors, where each list defines a parallel branch. All
  branches execute simultaneously when frames flow through the pipeline.
</ParamField>

## Usage Examples

### Multi-Agent Conversation

Create a conversation with two AI agents that can interact with the user independently:

```python theme={null}
pipeline = Pipeline([
    transport.input(),
    ParallelPipeline(
        # Agent 1: Customer service representative
        [
            stt_1,
            context_aggregator.user_a(),
            llm_agent_1,
            tts_agent_1,
        ],
        # Agent 2: Technical specialist
        [   stt_2,
            context_aggregator.user_b(),
            llm_agent_2,
            tts_agent_2,
        ]
    ),
    transport.output(),
])
```

### Redundant Services with Failover

Set up redundant services with automatic failover:

```python theme={null}
pipeline = Pipeline([
    transport.input(),
    stt,
    ParallelPipeline(
        # Primary LLM service
        [   gate_primary,
            primary_llm,
            error_detector,
        ],
        # Backup LLM service (used only if primary fails)
        [   gate_backup,
            backup_llm,
            fallback_processor,
        ]
    ),
    tts,
    transport.output(),
])
```

### Cross-Branch Communication

Using Producer/Consumer processors to share data between branches:

```python theme={null}
# Create producer/consumer pair for cross-branch communication
frame_producer = ProducerProcessor(filter=is_important_frame)
frame_consumer = ConsumerProcessor(producer=frame_producer)

pipeline = Pipeline([
    transport.input(),
    ParallelPipeline(
        # Branch that generates important frames
        [
            stt,
            llm,
            tts,
            frame_producer,  # Share frames with other branch
        ],
        # Branch that consumes those frames
        [
            frame_consumer,  # Receive frames from other branch
            llm,             # Speech to Speech LLM (audio in)
        ]
    ),
    transport.output(),
])
```

## How It Works

1. `ParallelPipeline` adds special source and sink processors to each branch
2. System frames (like `StartFrame` and `EndFrame`) are sent to all branches
3. Other frames flow downstream to all branch sources
4. Results from each branch are collected at the sinks
5. The pipeline ensures `EndFrame`s are only passed through after all branches complete


# Pipeline Idle Detection
Source: https://docs.pipecat.ai/server/pipeline/pipeline-idle-detection

Automatically detect and handle idle pipelines with no bot activity

## Overview

Pipeline idle detection is a feature that monitors activity in your pipeline and can automatically cancel tasks when no meaningful bot interactions are occurring. This helps prevent pipelines from running indefinitely when a conversation has naturally ended but wasn't properly terminated.

## How It Works

The system monitors specific "activity frames" that indicate the bot is actively engaged in the conversation. By default, these are:

* `BotSpeakingFrame` - When the bot is speaking
* `LLMFullResponseEndFrame` - When the LLM has completed a response

If no activity frames are detected within the configured timeout period (5 minutes by default), the system considers the pipeline idle and can automatically terminate it.

<Note>
  Idle detection only starts after the pipeline has begun processing frames. The
  idle timer resets whenever an activity frame (as specified in
  `idle_timeout_frames`) is received.
</Note>

## Configuration

You can configure idle detection behavior when creating a `PipelineTask`:

```python theme={null}
from pipecat.pipeline.task import PipelineParams, PipelineTask

# Default configuration - cancel after 5 minutes of inactivity
task = PipelineTask(pipeline)

# Custom configuration
task = PipelineTask(
    pipeline,
    idle_timeout_secs=600,  # 10 minute timeout
    idle_timeout_frames=(BotSpeakingFrame,),  # Only monitor bot speaking
    cancel_on_idle_timeout=False,  # Don't auto-cancel, just notify
)
```

## Configuration Parameters

<ParamField type="Optional[float]">
  Timeout in seconds before considering the pipeline idle. Set to `None` to
  disable idle detection.
</ParamField>

<ParamField type="Tuple[Type[Frame], ...]">
  Frame types that should prevent the pipeline from being considered idle.
</ParamField>

<ParamField type="bool">
  Whether to automatically cancel the pipeline task when idle timeout is
  reached.
</ParamField>

## Handling Idle Timeouts

You can respond to idle timeout events by adding an event handler:

```python theme={null}
@task.event_handler("on_idle_timeout")
async def on_idle_timeout(task):
    logger.info("Pipeline has been idle for too long")
    # Perform any custom cleanup or logging
    # Note: If cancel_on_idle_timeout=True, the pipeline will be cancelled after this handler runs
```

## Example Implementation

Here's a complete example showing how to configure idle detection with custom handling:

```python theme={null}
from pipecat.frames.frames import BotSpeakingFrame, LLMFullResponseEndFrame, TTSSpeakFrame
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask

# Create pipeline
pipeline = Pipeline([...])

# Configure task with custom idle settings
task = PipelineTask(
    pipeline,
    idle_timeout_secs=180,  # 3 minutes
    cancel_on_idle_timeout=False  # Don't auto-cancel
)

# Add event handler for idle timeout
@task.event_handler("on_idle_timeout")
async def on_idle_timeout(task):
    logger.info("Conversation has been idle for 3 minutes")

    # Add a farewell message
    await task.queue_frame(TTSSpeakFrame("I haven't heard from you in a while. Goodbye!"))

    # Then end the conversation gracefully
    await task.stop_when_done()

runner = PipelineRunner()

await runner.run(task)
```


# PipelineParams
Source: https://docs.pipecat.ai/server/pipeline/pipeline-params

Configure pipeline execution with PipelineParams

## Overview

The `PipelineParams` class provides a structured way to configure various aspects of pipeline execution. These parameters control behaviors like audio settings, metrics collection, heartbeat monitoring, and interruption handling.

## Basic Usage

```python theme={null}
from pipecat.pipeline.task import PipelineParams, PipelineTask

# Create with default parameters
params = PipelineParams()

# Or customize specific parameters
params = PipelineParams(
    audio_in_sample_rate=16000,
    enable_metrics=True
)

# Pass to PipelineTask
pipeline = Pipeline([...])
task = PipelineTask(pipeline, params=params)
```

## Available Parameters

<ParamField type="bool">
  <Warning>
    DEPRECATED: This parameter is deprecated. Configure interruption behavior
    via [User Turn
    Strategies](/server/utilities/turn-management/user-turn-strategies) instead.
    See the `enable_interruptions` parameter on start strategies.
  </Warning>

  Whether to allow pipeline interruptions. When enabled, a user's speech will immediately
  interrupt the bot's response.
</ParamField>

<ParamField type="int">
  Input audio sample rate in Hz.

  <Info>
    Setting the `audio_in_sample_rate` as a `PipelineParam` sets the input sample
    rate for all corresponding services in the pipeline.
  </Info>
</ParamField>

<ParamField type="int">
  Output audio sample rate in Hz.

  <Info>
    Setting the `audio_out_sample_rate` as a `PipelineParam` sets the output
    sample rate for all corresponding services in the pipeline.
  </Info>
</ParamField>

<ParamField type="bool">
  Whether to enable heartbeat monitoring to detect pipeline stalls. See
  [Heartbeats](/server/pipeline/heartbeats) for details.
</ParamField>

<ParamField type="float">
  Period between heartbeats in seconds (when heartbeats are enabled).
</ParamField>

<ParamField type="bool">
  Whether to enable metrics collection for pipeline performance.
</ParamField>

<ParamField type="bool">
  Whether to enable usage metrics tracking.
</ParamField>

<ParamField type="bool">
  Whether to report only initial time to first byte metric.
</ParamField>

<ParamField type="bool">
  Whether to send initial empty metrics frame at pipeline start.
</ParamField>

<ParamField type="Dict[str, Any]">
  Additional metadata to include in the StartFrame.
</ParamField>

## Common Configurations

### Audio Processing Configuration

You can set the audio input and output sample rates in the `PipelineParams` to set the sample rate for all input and output services in the pipeline. This acts as a convenience to avoid setting the sample rate for each service individually. Note, if services are set individually, they will supersede the values set in `PipelineParams`.

```python theme={null}
params = PipelineParams(
    audio_in_sample_rate=8000,   # Lower quality input audio
    audio_out_sample_rate=8000  # High quality output audio
)
```

### Performance Monitoring Configuration

Pipeline heartbeats provide a way to monitor the health of your pipeline by sending periodic heartbeat frames through the system. When enabled, the pipeline will send heartbeat frames every second and monitor their progress through the pipeline.

```python theme={null}
params = PipelineParams(
    enable_heartbeats=True,
    heartbeats_period_secs=2.0,  # Send heartbeats every 2 seconds
    enable_metrics=True
)
```

## How Parameters Are Used

The parameters you set in `PipelineParams` are passed to various components of the pipeline:

1. **StartFrame**: Many parameters are included in the StartFrame that initializes the pipeline
2. **Metrics Collection**: Metrics settings configure what performance data is gathered
3. **Heartbeat Monitoring**: Controls the pipeline's health monitoring system
4. **Audio Processing**: Sample rates affect how audio is processed throughout the pipeline

## Complete Example

```python theme={null}
from pipecat.frames.frames import TTSSpeakFrame
from pipecat.observers.file_observer import FileObserver
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.pipeline.runner import PipelineRunner

# Create comprehensive parameters
params = PipelineParams(
    audio_in_sample_rate=8000,
    audio_out_sample_rate=8000,
    enable_heartbeats=True,
    enable_metrics=True,
    enable_usage_metrics=True,
    heartbeats_period_secs=1.0,
    report_only_initial_ttfb=False,
    start_metadata={
        "conversation_id": "conv-123",
        "session_data": {
            "user_id": "user-456",
            "start_time": "2023-10-25T14:30:00Z"
        }
    }
)

# Create pipeline and task
pipeline = Pipeline([...])
task = PipelineTask(
    pipeline,
    params=params,
    observers=[FileObserver("pipeline_logs.jsonl")]
)

# Run the pipeline
runner = PipelineRunner()
await runner.run(task)
```

## Additional Information

* Parameters are immutable once the pipeline starts
* The `start_metadata` dictionary can contain any serializable data
* For metrics collection to work properly, `enable_metrics` must be set to `True`


# PipelineTask
Source: https://docs.pipecat.ai/server/pipeline/pipeline-task

Manage pipeline execution and lifecycle with PipelineTask

## Overview

`PipelineTask` is the central class for managing pipeline execution. It handles the lifecycle of the pipeline, processes frames in both directions, manages task cancellation, and provides event handlers for monitoring pipeline activity.

## Basic Usage

```python theme={null}
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask

# Create a pipeline
pipeline = Pipeline([...])

# Create a task with the pipeline
task = PipelineTask(pipeline)

# Queue frames for processing
await task.queue_frame(TTSSpeakFrame("Hello, how can I help you today?"))

# Run the pipeline
runner = PipelineRunner()
await runner.run(task)
```

## Constructor Parameters

<ParamField type="BasePipeline">
  The pipeline to execute.
</ParamField>

<ParamField type="PipelineParams">
  Configuration parameters for the pipeline. See
  [PipelineParams](/server/pipeline/pipeline-params) for details.
</ParamField>

<ParamField type="List[BaseObserver]">
  List of observers for monitoring pipeline execution. See
  [Observers](/server/utilities/observers/observer-pattern) for details.
</ParamField>

<ParamField type="BaseClock">
  Clock implementation for timing operations.
</ParamField>

<ParamField type="Optional[BaseTaskManager]">
  Custom task manager for handling asyncio tasks. If None, a default TaskManager
  is used.
</ParamField>

<ParamField type="bool">
  Whether to check for processors' tasks finishing properly.
</ParamField>

<ParamField type="Optional[float]">
  Timeout in seconds before considering the pipeline idle. Set to None to
  disable idle detection. See [Pipeline Idle
  Detection](/server/pipeline/pipeline-idle-detection) for details.
</ParamField>

<ParamField type="Tuple[Type[Frame], ...]">
  Frame types that should prevent the pipeline from being considered idle. See
  [Pipeline Idle Detection](/server/pipeline/pipeline-idle-detection) for
  details.
</ParamField>

<ParamField type="bool">
  Whether to automatically cancel the pipeline task when idle timeout is
  reached. See [Pipeline Idle
  Detection](/server/pipeline/pipeline-idle-detection) for details.
</ParamField>

<ParamField type="bool">
  Whether to enable OpenTelemetry tracing. See [The OpenTelemetry
  guide](/server/utilities/opentelemetry) for details.
</ParamField>

<ParamField type="bool">
  Whether to enable turn tracking. See [The OpenTelemetry
  guide](/server/utilities/opentelemetry) for details.
</ParamField>

<ParamField type="Optional[str]">
  Custom ID for the conversation. If not provided, a UUID will be generated. See
  [The OpenTelemetry guide](/server/utilities/opentelemetry) for details.
</ParamField>

<ParamField type="Optional[dict]">
  Any additional attributes to add to top-level OpenTelemetry conversation span.
  See [The OpenTelemetry guide](/server/utilities/opentelemetry) for details.
</ParamField>

## Methods

### Task Lifecycle Management

<ResponseField name="run()" type="async">
  Starts and manages the pipeline execution until completion or cancellation.

  ```python theme={null}
  await task.run()
  ```
</ResponseField>

<ResponseField name="stop_when_done()" type="async">
  Sends an EndFrame to the pipeline to gracefully stop the task after all queued
  frames have been processed.

  ```python theme={null}
  await task.stop_when_done()
  ```
</ResponseField>

<ResponseField name="cancel()" type="async">
  Stops the running pipeline immediately by sending a CancelFrame.

  ```python theme={null}
    await task.cancel()
  ```
</ResponseField>

<ResponseField name="has_finished()" type="bool">
  Returns whether the task has finished (all processors have stopped).

  ```python theme={null}
  if task.has_finished(): print("Task is complete")
  ```
</ResponseField>

### Frame Management

<ResponseField name="queue_frame()" type="async">
  Queues a single frame to be pushed down the pipeline.

  ```python theme={null}
  await task.queue_frame(TTSSpeakFrame("Hello!"))
  ```
</ResponseField>

<ResponseField name="queue_frames()" type="async">
  Queues multiple frames to be pushed down the pipeline.

  ```python theme={null}
  frames = [TTSSpeakFrame("Hello!"), TTSSpeakFrame("How are you?")]

  await task.queue_frames(frames)

  ```
</ResponseField>

## Event Handlers

PipelineTask provides an event handler that can be registered using the `event_handler` decorator:

### on\_idle\_timeout

Triggered when no activity frames (as specified by `idle_timeout_frames`) have been received within the idle timeout period.

```python theme={null}
@task.event_handler("on_idle_timeout")
async def on_idle_timeout(task):
    print("Pipeline has been idle too long")
    await task.queue_frame(TTSSpeakFrame("Are you still there?"))
```


# Sentry Metrics
Source: https://docs.pipecat.ai/server/services/analytics/sentry

Performance monitoring integration with Sentry for Pipecat frame processors

## Overview

`SentryMetrics` extends `FrameProcessorMetrics` to provide performance monitoring integration with Sentry. It tracks Time to First Byte (TTFB) and processing duration metrics for frame processors, enabling real-time performance monitoring and error tracking for your Pipecat applications.

<CardGroup>
  <Card title="Sentry Metrics API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.processors.metrics.sentry.html">
    Pipecat's API methods for Sentry metrics integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/47-sentry-metrics.py">
    Browse examples using Sentry metrics
  </Card>

  <Card title="Sentry Documentation" icon="book" href="https://docs.sentry.io/platforms/python/">
    Official Sentry Python SDK documentation
  </Card>

  <Card title="Sentry Platform" icon="chart-line" href="https://sentry.io/">
    Access performance monitoring and error tracking
  </Card>
</CardGroup>

## Installation

To use Sentry analytics services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[sentry]"
```

## Prerequisites

### Sentry Account Setup

Before using Sentry metrics services, you need:

1. **Sentry Account**: Sign up at [Sentry Platform](https://sentry.io/)
2. **Project Setup**: Create a project and obtain your DSN
3. **SDK Initialization**: Configure Sentry SDK in your application
4. **Metrics Configuration**: Set up performance monitoring and error tracking

### Required Configuration

* **Sentry DSN**: Your project's Data Source Name for authentication
* **Traces Sample Rate**: Configure performance monitoring sampling
* **SDK Initialization**: Initialize Sentry before using metrics

### Key Features

* **Performance Monitoring**: Track TTFB and processing duration metrics
* **Error Tracking**: Automatic error capture and reporting
* **Frame Processor Metrics**: Monitor individual processor performance
* **Real-time Analytics**: Live performance data and alerting


# Community Integrations
Source: https://docs.pipecat.ai/server/services/community-integrations

Community-maintained service integrations for Pipecat

Community Integrations are service integrations built and maintained by developers in the Pipecat community. These are not officially supported by the Pipecat team but are listed here to help you discover what's available.

Want to add your integration? See our [Community Integrations Guide](https://github.com/pipecat-ai/pipecat/blob/main/COMMUNITY_INTEGRATIONS.md).

***

## Knowledge Retrieval

Semantic retrieval services enable context-aware search and retrieval of relevant information.

| Service                          | Repository                                                                         | Maintainer(s)                      |
| -------------------------------- | ---------------------------------------------------------------------------------- | ---------------------------------- |
| [Moss](https://www.usemoss.dev/) | [https://github.com/usemoss/pipecat-moss](https://github.com/usemoss/pipecat-moss) | [Moss](https://github.com/usemoss) |

## Large Language Models

LLMs receive text or audio based input and output a streaming text response.

| Service                          | Repository                                                                                                             | Maintainer(s)                           |
| -------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | --------------------------------------- |
| [Anannas AI](https://anannas.ai) | [https://github.com/Anannas-AI/anannas-pipecat-integration](https://github.com/Anannas-AI/anannas-pipecat-integration) | [Haleshot](https://github.com/Haleshot) |

## Observability

Observability services enable telemetry and metrics data to be passed to a Open Telemetry backend or another service.

| Service                                                    | Repository                                                                                                                                                | Maintainer(s)                                                             |
| ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- |
| [OpenInference](https://arize-ai.github.io/openinference/) | [openinference-instrumentation-pipecat](https://github.com/Arize-ai/openinference/tree/main/python/instrumentation/openinference-instrumentation-pipecat) | [Arize-ai](https://github.com/Arize-ai/openinference/graphs/contributors) |
| [Finchvox](https://finchvox.dev/)                          | [https://github.com/finchvox/finchvox](https://github.com/finchvox/finchvox)                                                                              | [Finchvox](https://github.com/finchvox)                                   |

## Telephony Serializers

Serializers convert between frames and media streams, enabling real-time communication over a websocket.

| Service                          | Repository                                                                               | Maintainer(s)                         |
| -------------------------------- | ---------------------------------------------------------------------------------------- | ------------------------------------- |
| [AwaazAI](https://www.awaaz.ai/) | [https://github.com/awaazde/pipecat-awaazai](https://github.com/awaazde/pipecat-awaazai) | [AwaazAI](https://github.com/awaazde) |

## Text-to-Speech

Text-to-Speech services receive text input and output audio streams or chunks.

| Service                                                          | Repository                                                                                                   | Maintainer(s)                                   |
| ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | ----------------------------------------------- |
| [Murf AI](https://murf.ai/api)                                   | [https://github.com/murf-ai/pipecat-murf-tts](https://github.com/murf-ai/pipecat-murf-tts)                   | [murf-ai](https://github.com/murf-ai)           |
| [Pipecat TTS Cache](https://pypi.org/project/pipecat-tts-cache/) | [https://github.com/omChauhanDev/pipecat-tts-cache](https://github.com/omChauhanDev/pipecat-tts-cache)       | [omChauhanDev](https://github.com/omChauhanDev) |
| [Respeecher](https://www.respeecher.com/real-time-tts-api)       | [https://github.com/respeecher/pipecat-respeecher](https://github.com/respeecher/pipecat-respeecher)         | [respeecher](https://github.com/respeecher)     |
| [Typecast](https://typecast.ai/)                                 | [https://github.com/neosapience/pipecat-typecast](https://github.com/neosapience/pipecat-typecast)           | [neosapience](https://github.com/neosapience)   |
| [Uplift AI](https://upliftai.org)                                | [https://github.com/havkerboi123/pipecat-upliftai-tts](https://github.com/havkerboi123/pipecat-upliftai-tts) | [havkerboi123](https://github.com/havkerboi123) |
| [Voice.ai](https://voice.ai/)                                    | [https://github.com/voice-ai/voice-ai-pipecat-tts](https://github.com/voice-ai/voice-ai-pipecat-tts)         | [voice-ai](https://github.com/voice-ai)         |

## Video

Video services enable you to build an avatar where audio and video are synchronized.

| Service                                           | Repository                                                                       | Maintainer(s)                         |
| ------------------------------------------------- | -------------------------------------------------------------------------------- | ------------------------------------- |
| [Beyond Presence](https://www.beyondpresence.ai/) | [https://github.com/bey-dev/pipecat-bey](https://github.com/bey-dev/pipecat-bey) | [bey-dev](https://github.com/bey-dev) |

## Image Generation

Image generation services receive text inputs and output images.

| Service                         | Repository | Maintainer(s) |
| ------------------------------- | ---------- | ------------- |
| *No community integrations yet* |            |               |

## Speech-to-Text

Speech-to-Text services receive and audio input and output transcriptions.

| Service                         | Repository | Maintainer(s) |
| ------------------------------- | ---------- | ------------- |
| *No community integrations yet* |            |               |

## Vision

Vision services receive a streaming video input and output text describing the video input.

| Service                         | Repository | Maintainer(s) |
| ------------------------------- | ---------- | ------------- |
| *No community integrations yet* |            |               |


# fal
Source: https://docs.pipecat.ai/server/services/image-generation/fal

Image generation service implementation using fal's fast SDXL models

## Overview

`FalImageGenService` provides high-speed image generation capabilities using fal's optimized Stable Diffusion XL models. It supports various image sizes, formats, and generation parameters with a focus on fast inference and low-latency image creation.

<CardGroup>
  <Card title="Fal Image Gen API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.fal.image.html">
    Pipecat's API methods for fal image generation integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/05-sync-speech-and-image.py">
    Browse examples using fal image generation
  </Card>

  <Card title="fal Documentation" icon="book" href="https://fal.ai/docs">
    Official fal API documentation and model guides
  </Card>

  <Card title="fal Platform" icon="image" href="https://fal.ai/">
    Access fast SDXL models and manage API keys
  </Card>
</CardGroup>

## Installation

To use fal image generation services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[fal]"
```

## Prerequisites

### fal Account Setup

Before using fal image generation services, you need:

1. **fal Account**: Sign up at [fal Platform](https://fal.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Selection**: Choose from available fast SDXL models
4. **HTTP Session**: Configure aiohttp session for image downloading

### Required Environment Variables

* `FAL_KEY`: Your fal API key for authentication


# Google Imagen
Source: https://docs.pipecat.ai/server/services/image-generation/google-imagen

Image generation service implementation using Google's Imagen models

## Overview

`GoogleImageGenService` provides high-quality image generation capabilities using Google's Imagen models. It supports generating multiple images from text prompts with various customization options and advanced prompt understanding for photorealistic and artistic image creation.

<CardGroup>
  <Card title="Google Imagen API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.google.image.html">
    Pipecat's API methods for Google Imagen integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/03b-still-frame-imagen.py">
    Browse examples using Google Imagen
  </Card>

  <Card title="Google Imagen Documentation" icon="book" href="https://cloud.google.com/vertex-ai/generative-ai/docs/image/overview">
    Official Google Imagen API documentation and guides
  </Card>

  <Card title="Google Cloud Console" icon="image" href="https://console.cloud.google.com/">
    Access Imagen models and manage API keys
  </Card>
</CardGroup>

## Installation

To use Google Imagen services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[google]"
```

## Prerequisites

### Google Cloud Setup

Before using Google Imagen services, you need:

1. **Google Cloud Account**: Set up at [Google Cloud Console](https://console.cloud.google.com/)
2. **API Key**: Generate a Google API key with Vertex AI access
3. **Project Configuration**: Enable Vertex AI API for your project
4. **Model Access**: Ensure access to Imagen generation models

### Required Environment Variables

* `GOOGLE_API_KEY`: Your Google API key for authentication


# OpenAI Image Generation
Source: https://docs.pipecat.ai/server/services/image-generation/openai

Image generation service implementation using OpenAI's DALL-E models

## Overview

`OpenAIImageGenService` provides high-quality image generation capabilities using OpenAI's DALL-E models. It transforms text prompts into images with various size options and model configurations, offering both artistic and photorealistic image creation capabilities.

<CardGroup>
  <Card title="OpenAI Image Gen API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.openai.image.html">
    Pipecat's API methods for OpenAI image generation integration
  </Card>

  <Card title="OpenAI Documentation" icon="book" href="https://platform.openai.com/docs/guides/images">
    Official OpenAI DALL-E API documentation and guides
  </Card>

  <Card title="OpenAI Platform" icon="image" href="https://platform.openai.com/">
    Access DALL-E models and manage API keys
  </Card>
</CardGroup>

## Installation

To use OpenAI image generation services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[openai]"
```

## Prerequisites

### OpenAI Account Setup

Before using OpenAI image generation services, you need:

1. **OpenAI Account**: Sign up at [OpenAI Platform](https://platform.openai.com/)
2. **API Key**: Generate an OpenAI API key from your account dashboard
3. **Model Access**: Ensure access to DALL-E models
4. **HTTP Session**: Configure aiohttp session for image downloading

### Required Environment Variables

* `OPENAI_API_KEY`: Your OpenAI API key for authentication


# Anthropic
Source: https://docs.pipecat.ai/server/services/llm/anthropic

Large Language Model service implementation using Anthropic's Claude API

## Overview

`AnthropicLLMService` provides integration with Anthropic's Claude models, supporting streaming responses, function calling, and prompt caching with specialized context handling for Anthropic's message format and advanced reasoning capabilities.

<CardGroup>
  <Card title="Anthropic LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.anthropic.llm.html">
    Pipecat's API methods for Anthropic Claude integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14a-function-calling-anthropic.py">
    Complete example with function calling
  </Card>

  <Card title="Anthropic Documentation" icon="book" href="https://docs.anthropic.com/en/api/messages">
    Official Anthropic API documentation and features
  </Card>

  <Card title="Anthropic Console" icon="microphone" href="https://console.anthropic.com/">
    Access Claude models and API keys
  </Card>
</CardGroup>

## Installation

To use Anthropic services, install the required dependency:

```bash theme={null}
pip install "pipecat-ai[anthropic]"
```

## Prerequisites

### Anthropic Account Setup

Before using Anthropic LLM services, you need:

1. **Anthropic Account**: Sign up at [Anthropic Console](https://console.anthropic.com/)
2. **API Key**: Generate an API key from your console dashboard
3. **Model Selection**: Choose from available Claude models (Claude 3.5 Sonnet, Claude 3 Opus, etc.)

### Required Environment Variables

* `ANTHROPIC_API_KEY`: Your Anthropic API key for authentication


# AWS Bedrock
Source: https://docs.pipecat.ai/server/services/llm/aws

Large Language Model service implementation using Amazon Bedrock API

## Overview

`AWSBedrockLLMService` provides access to Amazon's foundation models including Anthropic Claude and Amazon Nova, with streaming responses, function calling, and multimodal capabilities through Amazon's managed AI service for enterprise-grade LLM deployment.

<CardGroup>
  <Card title="AWS Bedrock LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.aws.llm.html">
    Pipecat's API methods for AWS Bedrock integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14r-function-calling-aws.py">
    Complete example with function calling
  </Card>

  <Card title="AWS Bedrock Documentation" icon="book" href="https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html">
    Official AWS Bedrock documentation and features
  </Card>

  <Card title="AWS Console" icon="microphone" href="https://console.aws.amazon.com/bedrock/">
    Access foundation models and manage IAM
  </Card>
</CardGroup>

## Installation

To use AWS Bedrock services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[aws]"
```

## Prerequisites

### AWS Account Setup

Before using AWS Bedrock LLM services, you need:

1. **AWS Account**: Sign up at [AWS Console](https://console.aws.amazon.com/)
2. **IAM User**: Create an IAM user with Amazon Bedrock permissions
3. **Model Access**: Request access to foundation models in your AWS region
4. **Credentials**: Set up AWS access keys and region configuration

### Required Environment Variables

* `AWS_ACCESS_KEY_ID`: Your AWS access key ID
* `AWS_SECRET_ACCESS_KEY`: Your AWS secret access key
* `AWS_SESSION_TOKEN`: Session token (if using temporary credentials)
* `AWS_REGION`: AWS region (defaults to "us-east-1")


# Azure
Source: https://docs.pipecat.ai/server/services/llm/azure

Large Language Model service implementation using Azure OpenAI API

## Overview

`AzureLLMService` provides access to Azure OpenAI's language models through an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports streaming responses, function calling, and context management with enterprise-grade security and compliance.

<CardGroup>
  <Card title="Azure LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.azure.llm.html">
    Pipecat's API methods for Azure OpenAI integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14h-function-calling-azure.py">
    Complete example with function calling
  </Card>

  <Card title="Azure OpenAI Documentation" icon="book" href="https://learn.microsoft.com/en-us/azure/ai-services/openai/">
    Official Azure OpenAI documentation and setup
  </Card>

  <Card title="Azure Portal" icon="microphone" href="https://portal.azure.com/">
    Create OpenAI resources and get credentials
  </Card>
</CardGroup>

## Installation

To use Azure OpenAI services, install the required dependency:

```bash theme={null}
pip install "pipecat-ai[azure]"
```

## Prerequisites

### Azure OpenAI Setup

Before using Azure OpenAI LLM services, you need:

1. **Azure Account**: Sign up at [Azure Portal](https://portal.azure.com/)
2. **OpenAI Resource**: Create an Azure OpenAI resource in your subscription
3. **Model Deployment**: Deploy your chosen model (GPT-4, GPT-3.5-turbo, etc.)
4. **Credentials**: Get your API key, endpoint, and deployment name

### Required Environment Variables

* `AZURE_CHATGPT_API_KEY`: Your Azure OpenAI API key
* `AZURE_CHATGPT_ENDPOINT`: Your Azure OpenAI endpoint URL
* `AZURE_CHATGPT_MODEL`: Your model deployment name


# Cerebras
Source: https://docs.pipecat.ai/server/services/llm/cerebras

LLM service implementation using Cerebras's API with OpenAI-compatible interface

## Overview

`CerebrasLLMService` provides access to Cerebras's language models through an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports streaming responses, function calling, and context management with ultra-fast inference speeds.

<CardGroup>
  <Card title="Cerebras LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.cerebras.llm.html">
    Pipecat's API methods for Cerebras integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14k-function-calling-cerebras.py">
    Complete example with function calling
  </Card>

  <Card title="Cerebras Documentation" icon="book" href="https://inference-docs.cerebras.ai/api-reference/chat-completions">
    Official Cerebras inference API documentation
  </Card>

  <Card title="Cerebras Platform" icon="microphone" href="https://cloud.cerebras.ai/">
    Access models and manage API keys
  </Card>
</CardGroup>

## Installation

To use Cerebras services, install the required dependency:

```bash theme={null}
pip install "pipecat-ai[cerebras]"
```

## Prerequisites

### Cerebras Account Setup

Before using Cerebras LLM services, you need:

1. **Cerebras Account**: Sign up at [Cerebras Cloud](https://cloud.cerebras.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Selection**: Choose from available Cerebras models with ultra-fast inference

### Required Environment Variables

* `CEREBRAS_API_KEY`: Your Cerebras API key for authentication


# DeepSeek
Source: https://docs.pipecat.ai/server/services/llm/deepseek

LLM service implementation using DeepSeek's API with OpenAI-compatible interface

## Overview

`DeepSeekLLMService` provides access to DeepSeek's language models through an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports streaming responses, function calling, and context management with advanced reasoning capabilities.

<CardGroup>
  <Card title="DeepSeek LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.deepseek.llm.html">
    Pipecat's API methods for DeepSeek integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14l-function-calling-deepseek.py">
    Complete example with function calling
  </Card>

  <Card title="DeepSeek Documentation" icon="book" href="https://api-docs.deepseek.com/api/create-chat-completion">
    Official DeepSeek API documentation and features
  </Card>

  <Card title="DeepSeek Platform" icon="microphone" href="https://platform.deepseek.com/">
    Access models and manage API keys
  </Card>
</CardGroup>

## Installation

To use DeepSeek services, install the required dependency:

```bash theme={null}
pip install "pipecat-ai[deepseek]"
```

## Prerequisites

### DeepSeek Account Setup

Before using DeepSeek LLM services, you need:

1. **DeepSeek Account**: Sign up at [DeepSeek Platform](https://platform.deepseek.com/)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Selection**: Choose from available DeepSeek models with reasoning capabilities

### Required Environment Variables

* `DEEPSEEK_API_KEY`: Your DeepSeek API key for authentication


# Fireworks AI
Source: https://docs.pipecat.ai/server/services/llm/fireworks

LLM service implementation using Fireworks AI's API with OpenAI-compatible interface

## Overview

`FireworksLLMService` provides access to Fireworks AI's language models through an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports streaming responses, function calling, and context management with optimized inference infrastructure.

<CardGroup>
  <Card title="Fireworks LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.fireworks.llm.html">
    Pipecat's API methods for Fireworks AI integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14i-function-calling-fireworks.py">
    Complete example with function calling
  </Card>

  <Card title="Fireworks Documentation" icon="book" href="https://docs.fireworks.ai/api-reference/post-chatcompletions">
    Official Fireworks AI API documentation and features
  </Card>

  <Card title="Fireworks Platform" icon="microphone" href="https://fireworks.ai/">
    Access models and manage API keys
  </Card>
</CardGroup>

## Installation

To use Fireworks AI services, install the required dependency:

```bash theme={null}
pip install "pipecat-ai[fireworks]"
```

## Prerequisites

### Fireworks AI Account Setup

Before using Fireworks AI LLM services, you need:

1. **Fireworks Account**: Sign up at [Fireworks AI](https://fireworks.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Selection**: Choose from available open-source and proprietary models

### Required Environment Variables

* `FIREWORKS_API_KEY`: Your Fireworks AI API key for authentication


# Google Gemini
Source: https://docs.pipecat.ai/server/services/llm/gemini

Large Language Model service implementation using Google's Gemini API

## Overview

`GoogleLLMService` provides integration with Google's Gemini models, supporting streaming responses, function calling, and multimodal inputs. It includes specialized context handling for Google's message format while maintaining compatibility with OpenAI-style contexts.

<CardGroup>
  <Card title="Gemini LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.google.llm.html">
    Pipecat's API methods for Google Gemini integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14e-function-calling-google.py">
    Complete example with function calling
  </Card>

  <Card title="Gemini Documentation" icon="book" href="https://ai.google.dev/gemini-api/docs">
    Official Google Gemini API documentation and features
  </Card>

  <Card title="Google AI Studio" icon="microphone" href="https://aistudio.google.com/">
    Access Gemini models and manage API keys
  </Card>
</CardGroup>

## Installation

To use Google Gemini services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[google]"
```

## Prerequisites

### Google Gemini Setup

Before using Google Gemini LLM services, you need:

1. **Google Account**: Sign up at [Google AI Studio](https://aistudio.google.com/)
2. **API Key**: Generate a Gemini API key from AI Studio
3. **Model Selection**: Choose from available Gemini models (Gemini 1.5 Pro, Flash, etc.)

### Required Environment Variables

* `GOOGLE_API_KEY`: Your Google Gemini API key for authentication


# Google Vertex AI
Source: https://docs.pipecat.ai/server/services/llm/google-vertex

LLM service implementation using Google's Vertex AI with OpenAI-compatible interface

## Overview

`GoogleVertexLLMService` provides access to Google's language models through Vertex AI while maintaining an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports all OpenAI features while connecting to Google's enterprise AI services with enhanced security and compliance.

<CardGroup>
  <Card title="Vertex AI LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.google.llm_vertex.html">
    Pipecat's API methods for Google Vertex AI integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14p-function-calling-gemini-vertex-ai.py">
    Browse examples using Vertex AI models
  </Card>

  <Card title="Vertex AI Documentation" icon="book" href="https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/inference">
    Official Google Vertex AI documentation
  </Card>

  <Card title="Google Cloud Console" icon="microphone" href="https://console.cloud.google.com/vertex-ai">
    Access Vertex AI and manage credentials
  </Card>
</CardGroup>

## Installation

To use Google Vertex AI services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[google]"
```

## Prerequisites

### Google Cloud Setup

Before using Google Vertex AI LLM services, you need:

1. **Google Cloud Account**: Sign up at [Google Cloud Console](https://console.cloud.google.com/)
2. **Project Setup**: Create a project and enable the Vertex AI API
3. **Service Account**: Create a service account with Vertex AI permissions
4. **Authentication**: Set up credentials via service account key or Application Default Credentials

### Required Environment Variables

* `GOOGLE_APPLICATION_CREDENTIALS`: Path to your service account key file (recommended)
* Or use Application Default Credentials for cloud deployments


# Grok
Source: https://docs.pipecat.ai/server/services/llm/grok

LLM service implementation using Grok's API with OpenAI-compatible interface

## Overview

`GrokLLMService` provides access to Grok's language models through an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports streaming responses, function calling, and context management with Grok's unique reasoning capabilities.

<CardGroup>
  <Card title="Grok LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.grok.llm.html">
    Pipecat's API methods for Grok integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14g-function-calling-grok.py">
    Complete example with function calling
  </Card>

  <Card title="Grok Documentation" icon="book" href="https://docs.x.ai/docs/api-reference#chat-completions">
    Official Grok API documentation and features
  </Card>

  <Card title="X.AI Platform" icon="microphone" href="https://console.x.ai/">
    Access Grok models and manage API keys
  </Card>
</CardGroup>

## Installation

To use Grok services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[grok]"
```

## Prerequisites

### Grok Account Setup

Before using Grok LLM services, you need:

1. **X.AI Account**: Sign up at [X.AI Console](https://console.x.ai/)
2. **API Key**: Generate an API key from your console dashboard
3. **Model Selection**: Choose from available Grok models

### Required Environment Variables

* `XAI_API_KEY`: Your X.AI API key for authentication


# Groq
Source: https://docs.pipecat.ai/server/services/llm/groq

LLM service implementation using Groq's API with OpenAI-compatible interface

## Overview

`GroqLLMService` provides access to Groq's language models through an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports streaming responses, function calling, and context management with ultra-fast inference speeds.

<CardGroup>
  <Card title="Groq LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.groq.llm.html">
    Pipecat's API methods for Groq integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14f-function-calling-groq.py">
    Complete example with function calling
  </Card>

  <Card title="Groq Documentation" icon="book" href="https://console.groq.com/docs/api-reference#chat-create">
    Official Groq API documentation and features
  </Card>

  <Card title="Groq Console" icon="microphone" href="https://console.groq.com/">
    Access models and manage API keys
  </Card>
</CardGroup>

## Installation

To use Groq services, install the required dependency:

```bash theme={null}
pip install "pipecat-ai[groq]"
```

## Prerequisites

### Groq Account Setup

Before using Groq LLM services, you need:

1. **Groq Account**: Sign up at [Groq Console](https://console.groq.com/)
2. **API Key**: Generate an API key from your console dashboard
3. **Model Selection**: Choose from available models with ultra-fast inference

### Required Environment Variables

* `GROQ_API_KEY`: Your Groq API key for authentication


# NVIDIA NIM
Source: https://docs.pipecat.ai/server/services/llm/nim

LLM service implementation using NVIDIA's NIM (NVIDIA Inference Microservice) API with OpenAI-compatible interface

## Overview

`NvidiaLLMService` provides access to NVIDIA's NIM language models through an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports streaming responses, function calling, and context management, with special handling for NVIDIA's incremental token reporting and enterprise deployment.

<CardGroup>
  <Card title="NVIDIA NIM LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.nim.llm.html">
    Pipecat's API methods for NVIDIA NIM integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14j-function-calling-nim.py">
    Complete example with function calling
  </Card>

  <Card title="NVIDIA NIM Documentation" icon="book" href="https://docs.nvidia.com/nim/">
    Official NVIDIA NIM documentation and setup
  </Card>

  <Card title="NVIDIA Developer Portal" icon="microphone" href="https://developer.nvidia.com/">
    Access NIM services and manage API keys
  </Card>
</CardGroup>

## Installation

To use NVIDIA NIM services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[nvidia]"
```

## Prerequisites

### NVIDIA NIM Setup

Before using NVIDIA NIM LLM services, you need:

1. **NVIDIA Developer Account**: Sign up at [NVIDIA Developer Portal](https://developer.nvidia.com/)
2. **API Key**: Generate an NVIDIA API key for NIM services
3. **Model Selection**: Choose from available NIM-hosted models
4. **Enterprise Setup**: Configure NIM for on-premises deployment if needed

### Required Environment Variables

* `NVIDIA_API_KEY`: Your NVIDIA API key for authentication


# Ollama
Source: https://docs.pipecat.ai/server/services/llm/ollama

LLM service implementation using Ollama with OpenAI-compatible interface

## Overview

`OLLamaLLMService` provides access to locally-run Ollama models through an OpenAI-compatible interface. It inherits from `BaseOpenAILLMService` and allows you to run various open-source models locally while maintaining compatibility with OpenAI's API format for privacy and cost control.

<CardGroup>
  <Card title="Ollama LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.ollama.llm.html">
    Pipecat's API methods for Ollama integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/tree/main/examples">
    Browse examples using Ollama models
  </Card>

  <Card title="Ollama Documentation" icon="book" href="https://ollama.com/library">
    Official Ollama documentation and model library
  </Card>

  <Card title="Download Ollama" icon="microphone" href="https://ollama.com/download">
    Download and setup instructions for Ollama
  </Card>
</CardGroup>

## Installation

To use Ollama services, you need to install both Ollama and the Pipecat dependency:

1. **Install Ollama** on your system from [ollama.com/download](https://ollama.com/download)
2. **Install Pipecat dependency**:

```bash theme={null}
pip install "pipecat-ai[ollama]"
```

3. **Pull a model** (first time only):

```bash theme={null}
ollama pull llama2
```

## Prerequisites

### Ollama Local Setup

Before using Ollama LLM services, you need:

1. **Ollama Installation**: Download and install Ollama from [ollama.com](https://ollama.com/download)
2. **Model Selection**: Pull your desired models (llama2, mistral, codellama, etc.)
3. **Local Service**: Ensure Ollama service is running (default port 11434)
4. **Hardware**: Sufficient RAM and storage for your chosen models

### Configuration

* **No API Keys Required**: Ollama runs entirely locally
* **Model Management**: Use `ollama pull <model>` to download models
* **Service URL**: Default is `http://localhost:11434` (configurable)

<Tip>
  Ollama runs as a local service on port 11434. No API key required for complete
  privacy!
</Tip>


# OpenAI
Source: https://docs.pipecat.ai/server/services/llm/openai

Large Language Model services using OpenAI's chat completion API

## Overview

`OpenAILLMService` provides chat completion capabilities using OpenAI's API, supporting streaming responses, function calling, vision input, and advanced context management for conversational AI applications with state-of-the-art language models.

<CardGroup>
  <Card title="OpenAI LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.openai.base_llm.html">
    Pipecat's API methods for OpenAI integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14-function-calling.py">
    Function calling example with weather API
  </Card>

  <Card title="OpenAI Documentation" icon="book" href="https://platform.openai.com/docs/api-reference/chat">
    Official OpenAI API documentation
  </Card>

  <Card title="OpenAI Platform" icon="microphone" href="https://platform.openai.com/api-keys">
    Access models and manage API keys
  </Card>
</CardGroup>

## Installation

To use OpenAI services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[openai]"
```

## Prerequisites

### OpenAI Account Setup

Before using OpenAI LLM services, you need:

1. **OpenAI Account**: Sign up at [OpenAI Platform](https://platform.openai.com/)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Selection**: Choose from available models (GPT-4o, GPT-4, GPT-3.5-turbo, etc.)
4. **Usage Limits**: Set up billing and usage limits as needed

### Required Environment Variables

* `OPENAI_API_KEY`: Your OpenAI API key for authentication


# OpenPipe
Source: https://docs.pipecat.ai/server/services/llm/openpipe

LLM service implementation using OpenPipe for LLM request logging and fine-tuning

## Overview

`OpenPipeLLMService` extends the BaseOpenAILLMService to provide integration with OpenPipe, enabling request logging, model fine-tuning, and performance monitoring. It maintains compatibility with OpenAI's API while adding OpenPipe's logging and optimization capabilities.

<CardGroup>
  <Card title="OpenPipe LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.openpipe.llm.html">
    Pipecat's API methods for OpenPipe integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/tree/main/examples">
    Browse examples using OpenPipe logging
  </Card>

  <Card title="OpenPipe Documentation" icon="book" href="https://docs.openpipe.ai/api-reference/post-chatcompletions">
    Official OpenPipe API documentation and features
  </Card>

  <Card title="OpenPipe Platform" icon="microphone" href="https://openpipe.ai/">
    Access logging and fine-tuning features
  </Card>
</CardGroup>

## Installation

To use OpenPipe services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[openpipe]"
```

## Prerequisites

### OpenPipe Account Setup

Before using OpenPipe LLM services, you need:

1. **OpenPipe Account**: Sign up at [OpenPipe](https://openpipe.ai/)
2. **API Keys**: Generate both OpenPipe and OpenAI API keys
3. **Project Setup**: Configure logging and fine-tuning projects

### Required Environment Variables

* `OPENPIPE_API_KEY`: Your OpenPipe API key for logging and fine-tuning
* `OPENAI_API_KEY`: Your OpenAI API key for underlying model access


# OpenRouter
Source: https://docs.pipecat.ai/server/services/llm/openrouter

LLM service implementation using OpenRouter's API with OpenAI-compatible interface

## Overview

`OpenRouterLLMService` provides access to OpenRouter's language models through an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports streaming responses, function calling, and context management with access to multiple model providers through a single API.

<CardGroup>
  <Card title="OpenRouter LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.openrouter.llm.html">
    Pipecat's API methods for OpenRouter integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14m-function-calling-openrouter.py">
    Complete example with function calling
  </Card>

  <Card title="OpenRouter Documentation" icon="book" href="https://openrouter.ai/docs/api-reference/chat-completion">
    Official OpenRouter API documentation and features
  </Card>

  <Card title="OpenRouter Platform" icon="microphone" href="https://openrouter.ai/">
    Access multiple model providers and manage API keys
  </Card>
</CardGroup>

## Installation

To use OpenRouter services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[openrouter]"
```

## Prerequisites

### OpenRouter Account Setup

Before using OpenRouter LLM services, you need:

1. **OpenRouter Account**: Sign up at [OpenRouter](https://openrouter.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Selection**: Choose from hundreds of available models from different providers
4. **Credits**: Add credits to your account for model usage

### Required Environment Variables

* `OPENROUTER_API_KEY`: Your OpenRouter API key for authentication


# Perplexity
Source: https://docs.pipecat.ai/server/services/llm/perplexity

LLM service implementation using Perplexity's API with OpenAI-compatible interface

## Overview

`PerplexityLLMService` provides access to Perplexity's language models through an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports streaming responses and context management, with special handling for Perplexity's incremental token reporting and built-in internet search capabilities.

<CardGroup>
  <Card title="Perplexity LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.perplexity.llm.html">
    Pipecat's API methods for Perplexity integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14n-function-calling-perplexity.py">
    Complete example with search capabilities
  </Card>

  <Card title="Perplexity Documentation" icon="book" href="https://docs.perplexity.ai/api-reference/chat-completions-post">
    Official Perplexity API documentation and features
  </Card>

  <Card title="Perplexity Platform" icon="microphone" href="https://www.perplexity.ai/">
    Access search-enhanced models and API keys
  </Card>
</CardGroup>

## Installation

To use Perplexity services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[perplexity]"
```

## Prerequisites

### Perplexity Account Setup

Before using Perplexity LLM services, you need:

1. **Perplexity Account**: Sign up at [Perplexity](https://www.perplexity.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Selection**: Choose from available models with built-in search capabilities

### Required Environment Variables

* `PERPLEXITY_API_KEY`: Your Perplexity API key for authentication

<Note>
  Unlike other LLM services, Perplexity does not support function calling.
  Instead, they offer native internet search built in without requiring special
  function calls.
</Note>


# Qwen
Source: https://docs.pipecat.ai/server/services/llm/qwen

LLM service implementation using Alibaba Cloud's Qwen models through an OpenAI-compatible interface

## Overview

`QwenLLMService` provides access to Alibaba Cloud's Qwen language models through an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports streaming responses, function calling, and context management, with particularly strong capabilities for Chinese language processing.

<CardGroup>
  <Card title="Qwen LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.qwen.llm.html">
    Pipecat's API methods for Qwen integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14q-function-calling-qwen.py">
    Complete example with function calling
  </Card>

  <Card title="Qwen Documentation" icon="book" href="https://www.alibabacloud.com/help/en/model-studio/use-qwen-by-calling-api">
    Official Qwen API documentation and features
  </Card>

  <Card title="Alibaba Cloud Console" icon="microphone" href="https://www.alibabacloud.com/">
    Access Qwen models and manage API keys
  </Card>
</CardGroup>

## Installation

To use Qwen services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[qwen]"
```

## Prerequisites

### Qwen Account Setup

Before using Qwen LLM services, you need:

1. **Alibaba Cloud Account**: Sign up at [Alibaba Cloud](https://www.alibabacloud.com/)
2. **API Key**: Generate an API key from your Model Studio dashboard
3. **Model Selection**: Choose from available Qwen models with multilingual capabilities

### Required Environment Variables

* `QWEN_API_KEY`: Your Qwen API key for authentication


# SambaNova
Source: https://docs.pipecat.ai/server/services/llm/sambanova

LLM service implementation using SambaNova's API with OpenAI-compatible interface

## Overview

`SambaNovaLLMService` provides access to SambaNova's language models through an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports streaming responses, function calling, and context management with SambaNova's high-performance inference platform.

<CardGroup>
  <Card title="SambaNova LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.sambanova.llm.html">
    Pipecat's API methods for SambaNova integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14s-function-calling-sambanova.py">
    Complete example with function calling
  </Card>

  <Card title="SambaNova Documentation" icon="book" href="https://docs.sambanova.ai/">
    Official SambaNova API documentation and features
  </Card>

  <Card title="SambaNova Cloud" icon="microphone" href="https://cloud.sambanova.ai/?utm_source=pipecat&utm_medium=external&utm_campaign=cloud_signup">
    Access models and manage API keys
  </Card>
</CardGroup>

## Installation

To use SambaNova services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[sambanova]"
```

## Prerequisites

### SambaNova Account Setup

Before using SambaNova LLM services, you need:

1. **SambaNova Account**: Sign up at [SambaNova Cloud](https://cloud.sambanova.ai/?utm_source=pipecat\&utm_medium=external\&utm_campaign=cloud_signup)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Selection**: Choose from available high-performance models

### Required Environment Variables

* `SAMBANOVA_API_KEY`: Your SambaNova API key for authentication


# Together AI
Source: https://docs.pipecat.ai/server/services/llm/together

LLM service implementation using Together AI's API with OpenAI-compatible interface

## Overview

`TogetherLLMService` provides access to Together AI's language models, including Meta's Llama 3.1 and 3.2 models, through an OpenAI-compatible interface. It inherits from `OpenAILLMService` and supports streaming responses, function calling, and context management with optimized open-source model hosting.

<CardGroup>
  <Card title="Together AI LLM API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.together.llm.html">
    Pipecat's API methods for Together AI integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14c-function-calling-together.py">
    Complete example with function calling
  </Card>

  <Card title="Together AI Documentation" icon="book" href="https://docs.together.ai/reference/chat-completions-1">
    Official Together AI API documentation and features
  </Card>

  <Card title="Together AI Platform" icon="microphone" href="https://together.ai/">
    Access open-source models and manage API keys
  </Card>
</CardGroup>

## Installation

To use Together AI services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[together]"
```

## Prerequisites

### Together AI Account Setup

Before using Together AI LLM services, you need:

1. **Together AI Account**: Sign up at [Together AI](https://together.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Selection**: Choose from available open-source models (Llama, Mistral, etc.)

### Required Environment Variables

* `TOGETHER_API_KEY`: Your Together AI API key for authentication


# Mem0
Source: https://docs.pipecat.ai/server/services/memory/mem0

Long-term conversation memory service powered by Mem0

## Overview

`Mem0MemoryService` provides long-term memory capabilities for conversational agents by integrating with Mem0's API. It automatically stores conversation history and retrieves relevant past context based on the current conversation, enhancing LLM responses with persistent memory across sessions.

<CardGroup>
  <Card title="Mem0 Memory API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.mem0.memory.html">
    Pipecat's API methods for Mem0 memory integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/37-mem0.py">
    Browse examples using Mem0 memory
  </Card>

  <Card title="Mem0 Documentation" icon="book" href="https://docs.mem0.ai/">
    Official Mem0 API documentation and guides
  </Card>

  <Card title="Mem0 Platform" icon="microphone" href="https://mem0.ai">
    Access memory services and manage API keys
  </Card>
</CardGroup>

## Installation

To use Mem0 memory services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[mem0]"
```

## Prerequisites

### Mem0 Account Setup

Before using Mem0 memory services, you need:

1. **Mem0 Account**: Sign up at [Mem0 Platform](https://mem0.ai)
2. **API Key**: Generate an API key from your account dashboard
3. **Host Configuration**: Set up your Mem0 host endpoint
4. **User Management**: Configure user IDs for memory association

### Required Environment Variables

* `MEM0_API_KEY`: Your Mem0 API key for authentication

### Configuration Options

* **User ID**: Unique identifier for associating memories with specific users
* **Agent ID**: Identifier for the agent using the memory service
* **Run ID**: Identifier for specific conversation sessions
* **Memory Retrieval**: Configure how past context is retrieved and used

### Key Features

* **Persistent Memory**: Long-term conversation history across sessions
* **Context Retrieval**: Automatic retrieval of relevant past conversations
* **User Association**: Memory tied to specific users for personalization
* **Session Management**: Track conversations across different runs and agents


# AWS Nova Sonic
Source: https://docs.pipecat.ai/server/services/s2s/aws

Real-time speech-to-speech service implementation using AWS Nova Sonic

## Overview

`AWSNovaSonicLLMService` enables natural, real-time conversations with AWS Nova Sonic. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences with bidirectional audio streaming, text generation, and function calling capabilities.

<CardGroup>
  <Card title="AWS Nova Sonic API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.aws.nova_sonic.llm.html">
    Pipecat's API methods for AWS Nova Sonic integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/40-aws-nova-sonic.py">
    Complete AWS Nova Sonic conversation example
  </Card>

  <Card title="AWS Bedrock Documentation" icon="book" href="https://docs.aws.amazon.com/bedrock/">
    Official AWS Bedrock and Nova Sonic documentation
  </Card>

  <Card title="AWS Console" icon="external-link" href="https://console.aws.amazon.com/bedrock/">
    Access AWS Bedrock and manage Nova Sonic models
  </Card>
</CardGroup>

## Installation

To use AWS Nova Sonic services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[aws-nova-sonic]"
```

## Prerequisites

### AWS Account Setup

Before using AWS Nova Sonic services, you need:

1. **AWS Account**: Set up at [AWS Console](https://console.aws.amazon.com/)
2. **Bedrock Access**: Enable AWS Bedrock service in your region
3. **Model Access**: Request access to Nova Sonic models in Bedrock
4. **IAM Credentials**: Configure AWS access keys with Bedrock permissions

### Required Environment Variables

* `AWS_SECRET_ACCESS_KEY`: Your AWS secret access key
* `AWS_ACCESS_KEY_ID`: Your AWS access key ID
* `AWS_REGION`: AWS region where Bedrock is available

### Key Features

* **Real-time Speech-to-Speech**: Direct audio input to audio output processing
* **Built-in Transcription**: Automatic speech-to-text with real-time streaming
* **Voice Activity Detection**: Automatic detection of speech start/stop
* **Function Calling**: Support for external function and API integration
* **Multiple Voices**: Choose from matthew, tiffany, and amy voice options


# Gemini Live
Source: https://docs.pipecat.ai/server/services/s2s/gemini-live

A real-time, multimodal conversational AI service powered by Google's Gemini

## Overview

`GeminiLiveLLMService` enables natural, real-time conversations with Google's Gemini model. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences with multimodal capabilities including audio, video, and text processing.

<Tip>
  Want to start building? Check out our [Gemini Live
  Guide](/guides/features/gemini-live).
</Tip>

<CardGroup>
  <Card title="Gemini Live API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.google.gemini_live.llm.html">
    Pipecat's API methods for Gemini Live integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/26b-gemini-live-function-calling.py">
    Complete Gemini Live function calling example
  </Card>

  <Card title="Gemini Documentation" icon="book" href="https://ai.google.dev/gemini-api/docs/live">
    Official Google Gemini Live API documentation
  </Card>

  <Card title="Gemini Live Model Card" icon="book" href="https://ai.google.dev/gemini-api/docs/models#gemini-2.5-flash-live">
    Gemini Live available models
  </Card>
</CardGroup>

## Installation

To use Gemini Live services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[google]"
```

## Prerequisites

### Google AI Setup

Before using Gemini Live services, you need:

1. **Google Account**: Set up at [Google AI Studio](https://aistudio.google.com/)
2. **API Key**: Generate a Gemini API key from AI Studio
3. **Model Access**: Ensure access to Gemini Live models
4. **Multimodal Configuration**: Set up audio, video, and text modalities

### Required Environment Variables

* `GOOGLE_API_KEY`: Your Google Gemini API key for authentication

### Key Features

* **Multimodal Processing**: Handle audio, video, and text inputs simultaneously
* **Real-time Streaming**: Low-latency audio and video processing
* **Voice Activity Detection**: Automatic speech detection and turn management
* **Function Calling**: Advanced tool integration and API calling capabilities
* **Context Management**: Intelligent conversation history and system instruction handling


# Gemini Live Vertex AI
Source: https://docs.pipecat.ai/server/services/s2s/gemini-live-vertex

A real-time, multimodal conversational AI service powered by Google's Gemini via Vertex AI

## Overview

`GeminiLiveVertexLLMService` enables natural, real-time conversations with Google's Gemini model through Vertex AI. It provides built-in audio transcription, voice activity detection, and context management for creating interactive AI experiences with multimodal capabilities including audio, video, and text processing.

<Tip>
  Want to start building? Check out our [Gemini Live
  Guide](/guides/features/gemini-live) for general concepts, then follow the
  Vertex AI-specific setup below.
</Tip>

<CardGroup>
  <Card title="Gemini Live Vertex API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.google.gemini_live.llm_vertex.html">
    Pipecat's API methods for Gemini Live Vertex AI integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/26h-gemini-live-vertex-function-calling.py">
    Complete Gemini Live Vertex AI function calling example
  </Card>

  <Card title="Vertex AI Gemini Documentation" icon="book" href="https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/multimodal-live">
    Official Vertex AI Gemini Live API documentation
  </Card>

  <Card title="Gemini Live Model Card" icon="book" href="https://docs.cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-flash-live-api">
    Gemini Live available models
  </Card>
</CardGroup>

## Installation

To use Gemini Live Vertex AI services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[google]"
```

## Prerequisites

### Google Cloud Setup

Before using Gemini Live Vertex AI services, you need:

1. **Google Cloud Project**: Set up a project in the [Google Cloud Console](https://console.cloud.google.com/)
2. **Vertex AI API**: Enable the Vertex AI API in your project
3. **Service Account**: Create a service account with `roles/aiplatform.user` and `roles/ml.developer` permissions
4. **Authentication**: Set up service account credentials or Application Default Credentials

### Required Environment Variables

* `GOOGLE_VERTEX_TEST_CREDENTIALS`: JSON string of service account credentials (optional if using ADC)
* `GOOGLE_CLOUD_PROJECT_ID`: Your Google Cloud project ID
* `GOOGLE_CLOUD_LOCATION`: Vertex AI region (e.g., "us-east4")

### Key Features

* **Enterprise Authentication**: Secure service account-based authentication
* **Multimodal Processing**: Handle audio, video, and text inputs simultaneously
* **Real-time Streaming**: Low-latency audio and video processing
* **Voice Activity Detection**: Automatic speech detection and turn management
* **Function Calling**: Advanced tool integration and API calling capabilities
* **Context Management**: Intelligent conversation history and system instruction handling


# OpenAI Realtime
Source: https://docs.pipecat.ai/server/services/s2s/openai

Real-time speech-to-speech service implementation using OpenAI's Realtime API

## Overview

`OpenAIRealtimeLLMService` provides real-time, multimodal conversation capabilities using OpenAI's Realtime API. It supports speech-to-speech interactions with integrated LLM processing, function calling, and advanced conversation management with minimal latency response times.

<CardGroup>
  <Card title="OpenAI Realtime API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.openai.realtime.llm.html">
    Pipecat's API methods for OpenAI Realtime integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/19-openai-realtime.py">
    Complete OpenAI Realtime conversation example
  </Card>

  <Card title="OpenAI Documentation" icon="book" href="https://platform.openai.com/docs/guides/realtime">
    Official OpenAI Realtime API documentation
  </Card>

  <Card title="OpenAI Platform" icon="external-link" href="https://platform.openai.com/">
    Access Realtime models and manage API keys
  </Card>
</CardGroup>

## Installation

To use OpenAI Realtime services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[openai]"
```

## Prerequisites

### OpenAI Account Setup

Before using OpenAI Realtime services, you need:

1. **OpenAI Account**: Sign up at [OpenAI Platform](https://platform.openai.com/)
2. **API Key**: Generate an OpenAI API key from your account dashboard
3. **Model Access**: Ensure access to GPT-4o Realtime models
4. **Usage Limits**: Configure appropriate usage limits and billing

### Required Environment Variables

* `OPENAI_API_KEY`: Your OpenAI API key for authentication

### Key Features

* **Real-time Speech-to-Speech**: Direct audio processing with minimal latency
* **Advanced Turn Detection**: Multiple voice activity detection options including semantic detection
* **Function Calling**: Seamless support for external functions and APIs
* **Voice Options**: Multiple voice personalities and speaking styles
* **Conversation Management**: Intelligent context handling and conversation flow control


# Ultravox Realtime
Source: https://docs.pipecat.ai/server/services/s2s/ultravox

Real-time speech-to-speech service implementation using Ultravox's Realtime API

## Overview

`UltravoxRealtimeLLMService` provides real-time conversational AI capabilities using Ultravox's Realtime API. It supports both text and audio modalities with voice transcription, streaming responses, and tool usage for creating interactive AI experiences.

<CardGroup>
  <Card title="Ultravox Realtime API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.ultravox.llm.html">
    Pipecat's API methods for Ultravox Realtime integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/50-ultravox-realtime.py">
    Complete Ultravox Realtime conversation example
  </Card>

  <Card title="Ultravox Documentation" icon="book" href="https://docs.ultravox.ai/overview">
    Official Ultravox API documentation
  </Card>

  <Card title="Ultravox Console" icon="external-link" href="https://app.ultravox.ai/">
    Access Ultravox models and manage API keys
  </Card>
</CardGroup>

## Installation

To use Ultravox Realtime services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[ultravox]"
```

## Prerequisites

### Ultravox Account Setup

Before using Ultravox Realtime services, you need:

1. **Ultravox Account**: Sign up at [Ultravox Console](https://app.ultravox.ai/)
2. **API Key**: Generate an Ultravox API key from your account dashboard
3. **Model Access**: Ensure access to Ultravox Realtime models
4. **Usage Limits**: Configure appropriate usage limits and billing

### Required Environment Variables

* `ULTRAVOX_API_KEY`: Your Ultravox API key for authentication

### Key Features

* **Audio-Native Model**: Ultravox is an audio-native model for natural voice interactions
* **Real-time Streaming**: Low-latency audio processing and streaming responses
* **Multiple Input Modes**: Support for Agent, One-Shot, and Join URL input parameters
* **Voice Transcription**: Built-in transcription with streaming output
* **Function Calling**: Support for tool integration and API calling
* **Configurable Duration**: Set maximum call duration limits


# ExotelFrameSerializer
Source: https://docs.pipecat.ai/server/services/serializers/exotel

Serializer for Exotel WebSocket media streaming protocol

## Overview

`ExotelFrameSerializer` enables integration with Exotel's WebSocket media streaming protocol, allowing your Pipecat application to handle phone calls via Exotel's voice services with bidirectional audio conversion and DTMF event handling for Indian telephony infrastructure.

<CardGroup>
  <Card title="Exotel Serializer API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.serializers.exotel.html">
    Pipecat's API methods for Exotel WebSocket integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/exotel-chatbot">
    Complete telephony examples with Exotel
  </Card>

  <Card title="Exotel Documentation" icon="book" href="https://developer.exotel.com/">
    Official Exotel developer documentation
  </Card>

  <Card title="Exotel Console" icon="microphone" href="https://my.exotel.com/">
    Manage phone numbers and streaming configuration
  </Card>
</CardGroup>

## Installation

The `ExotelFrameSerializer` does not require any additional dependencies beyond the core Pipecat library:

```bash theme={null}
pip install "pipecat-ai"
```

## Prerequisites

### Exotel Account Setup

Before using ExotelFrameSerializer, you need:

1. **Exotel Account**: Sign up at [Exotel Console](https://my.exotel.com/)
2. **Phone Number**: Purchase an Exotel phone number with voice capabilities
3. **Media Streaming**: Configure your phone number for WebSocket streaming
4. **Webhook Configuration**: Set up webhook endpoints for call handling

### Required Configuration

* **Stream ID**: Provided by Exotel during WebSocket connection
* **Call SID**: Associated Exotel Call SID (optional)

### Key Features

* **Bidirectional Audio**: Convert between Pipecat and Exotel audio formats
* **DTMF Handling**: Process touch-tone events from callers
* **Indian Telephony**: Optimized for Indian voice infrastructure
* **WebSocket Streaming**: Real-time audio streaming via WebSocket protocol


# Frame Serializer Overview
Source: https://docs.pipecat.ai/server/services/serializers/introduction

Overview of frame serializers for converting between Pipecat frames and external protocols

## Overview

Frame serializers are components that convert between Pipecat's internal frame format and external protocols or formats. They're essential when integrating with third-party services or APIs that have their own message formats.

## Core Responsibilities

Serializers handle:

1. **Serialization**: Converting Pipecat frames to external formats or protocols
2. **Deserialization**: Converting external messages to Pipecat frames
3. **Protocol-specific behaviors**: Managing unique aspects of each integration

## Available Serializers

Pipecat includes serializers for popular voice and communications platforms:

<CardGroup>
  <Card title="Exotel Serializer" icon="phone" href="/server/services/serializers/exotel">
    For integrating with Exotel WebSocket media streaming
  </Card>

  <Card title="Plivo Serializer" icon="phone" href="/server/services/serializers/plivo">
    For integrating with Telnyx WebSocket media streaming
  </Card>

  <Card title="Telnyx Serializer" icon="phone" href="/server/services/serializers/telnyx">
    For integrating with Telnyx WebSocket media streaming
  </Card>

  <Card title="Twilio Serializer" icon="phone" href="/server/services/serializers/twilio">
    For integrating with Twilio Media Streams WebSocket protocol
  </Card>

  <Card title="Vonage Serializer" icon="waveform" href="/server/services/serializers/vonage">
    For integrating with Vonage Video API Audio Connector WebSocket protocol
  </Card>
</CardGroup>

## Custom Serializers

You can create custom serializers by implementing the `FrameSerializer` base class:

```python theme={null}
from pipecat.serializers.base_serializer import FrameSerializer, FrameSerializerType
from pipecat.frames.frames import Frame, StartFrame

class MyCustomSerializer(FrameSerializer):
    @property
    def type(self) -> FrameSerializerType:
        return FrameSerializerType.TEXT  # or BINARY

    async def setup(self, frame: StartFrame):
        # Initialize with pipeline configuration
        pass

    async def serialize(self, frame: Frame) -> str | bytes | None:
        # Convert Pipecat frame to external format
        pass

    async def deserialize(self, data: str | bytes) -> Frame | None:
        # Convert external data to Pipecat frame
        pass
```


# PlivoFrameSerializer
Source: https://docs.pipecat.ai/server/services/serializers/plivo

Serializer for Plivo Audio Streaming WebSocket protocol

## Overview

`PlivoFrameSerializer` enables integration with Plivo's Audio Streaming WebSocket protocol, allowing your Pipecat application to handle phone calls via Plivo's voice services with bidirectional audio conversion, DTMF event handling, and automatic call termination.

<CardGroup>
  <Card title="Plivo Serializer API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.serializers.plivo.html">
    Pipecat's API methods for Plivo Audio Streaming integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/plivo-chatbot">
    Complete telephony examples with Plivo
  </Card>

  <Card title="Plivo Documentation" icon="book" href="https://www.plivo.com/docs/voice/api/stream/">
    Official Plivo Audio Streaming documentation
  </Card>

  <Card title="Plivo Console" icon="microphone" href="https://console.plivo.com/">
    Manage phone numbers and streaming configuration
  </Card>
</CardGroup>

## Installation

The `PlivoFrameSerializer` does not require any additional dependencies beyond the core Pipecat library:

```bash theme={null}
pip install "pipecat-ai"
```

## Prerequisites

### Plivo Account Setup

Before using PlivoFrameSerializer, you need:

1. **Plivo Account**: Sign up at [Plivo Console](https://console.plivo.com/)
2. **Phone Number**: Purchase a Plivo phone number with voice capabilities
3. **Audio Streaming**: Configure your phone number for WebSocket streaming
4. **Webhook Configuration**: Set up webhook endpoints for call handling

### Required Environment Variables

* `PLIVO_AUTH_ID`: Your Plivo Auth ID for authentication
* `PLIVO_AUTH_TOKEN`: Your Plivo Auth Token for API operations

### Required Configuration

* **Stream ID**: Provided by Plivo during Audio Streaming connection
* **Call ID**: Required for automatic call termination (optional)

### Key Features

* **Bidirectional Audio**: Convert between Pipecat and Plivo audio formats
* **DTMF Handling**: Process touch-tone events from callers
* **Auto Hang-up**: Terminate calls via Plivo's REST API
* **μ-law Encoding**: Handle Plivo's standard audio encoding format


# TelnyxFrameSerializer
Source: https://docs.pipecat.ai/server/services/serializers/telnyx

Serializer for Telnyx WebSocket media streaming protocol

## Overview

`TelnyxFrameSerializer` enables integration with Telnyx's WebSocket media streaming protocol, allowing your Pipecat application to handle phone calls via Telnyx's voice services with bidirectional audio conversion, DTMF event handling, and support for multiple audio encodings.

<CardGroup>
  <Card title="Telnyx Serializer API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.serializers.telnyx.html">
    Pipecat's API methods for Telnyx WebSocket integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/telnyx-chatbot">
    Complete telephony examples with Telnyx
  </Card>

  <Card title="Telnyx Documentation" icon="book" href="https://developers.telnyx.com/docs/voice/media-streaming">
    Official Telnyx media streaming documentation
  </Card>

  <Card title="Telnyx Portal" icon="microphone" href="https://portal.telnyx.com/">
    Manage phone numbers and streaming configuration
  </Card>
</CardGroup>

## Installation

The `TelnyxFrameSerializer` does not require any additional dependencies beyond the core Pipecat library:

```bash theme={null}
pip install "pipecat-ai"
```

## Prerequisites

### Telnyx Account Setup

Before using TelnyxFrameSerializer, you need:

1. **Telnyx Account**: Sign up at [Telnyx Portal](https://portal.telnyx.com/)
2. **Phone Number**: Purchase a Telnyx phone number with voice capabilities
3. **Media Streaming**: Configure your phone number for WebSocket streaming
4. **Webhook Configuration**: Set up webhook endpoints for call handling

### Required Environment Variables

* `TELNYX_API_KEY`: Your Telnyx API key for authentication and call control

### Required Configuration

* **Stream ID**: Provided by Telnyx during WebSocket connection
* **Audio Encodings**: Configure inbound/outbound encodings (PCMU, PCMA)
* **Call Control ID**: Required for automatic call termination (optional)

### Key Features

* **Bidirectional Audio**: Convert between Pipecat and Telnyx audio formats
* **DTMF Handling**: Process touch-tone events from callers
* **Auto Hang-up**: Terminate calls via Telnyx's REST API
* **Multiple Encodings**: Support for PCMU and PCMA audio formats


# TwilioFrameSerializer
Source: https://docs.pipecat.ai/server/services/serializers/twilio

Serializer for Twilio Media Streams WebSocket protocol

## Overview

`TwilioFrameSerializer` enables integration with Twilio's Media Streams WebSocket protocol, allowing your Pipecat application to handle phone calls via Twilio's voice services with bidirectional audio conversion, DTMF event handling, and automatic call termination.

<CardGroup>
  <Card title="Twilio Serializer API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.serializers.twilio.html">
    Pipecat's API methods for Twilio Media Streams integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/twilio-chatbot">
    Complete telephony examples with Twilio
  </Card>

  <Card title="Twilio Documentation" icon="book" href="https://www.twilio.com/docs/voice/media-streams">
    Official Twilio Media Streams documentation
  </Card>

  <Card title="Twilio Console" icon="microphone" href="https://console.twilio.com/">
    Manage phone numbers and Media Stream configuration
  </Card>
</CardGroup>

## Installation

The `TwilioFrameSerializer` does not require any additional dependencies beyond the core Pipecat library:

```bash theme={null}
pip install "pipecat-ai"
```

## Prerequisites

### Twilio Account Setup

Before using TwilioFrameSerializer, you need:

1. **Twilio Account**: Sign up at [Twilio Console](https://console.twilio.com/)
2. **Phone Number**: Purchase a Twilio phone number with voice capabilities
3. **Media Streams**: Configure your phone number to use Media Streams
4. **Webhook Configuration**: Set up webhook endpoints for call handling

### Required Environment Variables

* `TWILIO_ACCOUNT_SID`: Your Twilio Account SID for authentication
* `TWILIO_AUTH_TOKEN`: Your Twilio Auth Token for API operations

### Required Configuration

* **Stream SID**: Provided by Twilio during Media Stream connection
* **Call SID**: Required for automatic call termination (optional)

### Key Features

* **Bidirectional Audio**: Convert between Pipecat and Twilio audio formats
* **DTMF Handling**: Process touch-tone events from callers
* **Auto Hang-up**: Terminate calls via Twilio's REST API
* **μ-law Encoding**: Handle Twilio's standard audio encoding format


# VonageFrameSerializer
Source: https://docs.pipecat.ai/server/services/serializers/vonage

Serializer for Vonage Video API Audio Connector WebSocket protocol

## Overview

`VonageFrameSerializer` enables integration with the Vonage Video API Audio Connector WebSocket protocol, allowing Pipecat applications to process real-time audio streams from active Vonage video sessions.

<CardGroup>
  <Card title="Vonage Serializer API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.serializers.vonage.html">
    Pipecat's API methods for Vonage Audio Connector Streams integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/vonage-audio-bot">
    End-to-end Pipecat example using Vonage Audio Connector
  </Card>

  <Card title="Vonage Audio Connector Documentation" icon="book" href="https://developer.vonage.com/en/video/guides/audio-connector">
    Official Vonage Video API Audio Connector documentation
  </Card>

  <Card title="Vonage Video API Console" icon="microphone" href="https://tokbox.com/account/">
    Manage Vonage Video API projects
  </Card>
</CardGroup>

## Installation

The `VonageFrameSerializer` does not require any additional dependencies beyond the core Pipecat library:

```bash theme={null}
pip install "pipecat-ai"
```

## Prerequisites

### Vonage Video API Account Setup

Before using VonageFrameSerializer, you need:

1. **Vonage (TokBox) Account**: Sign up at [Vonage Video API Console](https://tokbox.com/account/)
2. **Vonage Video API Project**: Create a project to obtain Project API Key and Project Secret
3. **Existing Vonage Video Session**: A Vonage session must already exist. Sessions can be created using TokBox Playground or Vonage Video API SDKs

### Required Environment Variables

* `VONAGE_API_KEY`: Your Vonage Video API project key
* `VONAGE_API_SECRET`: Your Vonage Video API project secret
* `VONAGE_SESSION_ID`: The existing routed session ID
* `WS_URI`: Public WebSocket endpoint URI of the server application running Pipecat (e.g. via ngrok)

### Required Configuration

* **WebSocket Endpoint (/ws)**: A WebSocket server application (e.g. FastAPI) running Pipecat that accepts raw PCM audio frames.
* **Audio Connector /connect Request**: Triggers Vonage to open a WebSocket connection to your server and begin streaming audio from the active session.

### Key Features

* **Bidirectional Audio**: Convert between Pipecat and Vonage Audio Connector formats
* **Real-Time AI Pipelines**: Stream live audio into Pipecat and process it through any real-time pipeline configuration supported by the framework
* **Session Control Events**: Handle Vonage Audio Connector JSON events
* **Linear PCM Audio**: Handle raw 16-bit linear PCM audio streams used by the Vonage Video API Audio Connector


# AssemblyAI
Source: https://docs.pipecat.ai/server/services/stt/assemblyai

Speech-to-text service implementation using AssemblyAI's real-time transcription API

## Overview

`AssemblyAISTTService` provides real-time speech recognition using AssemblyAI's WebSocket API with support for interim results, end-of-turn detection, and configurable audio processing parameters for accurate transcription in conversational AI applications.

<CardGroup>
  <Card title="AssemblyAI STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.assemblyai.stt.html">
    Pipecat's API methods for AssemblyAI STT integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07o-interruptible-assemblyai.py">
    Complete example with interruption handling
  </Card>

  <Card title="AssemblyAI Documentation" icon="book" href="https://www.assemblyai.com/docs/api-reference/overview">
    Official AssemblyAI documentation and features
  </Card>

  <Card title="AssemblyAI Console" icon="microphone" href="https://www.assemblyai.com/dashboard/signup">
    Access API keys and transcription features
  </Card>
</CardGroup>

## Installation

To use AssemblyAI services, install the required dependency:

```bash theme={null}
pip install "pipecat-ai[assemblyai]"
```

## Prerequisites

### AssemblyAI Account Setup

Before using AssemblyAI STT services, you need:

1. **AssemblyAI Account**: Sign up at [AssemblyAI Console](https://www.assemblyai.com/dashboard/signup)
2. **API Key**: Generate an API key from your dashboard
3. **Model Selection**: Choose from available transcription models and features

### Required Environment Variables

* `ASSEMBLYAI_API_KEY`: Your AssemblyAI API key for authentication


# AWS Transcribe
Source: https://docs.pipecat.ai/server/services/stt/aws

Speech-to-text service implementation using Amazon Transcribe's real-time transcription API

## Overview

`AWSTranscribeSTTService` provides real-time speech recognition using Amazon Transcribe's WebSocket streaming API with support for interim results, multiple languages, and configurable audio processing parameters for enterprise-grade transcription.

<CardGroup>
  <Card title="AWS Transcribe STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.aws.stt.html">
    Pipecat's API methods for AWS Transcribe integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07m-interruptible-aws.py">
    Complete example with AWS services integration
  </Card>

  <Card title="AWS Transcribe Documentation" icon="book" href="https://docs.aws.amazon.com/transcribe/latest/dg/streaming.html">
    Official AWS Transcribe documentation and features
  </Card>

  <Card title="AWS Console" icon="microphone" href="https://console.aws.amazon.com/transcribe/">
    Access AWS Transcribe services and IAM setup
  </Card>
</CardGroup>

## Installation

To use AWS Transcribe services, install the required dependency:

```bash theme={null}
pip install "pipecat-ai[aws]"
```

## Prerequisites

### AWS Account Setup

Before using AWS Transcribe STT services, you need:

1. **AWS Account**: Sign up at [AWS Console](https://console.aws.amazon.com/)
2. **IAM User**: Create an IAM user with Amazon Transcribe permissions
3. **Credentials**: Set up AWS access keys and region configuration

### Required Environment Variables

* `AWS_ACCESS_KEY_ID`: Your AWS access key ID
* `AWS_SECRET_ACCESS_KEY`: Your AWS secret access key
* `AWS_SESSION_TOKEN`: Session token (if using temporary credentials)
* `AWS_REGION`: AWS region (defaults to "us-east-1")


# Azure
Source: https://docs.pipecat.ai/server/services/stt/azure

Speech-to-text service using Azure Cognitive Services Speech SDK

## Overview

`AzureSTTService` provides real-time speech recognition using Azure's Cognitive Services Speech SDK with support for continuous recognition, extensive language support, and configurable audio processing for enterprise applications.

<CardGroup>
  <Card title="Azure STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.azure.stt.html">
    Pipecat's API methods for Azure Speech integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07f-interruptible-azure.py">
    Complete example with Azure services integration
  </Card>

  <Card title="Azure Speech Documentation" icon="book" href="https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/speech-to-text">
    Official Azure Speech Service documentation and features
  </Card>

  <Card title="Azure Portal" icon="microphone" href="https://portal.azure.com/">
    Create Speech Services resource and get API keys
  </Card>
</CardGroup>

## Installation

To use Azure Speech services, install the required dependency:

```bash theme={null}
pip install "pipecat-ai[azure]"
```

## Prerequisites

### Azure Account Setup

Before using Azure STT services, you need:

1. **Azure Account**: Sign up at [Azure Portal](https://portal.azure.com/)
2. **Speech Services Resource**: Create a Speech Services resource in Azure
3. **API Credentials**: Get your API key and region from the resource

### Required Environment Variables

* `AZURE_SPEECH_API_KEY`: Your Azure Speech API key
* `AZURE_SPEECH_REGION`: Your Azure Speech region


# Cartesia
Source: https://docs.pipecat.ai/server/services/stt/cartesia

Speech-to-text service implementation using Cartesia's real-time transcription API

## Overview

`CartesiaSTTService` provides real-time speech recognition using Cartesia's WebSocket API with the `ink-whisper` model, supporting streaming transcription with both interim and final results for low-latency applications.

<CardGroup>
  <Card title="Cartesia STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.cartesia.stt.html">
    Pipecat's API methods for Cartesia STT integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/13f-cartesia-transcription.py">
    Complete example with transcription logging
  </Card>

  <Card title="Cartesia Documentation" icon="book" href="https://docs.cartesia.ai/api-reference/stt/stt">
    Official Cartesia STT documentation and features
  </Card>

  <Card title="Cartesia Platform" icon="microphone" href="https://cartesia.ai/">
    Access API keys and transcription models
  </Card>
</CardGroup>

## Installation

To use Cartesia services, install the required dependency:

```bash theme={null}
pip install "pipecat-ai[cartesia]"
```

## Prerequisites

### Cartesia Account Setup

Before using Cartesia STT services, you need:

1. **Cartesia Account**: Sign up at [Cartesia](https://cartesia.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Access**: Ensure access to the ink-whisper transcription model

### Required Environment Variables

* `CARTESIA_API_KEY`: Your Cartesia API key for authentication


# Deepgram
Source: https://docs.pipecat.ai/server/services/stt/deepgram

Speech-to-text service implementations using Deepgram's real-time transcription and Flux APIs

## Overview

Deepgram provides two STT service implementations:

* `DeepgramSTTService` for real-time speech recognition using Deepgram's standard WebSocket API with support for interim results, language detection, and voice activity detection (VAD)
* `DeepgramFluxSTTService` for advanced conversational AI with Flux capabilities including intelligent turn detection, eager end-of-turn events, and enhanced speech processing for improved response timing.

<Note>
  Since Deepgram Flux provides its own user turn start and end detection, you
  should use `ExternalUserTurnStrategies` to let Flux handle turn management.
  See [User Turn
  Strategies](/server/utilities/turn-management/user-turn-strategies) for
  configuration details.
</Note>

<CardGroup>
  <Card title="Deepgram STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.deepgram.stt.html">
    Pipecat's API methods for standard Deepgram STT
  </Card>

  <Card title="Deepgram Flux API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.deepgram.flux.stt.html">
    Pipecat's API methods for Deepgram Flux STT
  </Card>

  <Card title="Standard STT Example" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07c-interruptible-deepgram.py">
    Complete example with standard Deepgram STT
  </Card>

  <Card title="Flux STT Example" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07c-interruptible-deepgram-flux.py">
    Complete example with Deepgram Flux STT
  </Card>

  <Card title="Deepgram Documentation" icon="book" href="https://developers.deepgram.com/docs/pre-recorded-audio">
    Official Deepgram documentation and features
  </Card>

  <Card title="Deepgram Console" icon="microphone" href="https://console.deepgram.com/signup">
    Access API keys and transcription models
  </Card>
</CardGroup>

## Installation

To use Deepgram services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[deepgram]"
```

## Prerequisites

### Deepgram Account Setup

Before using Deepgram STT services, you need:

1. **Deepgram Account**: Sign up at [Deepgram Console](https://console.deepgram.com/signup)
2. **API Key**: Generate an API key from your console dashboard
3. **Model Selection**: Choose from available transcription models and features

### Required Environment Variables

* `DEEPGRAM_API_KEY`: Your Deepgram API key for authentication


# ElevenLabs
Source: https://docs.pipecat.ai/server/services/stt/elevenlabs

Speech-to-text service implementation using ElevenLabs' file-based transcription API

## Overview

`ElevenLabsSTTService` provides speech recognition using ElevenLabs' Speech-to-Text API with segmented audio processing. The service uploads audio files to ElevenLabs and receives transcription results directly, supporting multiple languages and audio event tagging for comprehensive transcription.

<CardGroup>
  <Card title="ElevenLabs STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.elevenlabs.stt.html">
    Pipecat's API methods for ElevenLabs STT integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07d-interruptible-elevenlabs-http.py">
    Complete example with ElevenLabs STT and TTS
  </Card>

  <Card title="ElevenLabs Documentation" icon="book" href="https://elevenlabs.io/docs/api-reference/speech-to-text/get">
    Official ElevenLabs STT API documentation
  </Card>

  <Card title="ElevenLabs Platform" icon="microphone" href="https://elevenlabs.io/">
    Access API keys and speech-to-text models
  </Card>
</CardGroup>

## Installation

To use ElevenLabs STT services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[elevenlabs]"
```

## Prerequisites

### ElevenLabs Account Setup

Before using ElevenLabs STT services, you need:

1. **ElevenLabs Account**: Sign up at [ElevenLabs Platform](https://elevenlabs.io/)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Access**: Ensure access to the Scribe v1 transcription model
4. **HTTP Session**: Configure aiohttp session for file uploads

### Required Environment Variables

* `ELEVENLABS_API_KEY`: Your ElevenLabs API key for authentication


# Fal (Wizper)
Source: https://docs.pipecat.ai/server/services/stt/fal

Speech-to-text service implementation using Fal's Wizper API

## Overview

`FalSTTService` provides speech-to-text capabilities using Fal's Wizper API with Voice Activity Detection (VAD) to process only speech segments, optimizing API usage and improving response time for efficient transcription.

<CardGroup>
  <Card title="Fal STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.fal.stt.html">
    Pipecat's API methods for Fal Wizper integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07w-interruptible-fal.py">
    Complete example with VAD integration
  </Card>

  <Card title="Fal Documentation" icon="book" href="https://fal.ai/models/fal-ai/wizper">
    Official Fal Wizper documentation and features
  </Card>

  <Card title="Fal Platform" icon="microphone" href="https://fal.ai/">
    Access API keys and Wizper models
  </Card>
</CardGroup>

## Installation

To use Fal services, install the required dependency:

```bash theme={null}
pip install "pipecat-ai[fal]"
```

## Prerequisites

### Fal Account Setup

Before using Fal STT services, you need:

1. **Fal Account**: Sign up at [Fal Platform](https://fal.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Access**: Ensure access to the Wizper transcription model

### Required Environment Variables

* `FAL_KEY`: Your Fal API key for authentication


# Gladia
Source: https://docs.pipecat.ai/server/services/stt/gladia

Speech-to-text service implementation using Gladia's API

## Overview

`GladiaSTTService` provides real-time speech recognition using Gladia's WebSocket API with support for 99+ languages, custom vocabulary, translation, sentiment analysis, and advanced audio processing features for comprehensive transcription.

<CardGroup>
  <Card title="Gladia STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.gladia.html">
    Pipecat's API methods for Gladia STT integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07j-interruptible-gladia.py">
    Complete example with interruption handling
  </Card>

  <Card title="Gladia Documentation" icon="book" href="https://docs.gladia.io/api-reference/live-flow">
    Official Gladia documentation and features
  </Card>

  <Card title="Gladia Platform" icon="microphone" href="https://www.gladia.io/">
    Access multilingual transcription and API keys
  </Card>
</CardGroup>

## Installation

To use Gladia services, install the required dependency:

```bash theme={null}
pip install "pipecat-ai[gladia]"
```

## Prerequisites

### Gladia Account Setup

Before using Gladia STT services, you need:

1. **Gladia Account**: Sign up at [Gladia](https://www.gladia.io/)
2. **API Key**: Generate an API key from your account dashboard
3. **Region Selection**: Choose your preferred region (EU-West or US-West)

### Required Environment Variables

* `GLADIA_API_KEY`: Your Gladia API key for authentication
* `GLADIA_REGION`: Your preferred region (optional, defaults to "eu-west")


# Google
Source: https://docs.pipecat.ai/server/services/stt/google

Speech-to-text service implementation using Google Cloud's Speech-to-Text V2 API

## Overview

`GoogleSTTService` provides real-time speech recognition using Google Cloud's Speech-to-Text V2 API with support for 125+ languages, multiple models, voice activity detection, and advanced features like automatic punctuation and word-level confidence scores.

<CardGroup>
  <Card title="Google STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.google.stt.html">
    Pipecat's API methods for Google Cloud STT integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07n-interruptible-google.py">
    Complete example with Google Cloud services
  </Card>

  <Card title="Google Cloud Documentation" icon="book" href="https://cloud.google.com/speech-to-text/v2/docs">
    Official Google Cloud Speech-to-Text documentation
  </Card>

  <Card title="Google Cloud Console" icon="microphone" href="https://console.cloud.google.com/iam-admin/serviceaccounts">
    Create service accounts and manage API access
  </Card>
</CardGroup>

## Installation

To use Google Cloud Speech services, install the required dependency:

```bash theme={null}
pip install "pipecat-ai[google]"
```

## Prerequisites

### Google Cloud Setup

Before using Google Cloud STT services, you need:

1. **Google Cloud Account**: Sign up at [Google Cloud Console](https://console.cloud.google.com/)
2. **Project Setup**: Create a project and enable the Speech-to-Text API
3. **Service Account**: Create a service account with Speech-to-Text permissions
4. **Authentication**: Set up credentials via service account key or Application Default Credentials

### Required Environment Variables

* `GOOGLE_APPLICATION_CREDENTIALS`: Path to your service account key file (recommended)
* Or use Application Default Credentials for cloud deployments


# Groq (Whisper)
Source: https://docs.pipecat.ai/server/services/stt/groq

Speech-to-text service implementation using Groq's Whisper API

## Overview

`GroqSTTService` provides high-accuracy speech recognition using Groq's hosted Whisper API with ultra-fast inference speeds. It uses Voice Activity Detection (VAD) to process speech segments efficiently for optimal performance and accuracy.

<CardGroup>
  <Card title="Groq STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.groq.stt.html">
    Pipecat's API methods for Groq STT integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07l-interruptible-groq.py">
    Complete example with Groq ecosystem integration
  </Card>

  <Card title="Groq Documentation" icon="book" href="https://console.groq.com/docs/api-reference#audio-transcription">
    Official Groq STT documentation and features
  </Card>

  <Card title="Groq Console" icon="microphone" href="https://console.groq.com/keys">
    Access API keys and Whisper models
  </Card>
</CardGroup>

## Installation

To use Groq services, install the required dependency:

```bash theme={null}
pip install "pipecat-ai[groq]"
```

## Prerequisites

### Groq Account Setup

Before using Groq STT services, you need:

1. **Groq Account**: Sign up at [Groq Console](https://console.groq.com/)
2. **API Key**: Generate an API key from your console dashboard
3. **Model Access**: Ensure access to Whisper transcription models

### Required Environment Variables

* `GROQ_API_KEY`: Your Groq API key for authentication


# Hathora
Source: https://docs.pipecat.ai/server/services/stt/hathora

Speech-to-text service implementations hosted on Hathora

## Overview

Hathora is a hosting provider for several models for voice AI, which can be utilized under the single `HathoraSTTService`.

<CardGroup>
  <Card title="Hathora STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.hahotra.stt.html">
    Pipecat's API methods for Hathora-hosted STT models
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07ag-interruptible-hathora.py">
    Complete example using Hathora-hosted models
  </Card>

  <Card title="Hathora Models Documentation" icon="book" href="https://models.hathora.dev/">
    Official Hathora documentation and features
  </Card>
</CardGroup>

## Installation

To use Hathora services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[hathora]"
```

## Prerequisites

### Hathora Account Setup

Before using Hathora STT services, you need:

1. **Hathora Account**: Sign up at [Hathora Models Console](https://models.hathora.dev/)
2. **API Key**: Generate an API token from your [Tokens page](https://models.hathora.dev/tokens)

### Hathora Model Specifier

The `HathoraSTTService` accepts a `model: str` parameter which corresponds to the model you would like to use.

You can find available specifiers [here](https://models.hathora.dev/)


# OpenAI
Source: https://docs.pipecat.ai/server/services/stt/openai

Speech-to-text service implementation using OpenAI's Speech-to-Text APIs

## Overview

`OpenAISTTService` provides high-accuracy speech recognition using OpenAI's advanced transcription models, including the latest GPT-4o transcription model and the proven Whisper API. It uses Voice Activity Detection (VAD) to efficiently process speech segments with superior accuracy and context understanding.

<CardGroup>
  <Card title="OpenAI STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.openai.stt.html">
    Pipecat's API methods for OpenAI STT integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07g-interruptible-openai.py">
    Complete example with OpenAI ecosystem integration
  </Card>

  <Card title="OpenAI Documentation" icon="book" href="https://platform.openai.com/docs/api-reference/audio/createTranscription">
    Official OpenAI transcription documentation and features
  </Card>

  <Card title="OpenAI Platform" icon="microphone" href="https://platform.openai.com/api-keys">
    Access API keys and transcription models
  </Card>
</CardGroup>

## Installation

To use OpenAI services, install the required dependency:

```bash theme={null}
pip install "pipecat-ai[openai]"
```

## Prerequisites

### OpenAI Account Setup

Before using OpenAI STT services, you need:

1. **OpenAI Account**: Sign up at [OpenAI Platform](https://platform.openai.com/)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Access**: Ensure access to Whisper and GPT-4o transcription models

### Required Environment Variables

* `OPENAI_API_KEY`: Your OpenAI API key for authentication


# NVIDIA Riva
Source: https://docs.pipecat.ai/server/services/stt/riva

Speech-to-text service implementation using NVIDIA Riva

## Overview

NVIDIA Riva provides two STT service implementations: `NvidiaSTTService` for real-time streaming transcription using Parakeet models, and `NvidiaSegmentedSTTService` for segmented transcription using Canary models with advanced language support and enterprise-grade accuracy.

<CardGroup>
  <Card title="NVIDIA Riva STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.riva.stt.html">
    Pipecat's API methods for NVIDIA Riva STT integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07r-interruptible-riva-nim.py">
    Complete example with NVIDIA services integration
  </Card>

  <Card title="NVIDIA Riva Documentation" icon="book" href="https://docs.nvidia.com/deeplearning/riva/user-guide/docs/asr/asr-overview.html">
    Official NVIDIA Riva ASR documentation
  </Card>

  <Card title="NVIDIA Developer Portal" icon="microphone" href="https://developer.nvidia.com">
    Access API keys and Riva services
  </Card>
</CardGroup>

## Installation

To use NVIDIA Riva services, install the required dependency:

```bash theme={null}
pip install "pipecat-ai[nvidia]"
```

## Prerequisites

### NVIDIA Riva Setup

Before using NVIDIA Riva STT services, you need:

1. **NVIDIA Developer Account**: Sign up at [NVIDIA Developer Portal](https://developer.nvidia.com)
2. **API Key**: Generate an NVIDIA API key for Riva services
3. **Model Selection**: Choose between Parakeet (streaming) and Canary (segmented) models

### Required Environment Variables

* `NVIDIA_API_KEY`: Your NVIDIA API key for authentication


# SambaNova (Whisper)
Source: https://docs.pipecat.ai/server/services/stt/sambanova

Speech-to-text service implementation using SambaNova's Whisper API

## Overview

`SambaNovaSTTService` provides speech-to-text capabilities using SambaNova's hosted Whisper API with Voice Activity Detection (VAD) for optimized processing. It efficiently processes speech segments to deliver accurate transcription with SambaNova's high-performance inference platform.

<CardGroup>
  <Card title="SambaNova STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.sambanova.stt.html">
    Pipecat's API methods for SambaNova STT integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14s-function-calling-sambanova.py">
    Complete example with function calling
  </Card>

  <Card title="SambaNova Documentation" icon="book" href="https://docs.sambanova.ai/sambastudio/latest/open-ai-api.html">
    Official SambaNova API documentation and features
  </Card>

  <Card title="SambaNova Cloud" icon="microphone" href="https://cloud.sambanova.ai/?utm_source=pipecat&utm_medium=external&utm_campaign=cloud_signup">
    Access API keys and Whisper models
  </Card>
</CardGroup>

## Installation

To use SambaNova services, install the required dependency:

```bash theme={null}
pip install "pipecat-ai[sambanova]"
```

## Prerequisites

### SambaNova Account Setup

Before using SambaNova STT services, you need:

1. **SambaNova Account**: Sign up at [SambaNova Cloud](https://cloud.sambanova.ai/?utm_source=pipecat\&utm_medium=external\&utm_campaign=cloud_signup)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Access**: Ensure access to Whisper transcription models

### Required Environment Variables

* `SAMBANOVA_API_KEY`: Your SambaNova API key for authentication


# Sarvam
Source: https://docs.pipecat.ai/server/services/stt/sarvam

Speech-to-text service implementation using Sarvam AI's WebSocket-based streaming API

## Overview

`SarvamSTTService` provides real-time speech recognition using Sarvam AI's WebSocket API, supporting Indian language transcription with Voice Activity Detection (VAD) and multiple audio formats for high-accuracy speech recognition.

<CardGroup>
  <Card title="Sarvam STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.sarvam.stt.html">
    Pipecat's API methods for Sarvam STT integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07z-interruptible-sarvam.py">
    Complete example with interruption handling
  </Card>

  <Card title="Sarvam Documentation" icon="book" href="https://docs.sarvam.ai/api-reference-docs/api-guides-tutorials/speech-to-text/overview">
    Official Sarvam AI STT documentation and features
  </Card>

  <Card title="Sarvam AI Platform" icon="microphone" href="https://dashboard.sarvam.ai/">
    Access API keys and speech models
  </Card>
</CardGroup>

## Installation

To use Sarvam services, install the required dependency:

```bash theme={null}
pip install "pipecat-ai[sarvam]"
```

## Prerequisites

### Sarvam AI Account Setup

Before using Sarvam STT services, you need:

1. **Sarvam AI Account**: Sign up at [Sarvam AI](https://dashboard.sarvam.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Model Access**: Access to Saarika (STT) or Saaras (STT-Translate) models

### Required Environment Variables

* `SARVAM_API_KEY`: Your Sarvam AI API key for authentication


# Soniox
Source: https://docs.pipecat.ai/server/services/stt/soniox

Speech-to-text service implementation using Soniox's WebSocket API

## Overview

`SonioxSTTService` provides real-time speech-to-text transcription using Soniox's WebSocket API with support for over 60 languages, custom context, multiple languages in the same conversation, and advanced features for accurate multilingual transcription.

<CardGroup>
  <Card title="Soniox STT API Reference" icon="code" href="https://pipecat-docs.readthedocs.io/en/latest/api/pipecat.services.soniox.stt.html">
    Pipecat's API methods for Soniox STT integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07za-interruptible-soniox.py">
    Complete example with interruption handling
  </Card>

  <Card title="Soniox Documentation" icon="book" href="https://soniox.com/docs/speech-to-text/get-started">
    Official Soniox documentation and features
  </Card>

  <Card title="Soniox Console" icon="microphone" href="https://console.soniox.com/">
    Access multilingual models and API keys
  </Card>
</CardGroup>

## Installation

To use Soniox services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[soniox]"
```

## Prerequisites

### Soniox Account Setup

Before using Soniox STT services, you need:

1. **Soniox Account**: Sign up at [Soniox Console](https://console.soniox.com/)
2. **API Key**: Generate an API key from your console dashboard
3. **Language Selection**: Choose from 60+ supported languages and models

### Required Environment Variables

* `SONIOX_API_KEY`: Your Soniox API key for authentication


# Speechmatics
Source: https://docs.pipecat.ai/server/services/stt/speechmatics

Speech-to-text service implementation using Speechmatics' real-time transcription STT API

## Overview

`SpeechmaticsSTTService` enables real-time speech transcription using Speechmatics' WebSocket API with partial and final results, speaker diarization, and end of utterance detection (VAD) for comprehensive conversation analysis.

<Note>
  Since Speechmatics provides its own user turn start and end detection, you
  should use `ExternalUserTurnStrategies` to let Speechmatics handle turn
  management. See [User Turn
  Strategies](/server/utilities/turn-management/user-turn-strategies) for
  configuration details.
</Note>

<CardGroup>
  <Card title="Speechmatics STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.speechmatics.stt.html">
    Pipecat's API methods for Speechmatics STT integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07a-interruptible-speechmatics.py">
    Complete example with interruption handling
  </Card>

  <Card title="Speechmatics Documentation" icon="book" href="https://docs.speechmatics.com/rt-api-ref">
    Official Speechmatics documentation and features
  </Card>

  <Card title="Speaker Diarization Guide" icon="microphone" href="https://docs.speechmatics.com/speech-to-text/features/diarization#speaker-diarization">
    Learn about separating different speakers in audio
  </Card>
</CardGroup>

## Installation

To use Speechmatics services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[speechmatics]"
```

## Prerequisites

### Speechmatics Account Setup

Before using Speechmatics STT services, you need:

1. **Speechmatics Account**: Sign up at [Speechmatics](https://www.speechmatics.com/)
2. **API Key**: Generate an API key from your account dashboard
3. **Feature Selection**: Configure transcription features like speaker diarization

### Select Endpoint

Speechmatics STT supports the following endpoints (defaults to `EU2`):

| Region | Environment   | STT Endpoint                     | Access                    |
| ------ | ------------- | -------------------------------- | ------------------------- |
| EU     | EU1           | `wss://neu.rt.speechmatics.com/` | Self-Service / Enterprise |
| EU     | EU2 (Default) | `wss://eu2.rt.speechmatics.com/` | Self-Service / Enterprise |
| US     | US1           | `wss://wus.rt.speechmatics.com/` | Enterprise                |

### Required Environment Variables

* `SPEECHMATICS_API_KEY`: Your Speechmatics API key for authentication
* `SPEECHMATICS_RT_URL`: Speechmatics endpoint URL (optional, defaults to EU2)

## End of Turn detection

The Speechmatics STT service supports Pipecat's own end of turn detection (Silero VAD and Smart Turn) without any additional configuration. When using Pipecat's features, the `turn_detection_mode` must be set to `TurnDetectionMode.EXTERNAL` (which is the default).

### Default mode

By default, Speechmatics uses signals from Pipecat's VAD / smart turn detection as input to trigger the end of turn and finalization of the current transcript segment. This provides a seamless integration where Pipecat's voice activity detection and turn detection work in conjunction with Speechmatics' real-time processing capabilities.

<Note>
  If you wish to use features such as focussing on or ignoring other speakers,
  then you may see benefit from using `TurnDetectionMode.ADAPTIVE` or
  `TurnDetectionMode.SMART_TURN` modes.
</Note>

### Adaptive End of Turn detection

This mode looks at the content of the speech, pace of speaking and other acoustic information (using VAD) to determine when the user has finished speaking. This is especially important when using the plugin's ability to focus on a specific speaker and not have other speakers interrupt the agent / conversation.

To use this mode, set the `turn_detection_mode` to `TurnDetectionMode.ADAPTIVE` in your STT configuration. You must also remove any other VAD / smart turn features within Pipecat to ensure that there is not a conflict.

```python theme={null}
transport_params = TransportParams(
    audio_in_enabled=True,
    audio_out_enabled=True,
    # vad_analyzer=... <- REMOVE (use Speechmatics' built-in VAD)
    # turn_analyzer=... <- REMOVE (use Speechmatics' built-in end-of-turn detection)
)

...

stt = SpeechmaticsSTTService(
    api_key=os.getenv("SPEECHMATICS_API_KEY"),
    params=SpeechmaticsSTTService.InputParams(
        language=Language.EN,
        turn_detection_mode=SpeechmaticsSTTService.TurnDetectionMode.ADAPTIVE,
        speaker_active_format="<{speaker_id}>{text}</{speaker_id}>",
    ),
)
```

### Smart Turn detection

Further to `ADAPTIVE`, Speechmatics also provides its own smart turn detection which combines VAD and the use of Smart Turn v3 from Pipecat. This can be enabled by setting the `turn_detection_mode` parameter to `TurnDetectionMode.SMART_TURN`.

```python theme={null}
transport_params = TransportParams(
    audio_in_enabled=True,
    audio_out_enabled=True,
    # vad_analyzer=... <- REMOVE (use Speechmatics' built-in VAD)
    # turn_analyzer=... <- REMOVE (use Speechmatics' built-in end-of-turn detection)
)

...

stt = SpeechmaticsSTTService(
    api_key=os.getenv("SPEECHMATICS_API_KEY"),
    params=SpeechmaticsSTTService.InputParams(
        language=Language.EN,
        turn_detection_mode=SpeechmaticsSTTService.TurnDetectionMode.SMART_TURN,
        speaker_active_format="<{speaker_id}>{text}</{speaker_id}>",
    ),
)
```

## Speaker Diarization

Speechmatics STT supports speaker diarization, which separates out different speakers in the audio. The identity of each speaker is returned in the TranscriptionFrame objects in the `user_id` attribute.

If `speaker_active_format` or `speaker_passive_format` are provided, then the text output for the TranscriptionFrame will be formatted to this specification. Your system context can then be updated to include information about this format to understand which speaker spoke which words. The passive format is optional and is used when the engine has been told to focus on specific speakers and other speakers will then be formatted using the `speaker_passive_format` format.

* `speaker_active_format` -> the formatter for active speakers
* `speaker_passive_format` -> the formatter for passive / background speakers

Examples:

* `<{speaker_id}>{text}</{speaker_id}>` -> `<S1>Good morning.</S1>`.
* `@{speaker_id}: {text}` -> `@S1: Good morning.`.

### Available attributes

| Attribute          | Description                                 | Example                         |
| ------------------ | ------------------------------------------- | ------------------------------- |
| `speaker_id`       | The label of the speaker                    | `S1`                            |
| `text` / `content` | The transcribed text                        | `Good morning.`                 |
| `ts`               | The timestamp of the transcription          | `2025-09-15T19:47:29.096+00:00` |
| `start_time`       | The start time of the transcription segment | `0.0`                           |
| `end_time`         | The end time of the transcription segment   | `2.5`                           |
| `lang`             | The language of the transcription segment   | `en`                            |

## Speaker Lock

In conjunction with speaker diarization, it is possible to decide at the start or during a conversation to focus on a specific speaker, ignore or retain words from other speakers, or implicitly ignore one or more speakers altogether.

In the example below, the following will happen:

* `S1` will be transcribed as normal and drive the end of turn and the conversation flow
* `S2` will be ignored completely
* All other speakers' words will be transcribed and emitted as tagged segments, but ONLY when a speaker in focus also speaks

What this means is that if `S3` says "Hello", then it is not until `S1` speaks again that the transcription will be emitted.

```python theme={null}
stt = SpeechmaticsSTTService(
    api_key=os.getenv("SPEECHMATICS_API_KEY"),
    params=SpeechmaticsSTTService.InputParams(
        language=Language.EN,
        focus_speakers=["S1"],
        ignore_speakers=["S2"],
        focus_mode=SpeechmaticsSTTService.SpeakerFocusMode.RETAIN,
        speaker_active_format="<{speaker_id}>{text}</{speaker_id}>",
    ),
)
```

## Language Support

<Note>
  Refer to the [Speechmatics
  docs](https://docs.speechmatics.com/introduction/supported-languages) for more
  information on supported languages.
</Note>

Speechmatics STT supports the following languages and regional variants.

Setting a language can be done using the `language` parameter when creating the STT object. The exception to this is English / Mandarin which has the code `cmn_en`.

| Language Code  | Description | Locales                   |
| -------------- | ----------- | ------------------------- |
| `Language.AR`  | Arabic      | -                         |
| `Language.BA`  | Bashkir     | -                         |
| `Language.EU`  | Basque      | -                         |
| `Language.BE`  | Belarusian  | -                         |
| `Language.BG`  | Bulgarian   | -                         |
| `Language.BN`  | Bengali     | -                         |
| `Language.YUE` | Cantonese   | -                         |
| `Language.CA`  | Catalan     | -                         |
| `Language.HR`  | Croatian    | -                         |
| `Language.CS`  | Czech       | -                         |
| `Language.DA`  | Danish      | -                         |
| `Language.NL`  | Dutch       | -                         |
| `Language.EN`  | English     | `en-US`, `en-GB`, `en-AU` |
| `Language.EO`  | Esperanto   | -                         |
| `Language.ET`  | Estonian    | -                         |
| `Language.FA`  | Persian     | -                         |
| `Language.FI`  | Finnish     | -                         |
| `Language.FR`  | French      | -                         |
| `Language.GL`  | Galician    | -                         |
| `Language.DE`  | German      | -                         |
| `Language.EL`  | Greek       | -                         |
| `Language.HE`  | Hebrew      | -                         |
| `Language.HI`  | Hindi       | -                         |
| `Language.HU`  | Hungarian   | -                         |
| `Language.IA`  | Interlingua | -                         |
| `Language.IT`  | Italian     | -                         |
| `Language.ID`  | Indonesian  | -                         |
| `Language.GA`  | Irish       | -                         |
| `Language.JA`  | Japanese    | -                         |
| `Language.KO`  | Korean      | -                         |
| `Language.LV`  | Latvian     | -                         |
| `Language.LT`  | Lithuanian  | -                         |
| `Language.MS`  | Malay       | -                         |
| `Language.MT`  | Maltese     | -                         |
| `Language.CMN` | Mandarin    | `cmn-Hans`, `cmn-Hant`    |
| `Language.MR`  | Marathi     | -                         |
| `Language.MN`  | Mongolian   | -                         |
| `Language.NO`  | Norwegian   | -                         |
| `Language.PL`  | Polish      | -                         |
| `Language.PT`  | Portuguese  | -                         |
| `Language.RO`  | Romanian    | -                         |
| `Language.RU`  | Russian     | -                         |
| `Language.SK`  | Slovakian   | -                         |
| `Language.SL`  | Slovenian   | -                         |
| `Language.ES`  | Spanish     | -                         |
| `Language.SV`  | Swedish     | -                         |
| `Language.SW`  | Swahili     | -                         |
| `Language.TA`  | Tamil       | -                         |
| `Language.TH`  | Thai        | -                         |
| `Language.TR`  | Turkish     | -                         |
| `Language.UG`  | Uyghur      | -                         |
| `Language.UK`  | Ukrainian   | -                         |
| `Language.UR`  | Urdu        | -                         |
| `Language.VI`  | Vietnamese  | -                         |
| `Language.CY`  | Welsh       | -                         |

For bilingual transcription, use the `language` and `domain` parameters as follows:

| Language Code | Description        | Domain Options |
| ------------- | ------------------ | -------------- |
| `cmn_en`      | English / Mandarin | -              |
| `en_ms`       | English / Malay    | -              |
| `Language.ES` | English / Spanish  | `bilingual-en` |
| `en_ta`       | English / Tamil    | -              |

## Usage Examples

Examples are included in the Pipecat project:

* Using Speechmatics STT service -> [07a-interruptible-speechmatics.py](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07a-interruptible-speechmatics.py)
* Using Speechmatics STT service with VAD -> [07a-interruptible-speechmatics-vad.py](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07a-interruptible-speechmatics-vad.py)
* Transcribing with Speechmatics STT -> [13h-speechmatics-transcription.py](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/13h-speechmatics-transcription.py)

Sample projects:

* Guess Who -> [Guess Who](https://github.com/sam-s10s/pipecat-guess-who)
* Guess Who Board Game -> [Guess Who](https://github.com/sam-s10s/pipecat-guess-who-irl)

### Basic Configuration

Initialize the `SpeechmaticsSTTService` and use it in a pipeline:

```python theme={null}
from pipecat.services.speechmatics.stt import SpeechmaticsSTTService
from pipecat.transcriptions.language import Language

# Configure service
stt = SpeechmaticsSTTService(
    api_key="your-api-key",
    params=SpeechmaticsSTTService.InputParams(
        language=Language.FR,
    )
)

# Use in pipeline
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),
    context_aggregator.assistant()
])
```

### With Diarization

This will enable diarization and also only go to the LLM if words are spoken from the first speaker (`S1`). Words from other speakers are transcribed but only sent when the first speaker speaks. When using the `TurnDetectionMode.ADAPTIVE` or `TurnDetectionMode.SMART_TURN` options, this will use the speaker diarization to determine when a speaker is speaking. You will need to disable VAD options within the selected transport object to ensure this works correctly (see [07b-interruptible-speechmatics-vad.py](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07b-interruptible-speechmatics-vad.py) as an example).

Initialize the `SpeechmaticsSTTService` and use it in a pipeline:

```python theme={null}
from pipecat.services.speechmatics.stt import SpeechmaticsSTTService
from pipecat.transcriptions.language import Language

# Configure service
stt = SpeechmaticsSTTService(
    api_key="your-api-key",
    params=SpeechmaticsSTTService.InputParams(
        language=Language.EN,
        turn_detection_mode=SpeechmaticsSTTService.TurnDetectionMode.ADAPTIVE,
        focus_speakers=["S1"],
        speaker_active_format="<{speaker_id}>{text}</{speaker_id}>",
        speaker_passive_format="<PASSIVE><{speaker_id}>{text}</{speaker_id}></PASSIVE>",
    )
)

# Use in pipeline
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),
    context_aggregator.assistant()
])
```

## Additional Notes

* **Connection Management**: Automatically handles WebSocket connections and reconnections
* **Sample Rate**: The default sample rate of `16000` in `pcm_s16le` format
* **VAD Integration**: Optionally supports Speechmatics' built-in VAD and end of utterance detection


# Whisper
Source: https://docs.pipecat.ai/server/services/stt/whisper

Speech-to-text service implementation using locally-downloaded Whisper models

## Overview

`WhisperSTTService` provides offline speech recognition using OpenAI's Whisper models running locally. Supports multiple model sizes and hardware acceleration options including CPU, CUDA, and Apple Silicon (MLX) for privacy-focused transcription without external API calls.

<CardGroup>
  <Card title="Whisper STT API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.whisper.stt.html">
    Pipecat's API methods for Whisper STT integration
  </Card>

  <Card title="Standard Whisper Example" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/13-whisper-transcription.py">
    Complete example with standard Whisper
  </Card>

  <Card title="Whisper Documentation" icon="book" href="https://github.com/openai/whisper">
    OpenAI's Whisper research paper and model details
  </Card>

  <Card title="MLX Whisper Example" icon="microphone" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/13e-whisper-mlx.py">
    Apple Silicon optimized example
  </Card>
</CardGroup>

## Installation

Choose your installation based on your hardware:

### Standard Whisper (CPU/CUDA)

```bash theme={null}
pip install "pipecat-ai[whisper]"
```

### MLX Whisper (Apple Silicon)

```bash theme={null}
pip install "pipecat-ai[mlx-whisper]"
```

## Prerequisites

### Local Model Setup

Before using Whisper STT services, you need:

1. **Model Selection**: Choose appropriate Whisper model size (tiny, base, small, medium, large)
2. **Hardware Configuration**: Set up CPU, CUDA, or Apple Silicon acceleration
3. **Storage Space**: Ensure sufficient disk space for model downloads

### Configuration Options

* **Model Size**: Balance between accuracy and performance based on your hardware
* **Hardware Acceleration**: Configure CUDA for NVIDIA GPUs or MLX for Apple Silicon
* **Language Support**: Whisper supports 99+ languages out of the box

<Tip>
  No API keys required - Whisper runs entirely locally for complete privacy.
</Tip>


# Supported Services
Source: https://docs.pipecat.ai/server/services/supported-services

AI services integrated with Pipecat and their setup requirements

## Transports

Transports exchange audio and video streams between the user and bot.

| Service                                                                   | Setup                                 |
| ------------------------------------------------------------------------- | ------------------------------------- |
| [DailyTransport](/server/services/transport/daily)                        | `pip install "pipecat-ai[daily]"`     |
| [FastAPIWebSocketTransport](/server/services/transport/fastapi-websocket) | `pip install "pipecat-ai[websocket]"` |
| [HeyGenTransport](/server/services/transport/heygen)                      | `pip install "pipecat-ai[heygen]"`    |
| [LiveKitTransport](/server/services/transport/livekit)                    | `pip install "pipecat-ai[livekit]"`   |
| [SmallWebRTCTransport](/server/services/transport/small-webrtc)           | `pip install "pipecat-ai[webrtc]"`    |
| [TavusTransport](/server/services/transport/tavus)                        | `pip install "pipecat-ai[tavus]"`     |
| [WebSocket Transports](/server/services/transport/websocket-server)       | `pip install "pipecat-ai[websocket]"` |
| [WhatsAppTransport](/server/services/transport/whatsapp)                  | `pip install "pipecat-ai[webrtc]"`    |

## Serializers

Serializers convert between frames and media streams, enabling real-time communication over a websocket.

| Service                                       | Setup                    |
| --------------------------------------------- | ------------------------ |
| [Exotel](/server/services/serializers/exotel) | No dependencies required |
| [Plivo](/server/services/serializers/plivo)   | No dependencies required |
| [Telnyx](/server/services/serializers/telnyx) | No dependencies required |
| [Twilio](/server/services/serializers/twilio) | No dependencies required |
| [Vonage](/server/services/serializers/vonage) | No dependencies required |

## Speech-to-Text

Speech-to-Text services receive and audio input and output transcriptions.

| Service                                               | Setup                                    |
| ----------------------------------------------------- | ---------------------------------------- |
| [AssemblyAI](/server/services/stt/assemblyai)         | `pip install "pipecat-ai[assemblyai]"`   |
| [AWS Transcribe](/server/services/stt/aws)            | `pip install "pipecat-ai[aws]"`          |
| [Azure](/server/services/stt/azure)                   | `pip install "pipecat-ai[azure]"`        |
| [Cartesia](/server/services/stt/cartesia)             | `pip install "pipecat-ai[cartesia]"`     |
| [Deepgram](/server/services/stt/deepgram)             | `pip install "pipecat-ai[deepgram]"`     |
| [ElevenLabs](/server/services/stt/elevenlabs)         | `pip install "pipecat-ai[elevenlabs]"`   |
| [Fal Wizper](/server/services/stt/fal)                | `pip install "pipecat-ai[fal]"`          |
| [Gladia](/server/services/stt/gladia)                 | `pip install "pipecat-ai[gladia]"`       |
| [Google](/server/services/stt/google)                 | `pip install "pipecat-ai[google]"`       |
| [Gradium](/server/services/stt/gradium)               | `pip install "pipecat-ai[gradium]"`      |
| [Groq (Whisper)](/server/services/stt/groq)           | `pip install "pipecat-ai[groq]"`         |
| [Hathora](/server/services/stt/hathora)               | `pip install "pipecat-ai[hathora]"`      |
| [NVIDIA Riva](/server/services/stt/riva)              | `pip install "pipecat-ai[nvidia]"`       |
| [OpenAI (Whisper)](/server/services/stt/openai)       | `pip install "pipecat-ai[openai]"`       |
| [SambaNova (Whisper)](/server/services/stt/sambanova) | `pip install "pipecat-ai[sambanova]"`    |
| [Sarvam](/server/services/stt/sarvam)                 | `pip install "pipecat-ai[sarvam]"`       |
| [Soniox](/server/services/stt/soniox)                 | `pip install "pipecat-ai[soniox]"`       |
| [Speechmatics](/server/services/stt/speechmatics)     | `pip install "pipecat-ai[speechmatics]"` |
| [Whisper](/server/services/stt/whisper)               | `pip install "pipecat-ai[whisper]"`      |

## Large Language Models

LLMs receive text or audio based input and output a streaming text response.

| Service                                                | Setup                                  |
| ------------------------------------------------------ | -------------------------------------- |
| [Anthropic](/server/services/llm/anthropic)            | `pip install "pipecat-ai[anthropic]"`  |
| [AWS Bedrock](/server/services/llm/aws)                | `pip install "pipecat-ai[aws]"`        |
| [Azure](/server/services/llm/azure)                    | `pip install "pipecat-ai[azure]"`      |
| [Cerebras](/server/services/llm/cerebras)              | `pip install "pipecat-ai[cerebras]"`   |
| [DeepSeek](/server/services/llm/deepseek)              | `pip install "pipecat-ai[deepseek]"`   |
| [Fireworks AI](/server/services/llm/fireworks)         | `pip install "pipecat-ai[fireworks]"`  |
| [Google Gemini](/server/services/llm/gemini)           | `pip install "pipecat-ai[google]"`     |
| [Google Vertex AI](/server/services/llm/google-vertex) | `pip install "pipecat-ai[google]"`     |
| [Grok](/server/services/llm/grok)                      | `pip install "pipecat-ai[grok]"`       |
| [Groq](/server/services/llm/groq)                      | `pip install "pipecat-ai[groq]"`       |
| [NVIDIA NIM](/server/services/llm/nim)                 | `pip install "pipecat-ai[nvidia]"`     |
| [Ollama](/server/services/llm/ollama)                  | `pip install "pipecat-ai[ollama]"`     |
| [OpenAI](/server/services/llm/openai)                  | `pip install "pipecat-ai[openai]"`     |
| [OpenPipe](/server/services/llm/openpipe)              | `pip install "pipecat-ai[openpipe]"`   |
| [OpenRouter](/server/services/llm/openrouter)          | `pip install "pipecat-ai[openrouter]"` |
| [Perplexity](/server/services/llm/perplexity)          | `pip install "pipecat-ai[perplexity]"` |
| [Qwen](/server/services/llm/qwen)                      | `pip install "pipecat-ai[qwen]"`       |
| [SambaNova](/server/services/llm/sambanova)            | `pip install "pipecat-ai[sambanova]"`  |
| [Together AI](/server/services/llm/together)           | `pip install "pipecat-ai[together]"`   |

## Text-to-Speech

Text-to-Speech services receive text input and output audio streams or chunks.

| Service                                           | Setup                                    |
| ------------------------------------------------- | ---------------------------------------- |
| [Async](/server/services/tts/asyncai)             | `pip install "pipecat-ai[asyncai]"`      |
| [AWS Polly](/server/services/tts/aws)             | `pip install "pipecat-ai[aws]"`          |
| [Azure](/server/services/tts/azure)               | `pip install "pipecat-ai[azure]"`        |
| [Camb AI](/server/services/tts/camb)              | `pip install "pipecat-ai[camb]"`         |
| [Cartesia](/server/services/tts/cartesia)         | `pip install "pipecat-ai[cartesia]"`     |
| [Deepgram](/server/services/tts/deepgram)         | `pip install "pipecat-ai[deepgram]"`     |
| [ElevenLabs](/server/services/tts/elevenlabs)     | `pip install "pipecat-ai[elevenlabs]"`   |
| [Fish](/server/services/tts/fish)                 | `pip install "pipecat-ai[fish]"`         |
| [Google](/server/services/tts/google)             | `pip install "pipecat-ai[google]"`       |
| [Gradium](/server/services/tts/gradium)           | `pip install "pipecat-ai[gradium]"`      |
| [Groq](/server/services/tts/groq)                 | `pip install "pipecat-ai[groq]"`         |
| [Hathora](/server/services/tts/hathora)           | `pip install "pipecat-ai[hathora]"`      |
| [Hume](/server/services/tts/hume)                 | `pip install "pipecat-ai[hume]"`         |
| [Inworld](/server/services/tts/inworld)           | No dependencies required                 |
| [LMNT](/server/services/tts/lmnt)                 | `pip install "pipecat-ai[lmnt]"`         |
| [MiniMax](/server/services/tts/minimax)           | No dependencies required                 |
| [Neuphonic](/server/services/tts/neuphonic)       | `pip install "pipecat-ai[neuphonic]"`    |
| [NVIDIA Riva](/server/services/tts/riva)          | `pip install "pipecat-ai[nvidia]"`       |
| [OpenAI](/server/services/tts/openai)             | `pip install "pipecat-ai[openai]"`       |
| [Piper](/server/services/tts/piper)               | No dependencies required                 |
| [PlayHT](/server/services/tts/playht)             | `pip install "pipecat-ai[playht]"`       |
| [Resemble](/server/services/tts/resemble)         | `pip install "pipecat-ai[resemble]"`     |
| [Rime](/server/services/tts/rime)                 | `pip install "pipecat-ai[rime]"`         |
| [Sarvam](/server/services/tts/sarvam)             | No dependencies required                 |
| [Speechmatics](/server/services/tts/speechmatics) | `pip install "pipecat-ai[speechmatics]"` |
| [XTTS](/server/services/tts/xtts)                 | `pip install "pipecat-ai[xtts]"`         |

## Speech-to-Speech

Speech-to-Speech services are multi-modal LLM services that take in audio, video, or text and output audio or text.

| Service                                                          | Setup                                      |
| ---------------------------------------------------------------- | ------------------------------------------ |
| [AWS Nova Sonic](/server/services/s2s/aws)                       | `pip install "pipecat-ai[aws-nova-sonic]"` |
| [Gemini Multimodal Live](/server/services/s2s/gemini-live)       | `pip install "pipecat-ai[google]"`         |
| [Gemini Live Vertex AI](/server/services/s2s/gemini-live-vertex) | `pip install "pipecat-ai[google]"`         |
| [Grok Voice Agent](/server/services/s2s/grok)                    | `pip install "pipecat-ai[grok]"`           |
| [OpenAI Realtime](/server/services/s2s/openai)                   | `pip install "pipecat-ai[openai]"`         |
| [Ultravox](/server/services/s2s/ultravox)                        | `pip install "pipecat-ai[ultravox]"`       |

## Image Generation

Image generation services receive text inputs and output images.

| Service                                                          | Setup                              |
| ---------------------------------------------------------------- | ---------------------------------- |
| [fal](/server/services/image-generation/fal)                     | `pip install "pipecat-ai[fal]"`    |
| [Google Imagen](/server/services/image-generation/google-imagen) | `pip install "pipecat-ai[google]"` |
| [OpenAI](/server/services/image-generation/openai)               | `pip install "pipecat-ai[openai]"` |

## Video

Video services enable you to build an avatar where audio and video are synchronized.

| Service                                 | Setup                              |
| --------------------------------------- | ---------------------------------- |
| [HeyGen](/server/services/video/heygen) | `pip install "pipecat-ai[heygen]"` |
| [Simli](/server/services/video/simli)   | `pip install "pipecat-ai[simli]"`  |
| [Tavus](/server/services/video/tavus)   | `pip install "pipecat-ai[tavus]"`  |

## Memory

Memory services can be used to store and retrieve conversations.

| Service                              | Setup                            |
| ------------------------------------ | -------------------------------- |
| [mem0](/server/services/memory/mem0) | `pip install "pipecat-ai[mem0]"` |

## Vision

Vision services receive a streaming video input and output text describing the video input.

| Service                                        | Setup                                 |
| ---------------------------------------------- | ------------------------------------- |
| [Moondream](/server/services/vision/moondream) | `pip install "pipecat-ai[moondream]"` |

## Analytics & Monitoring

Analytics services help you better understand how your service operates.

| Service                                     | Setup                              |
| ------------------------------------------- | ---------------------------------- |
| [Sentry](/server/services/analytics/sentry) | `pip install "pipecat-ai[sentry]"` |


# DailyTransport
Source: https://docs.pipecat.ai/server/services/transport/daily

WebRTC transport implementation using Daily for real-time audio/video communication

## Overview

`DailyTransport` provides real-time audio and video communication using Daily's hosted WebRTC platform. It handles bidirectional media streams, participant management, transcription, recording, and telephony features without requiring your own WebRTC infrastructure. Daily manages all the complexity of WebRTC connections, NAT traversal, and global media routing.

<CardGroup>
  <Card title="DailyTransport API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.transports.daily.transport.html">
    Transport methods and configuration options
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/04a-transports-daily.py">
    Complete Daily transport voice agent example
  </Card>

  <Card title="Daily API Documentation" icon="book" href="https://docs.daily.co/reference/rest-api">
    Official Daily REST API and platform documentation
  </Card>

  <Card title="Daily Dashboard" icon="external-link" href="https://dashboard.daily.co/u/signup?pipecat=true">
    Sign up for Daily API access and room management
  </Card>
</CardGroup>

## Installation

To use `DailyTransport`, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[daily]"
```

## Prerequisites

### Daily Account Setup

Before using DailyTransport, you need:

1. **Daily Account**: Sign up at [Daily Dashboard](https://dashboard.daily.co/u/signup?pipecat=true)
2. **API Key**: Generate a Daily API key from your dashboard
3. **Room Creation**: Daily rooms must be created before connecting (see Usage section)

### Required Environment Variables

* `DAILY_API_KEY`: Your Daily API key for room creation and management

## Key Features

* **Hosted WebRTC**: No infrastructure setup required - Daily handles all WebRTC complexity
* **Multi-participant Support**: Handle multiple participants with individual audio/video tracks
* **Built-in Transcription**: Real-time speech-to-text with Deepgram integration
* **Telephony Integration**: \[Dial-in/dial-out support for phone numbers via [SIP](/guides/telephony/twilio-daily-sip)/[PSTN](/guides/telephony/daily-pstn)
* **Recording & Streaming**: Built-in call recording and live streaming capabilities
* **Global Infrastructure**: Daily's edge network ensures low latency worldwide
* **Advanced Controls**: Participant management, permissions, and media routing

## Usage

DailyTransport connects your Pipecat bot to Daily rooms where it can communicate with participants through audio, video, and data channels. Rooms must be created using the Daily API before your bot can join.

The transport integrates with Pipecat's pipeline to process participant audio through your STT, LLM, and TTS services, then send responses back to participants.

See the [complete example](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/04a-transports-daily.py) for a full implementation including:

* Daily room creation and token management
* Transport configuration with transcription and VAD
* Pipeline integration with participant event handling
* Advanced features like recording and dial-out

## Additional Resources

* [Daily REST Helper Utility](/server/utilities/daily/rest-helper)
* [Pipecat Development Runner's Transport Utilities](/server/utilities/runner/transport-utils)
* [Client SDK Integration](/client/js/transports/daily)


# FastAPIWebsocketTransport
Source: https://docs.pipecat.ai/server/services/transport/fastapi-websocket

WebSocket transport implementation for FastAPI web applications with telephony integration

## Overview

`FastAPIWebsocketTransport` provides WebSocket support for FastAPI web applications, enabling real-time audio communication over WebSocket connections. It's primarily designed for telephony integrations with providers like Twilio, Telnyx, and Plivo, supporting bidirectional audio streams with configurable serializers and voice activity detection.

<Warning>
  FastAPIWebsocketTransport is best suited for telephony applications and server-side WebSocket integrations.

  For general client/server applications, we recommend using WebRTC-based transports for more robust network and media handling.
</Warning>

<CardGroup>
  <Card title="FastAPI WebSocket API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.transports.websocket.fastapi.html">
    Pipecat's API methods for FastAPI WebSocket integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/twilio-chatbot">
    Complete Twilio telephony integration example
  </Card>

  <Card title="FastAPI Documentation" icon="book" href="https://fastapi.tiangolo.com/advanced/websockets/">
    Official FastAPI WebSocket documentation
  </Card>

  <Card title="Telephony Serializers" icon="settings" href="/server/services/serializers/introduction">
    Learn about supported FrameSerializers for telephony providers
  </Card>
</CardGroup>

## Installation

To use FastAPIWebsocketTransport, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[websocket]"
```

## Prerequisites

### FastAPI Application Setup

Before using FastAPIWebsocketTransport, you need:

1. **FastAPI Application**: Set up a FastAPI web application
2. **WebSocket Endpoint**: Configure WebSocket routes for real-time communication
3. **Telephony Provider**: Set up integration with Twilio, Telnyx, or Plivo
4. **Frame Serializers**: Configure appropriate serializers for your telephony provider

### Configuration Options

* **Serializer Selection**: Choose frame serializer based on telephony provider
* **Audio Parameters**: Configure sample rates and audio formats
* **VAD Integration**: Set up voice activity detection for optimal performance
* **Connection Management**: Handle WebSocket lifecycle and reconnections

### Key Features

* **Telephony Integration**: Optimized for Twilio, Telnyx, and Plivo WebSocket streams
* **Frame Serialization**: Built-in support for telephony provider audio formats
* **FastAPI Integration**: Seamless WebSocket handling within FastAPI applications
* **Bidirectional Audio**: Real-time audio streaming in both directions

## Usage

FastAPIWebsocketTransport integrates with your FastAPI application to handle telephony WebSocket connections. It works with telephony frame serializers to process audio streams from phone calls.

See the [complete example](https://github.com/pipecat-ai/pipecat-examples/tree/main/twilio-chatbot) for a full implementation including:

* FastAPI WebSocket endpoint configuration
* Telephony provider integration setup
* Frame serializer configuration
* Audio processing pipeline integration


# HeyGenTransport
Source: https://docs.pipecat.ai/server/services/transport/heygen

AI avatar video generation service for creating interactive conversational avatars

## Overview

`HeyGenTransport` enables your Pipecat bot to join the same virtual room as a HeyGen avatar and human participants. The transport integrates with the HeyGen [LiveAvatar](https://www.liveavatar.com/) platform to create interactive AI-powered video avatars that respond naturally in real-time conversations. The service handles bidirectional audio/video streaming, avatar animations, voice activity detection, and conversation interruptions to deliver engaging conversational AI experiences with lifelike visual presence.

When used, the Pipecat bot connects to a LiveKit room alongside the HeyGen avatar and user. The bot receives audio input from participants, processes it through your pipeline, and sends TTS audio to the HeyGen avatar for synchronized video rendering.

<CardGroup>
  <Card title="HeyGen Trasnport API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.transports.heygen.html">
    Pipecat's API methods for HeyGen video integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/43-heygen-transport.py">
    Complete example with interactive avatar
  </Card>

  <Card title="HeyGen Documentation" icon="book" href="https://docs.liveavatar.com/docs/getting-started">
    Official HeyGen API documentation and guides
  </Card>

  <Card title="HeyGen Platform" icon="microphone" href="https://www.liveavatar.com/">
    Access interactive avatars and API keys
  </Card>
</CardGroup>

## Installation

To use HeyGen services, install the required dependency:

```bash theme={null}
pip install "pipecat-ai[heygen]"
```

## Prerequisites

### HeyGen Account Setup

Before using HeyGen video services, you need:

1. **HeyGen Account**: Sign up at [HeyGen Platform](https://app.liveavatar.com/signin)
2. **API Key**: Generate an API key from your account dashboard
3. **Avatar Selection**: Choose from available interactive avatars
4. **Streaming Setup**: Configure real-time avatar streaming capabilities

### Required Environment Variables

* `HEYGEN_LIVE_AVATAR_API_KEY`: Your HeyGen LiveAvatar API key for authentication


# LiveKitTransport
Source: https://docs.pipecat.ai/server/services/transport/livekit

WebRTC transport implementation using LiveKit for real-time audio communication

## Overview

`LiveKitTransport` provides real-time audio communication capabilities using LiveKit's open-source WebRTC platform. It supports bidirectional audio streaming, data messaging, participant management, and room event handling for conversational AI applications with the flexibility of self-hosted or cloud infrastructure.

<CardGroup>
  <Card title="LiveKitTransport API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.transports.livekit.transport.html">
    Pipecat's API methods for LiveKit integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/04b-transports-livekit.py">
    Complete LiveKit voice agent example
  </Card>

  <Card title="LiveKit Documentation" icon="book" href="https://docs.livekit.io/reference/">
    Official LiveKit API documentation and guides
  </Card>

  <Card title="LiveKit Cloud" icon="external-link" href="https://cloud.livekit.io/login">
    Sign up for hosted LiveKit service
  </Card>
</CardGroup>

## Installation

To use LiveKitTransport, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[livekit]"
```

## Prerequisites

### LiveKit Setup

Before using LiveKitTransport, you need:

1. **LiveKit Server**: Set up self-hosted or use [LiveKit Cloud](https://cloud.livekit.io/)
2. **API Credentials**: Generate API key and secret from your LiveKit project
3. **Room Management**: Create rooms using LiveKit API or SDK
4. **Access Tokens**: Generate JWT tokens for room authentication

### Required Environment Variables

* `LIVEKIT_API_KEY`: Your LiveKit API key for authentication
* `LIVEKIT_API_SECRET`: Your LiveKit API secret for token generation
* `LIVEKIT_URL`: Your LiveKit server URL (wss\://your-domain.livekit.cloud)

### Key Features

* **Open Source**: Self-hosted or cloud-hosted WebRTC infrastructure
* **Multi-participant Support**: Handle multiple participants in rooms
* **Data Channels**: Real-time messaging alongside audio streams
* **Room Management**: Dynamic participant and room lifecycle management
* **Flexible Deployment**: Self-hosted or managed cloud options

## Usage

LiveKitTransport connects your Pipecat bot to LiveKit rooms where it can communicate with participants through audio and data channels. The transport handles room joining, participant events, and media streaming automatically.

See the [complete example](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/04b-transports-livekit.py) for a full implementation including:

* LiveKit room creation and token generation
* Transport configuration with participant management
* Pipeline integration with audio processing
* Event handling for room lifecycle management


# SmallWebRTCTransport
Source: https://docs.pipecat.ai/server/services/transport/small-webrtc

A lightweight WebRTC transport for peer-to-peer audio and video communication in Pipecat

## Overview

`SmallWebRTCTransport` enables peer-to-peer ("serverless") WebRTC connections between clients and your Pipecat application. It implements bidirectional audio, video and data channels using WebRTC for real-time communication. This transport is open source and self-contained, with no dependencies on any other infrastructure.

<Tip>
  For detailed notes on how to decide between using the SmallWebRTCTransport or
  other WebRTC transports like the DailyTransport, see [this
  post](https://www.daily.co/blog/you-dont-need-a-webrtc-server-for-your-voice-agents/).
</Tip>

<CardGroup>
  <Card title="SmallWebRTC Transport API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.transports.smallwebrtc.transport.html">
    Transport methods and configuration options
  </Card>

  <Card title="SmallWebRTC Connection API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.transports.smallwebrtc.connection.html">
    Connection management and signaling methods
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/p2p-webrtc/pipecat-cloud">
    Complete peer-to-peer voice agent example
  </Card>

  <Card title="WebRTC Documentation" icon="book" href="https://webrtc.org/getting-started/overview">
    Official WebRTC protocol documentation
  </Card>
</CardGroup>

## Installation

To use SmallWebRTCTransport, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[webrtc]"
```

## Prerequisites

### WebRTC Application Setup

Before using SmallWebRTCTransport, you need:

1. **Signaling Server**: Implement WebRTC offer/answer exchange (required)
2. **Client Implementation**: Set up WebRTC client for browser or application use
3. **ICE Configuration**: Configure STUN/TURN servers for NAT traversal (optional for local networks)
4. **Development Runner**: Use Pipecat's development runner for quick setup (recommended)

<Tip>
  No API keys are required since this is a peer-to-peer transport
  implementation. For production deployments across different networks, you may
  need to configure STUN/TURN servers for NAT traversal.
</Tip>

### Configuration Options

* **Development Runner**: Automatic server infrastructure and web interface (recommended)
* **Manual Implementation**: Custom signaling server for advanced use cases
* **ICE Servers**: STUN/TURN configuration for network traversal
* **Media Configuration**: Audio/video parameters and format handling

### Key Features

* **Serverless Architecture**: Direct peer-to-peer connections with no intermediate servers
* **Production Ready**: Heavily tested and used in Pipecat examples
* **Bidirectional Media**: Full-duplex audio and video streaming
* **Data Channels**: Application messaging and signaling support
* **Development Tools**: Built-in development runner with web interface

## Usage

SmallWebRTCTransport requires both a signaling server for WebRTC handshake and your Pipecat bot implementation. The easiest approach is using Pipecat's development runner which handles all server infrastructure automatically.

See the [complete examples](https://github.com/pipecat-ai/pipecat-examples/tree/main/p2p-webrtc) for full implementations including:

* Development runner setup with automatic web interface
* Manual signaling server implementation
* WebRTC client integration
* ICE server configuration for production deployment


# TavusTransport
Source: https://docs.pipecat.ai/server/services/transport/tavus

Conversational AI applications with Tavus avatars using real-time audio/video streaming

## Overview

TavusTransport enables your Pipecat bot to join the same virtual room as a Tavus avatar and human participants. The transport integrates with the Tavus platform to create conversational AI applications where a Tavus avatar provides synchronized video and audio output while your bot handles the conversation logic.

When used, the Pipecat bot connects to a Daily room alongside the Tavus avatar and user. The bot receives audio input from participants, processes it through your pipeline, and sends TTS audio to the Tavus avatar for synchronized video rendering.

<CardGroup>
  <Card title="Tavus Transport API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.transports.tavus.transport.html">
    Pipecat's API methods for Tavus platform integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/21a-tavus-video-service.py">
    Complete Tavus avatar conversation example
  </Card>

  <Card title="Tavus API Documentation" icon="book" href="https://docs.tavus.io/api-reference/phoenix-replica-model/create-replica">
    Tavus official API reference and replica creation
  </Card>

  <Card title="Client SDK Integration" icon="link" href="/client/js/transports/daily">
    Connect using Pipecat Client SDK with Daily transport
  </Card>
</CardGroup>

## Installation

To use TavusTransport, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[tavus]"
```

## Prerequisites

### Tavus Platform Setup

Before using TavusTransport, you need:

1. **Tavus Account**: Sign up at [Tavus.io](https://tavus.io)
2. **API Key**: Generate a Tavus API key for authentication
3. **Replica ID**: Create or obtain a Tavus replica model ID
4. **Persona Configuration**: Set up a persona (optional, defaults to Pipecat TTS voice)

<Tip>
  Use `persona_id="pipecat-stream"` to have Tavus use your Pipecat bot's TTS
  voice instead of a Tavus persona voice.
</Tip>

### Required Environment Variables

* `TAVUS_API_KEY`: Your Tavus API key for authentication
* `TAVUS_REPLICA_ID`: ID of the Tavus replica model to use

## Key Features

* **Avatar Integration**: Seamlessly integrates Tavus avatars with Pipecat conversations
* **Synchronized Audio/Video**: Tavus avatar renders video synchronized with bot's TTS output
* **Multi-participant Rooms**: Supports bot, avatar, and human participants in the same session
* **Conversation Management**: Handles Tavus conversation lifecycle through API
* **Real-time Streaming**: Bidirectional audio streaming with video avatar output

## Usage

TavusTransport creates a three-way conversation between your Pipecat bot, a Tavus avatar, and human participants. The transport manages the Tavus API integration and Daily room connectivity automatically.

See the [complete example](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/21a-tavus-video-service.py) for a full implementation including:

* Tavus transport configuration
* Avatar and replica setup
* Pipeline integration with TTS
* Event handling for participant management


# WebSocket Transports
Source: https://docs.pipecat.ai/server/services/transport/websocket-server

WebSocket transport implementations for real-time client-server communication

## Overview

WebSocket transports provide both client and server WebSocket implementations for real-time bidirectional communication. These transports support audio streaming, frame serialization, and connection management, making them ideal for prototyping and lightweight client-server applications where WebRTC might be overkill.

<Warning>
  WebSocket transports are best suited for prototyping and controlled network environments.

  For production client-server applications, we recommend WebRTC-based transports for more robust network handling, NAT traversal, and media optimization.
</Warning>

<CardGroup>
  <Card title="WebSocket Client API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.transports.websocket.client.html">
    Client transport methods and configuration
  </Card>

  <Card title="WebSocket Server API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.transports.websocket.server.html">
    Server transport methods and configuration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/websocket">
    Complete client and server WebSocket examples
  </Card>

  <Card title="WebSocket Protocol" icon="book" href="https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API">
    WebSocket protocol documentation and guides
  </Card>
</CardGroup>

## Installation

To use WebSocket transports, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[websocket]"
```

## Prerequisites

### WebSocket Application Setup

Before using WebSocket transports, you need:

1. **Server Implementation**: Set up WebSocket server using your preferred framework
2. **Client Implementation**: Configure WebSocket client for browser or application use
3. **Audio Configuration**: Set up audio streaming parameters and formats
4. **Connection Management**: Handle WebSocket lifecycle and error recovery

### Configuration Options

* **Transport Type**: Choose between client or server WebSocket transport
* **Audio Parameters**: Configure sample rates, channels, and audio formats
* **Frame Serialization**: Set up custom frame serializers if needed
* **Connection Handling**: Configure reconnection and error handling strategies

### Key Features

* **Bidirectional Communication**: Real-time audio and data streaming
* **Simple Protocol**: Lightweight WebSocket-based communication
* **Flexible Serialization**: Support for custom frame formats and audio codecs
* **Cross-Platform**: Works with any WebSocket-compatible client or server

## Usage

WebSocket transports provide simple client-server communication for audio streaming and real-time interaction. They're ideal for prototyping and controlled network environments.

See the [complete examples](https://github.com/pipecat-ai/pipecat-examples/tree/main/websocket) for full implementations including:

* WebSocket server setup and configuration
* Client-side WebSocket integration
* Audio streaming and frame handling
* Connection management and error recovery


# WhatsAppTransport
Source: https://docs.pipecat.ai/server/services/transport/whatsapp

Real-time voice communication through WhatsApp Business API using WebRTC

## Overview

WhatsAppTransport enables real-time voice conversations between WhatsApp users and your Pipecat bot through the WhatsApp Business API. When users call your WhatsApp Business number, the transport handles webhook events, establishes WebRTC connections, and manages the voice call lifecycle.

The transport integrates with Meta's WhatsApp Cloud API to receive incoming calls, process WebRTC signaling, and maintain call state throughout the conversation.

<CardGroup>
  <Card title="WhatsApp API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.transports.whatsapp.api.html">
    API methods for Pipecat's WhatsApp Cloud API integration
  </Card>

  <Card title="WhatsApp Client Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.transports.whatsapp.client.html">
    Pipecat's Client for handling webhooks and WebRTC connections
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/whatsapp">
    Complete WhatsApp voice bot example
  </Card>

  <Card title="WhatsApp Business API Docs" icon="book" href="https://developers.facebook.com/docs/whatsapp/cloud-api/calling/">
    Meta's official WhatsApp calling documentation
  </Card>
</CardGroup>

## Installation

To use WhatsAppTransport, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[webrtc]"
```

## Prerequisites

### WhatsApp Business API Setup

Before using WhatsAppTransport, you need:

1. **WhatsApp Business API Account**: Set up through [Meta Developer Console](https://developers.facebook.com/apps)
2. **Phone Number Configuration**: Enable voice calling capability for your business number
3. **Webhook Configuration**: Configure webhook endpoint to receive call events
4. **Access Tokens**: Generate API access token and phone number ID

<Tip>
  For development, Meta provides free test phone numbers valid for 90 days.
  Production use requires business verification.
</Tip>

### Required Environment Variables

* `WHATSAPP_TOKEN`: WhatsApp Business API access token
* `WHATSAPP_PHONE_NUMBER_ID`: Your business phone number ID
* `WHATSAPP_WEBHOOK_VERIFICATION_TOKEN`: Token for webhook verification

## Key Features

* **Incoming Call Handling**: Automatically accepts WhatsApp voice calls
* **WebRTC Integration**: Establishes peer-to-peer audio connections
* **Webhook Processing**: Handles Meta's webhook events for call lifecycle
* **Call Management**: Supports call acceptance, rejection, and termination
* **Real-time Audio**: Bidirectional audio streaming for natural conversations

## Usage

WhatsAppTransport requires a webhook server to handle incoming calls and a bot implementation to process the conversations. The transport works with `SmallWebRTCTransport` under the hood for WebRTC connectivity.

See the [complete example](https://github.com/pipecat-ai/pipecat-examples/tree/main/whatsapp) for a full implementation including:

* FastAPI webhook server
* WhatsApp client configuration
* Bot integration with Pipecat pipeline
* Environment setup and deployment


# Async
Source: https://docs.pipecat.ai/server/services/tts/asyncai

Text-to-speech services using Async's WebSocket and HTTP APIs

## Overview

Async provides high-quality text-to-speech synthesis with two service implementations: `AsyncAITTSService` (WebSocket-based) for real-time streaming with interruption support, and `AsyncAIHttpTTSService` (HTTP-based) for simpler synthesis. `AsyncAITTSService` is recommended for interactive applications requiring low latency.

<CardGroup>
  <Card title="AsyncAI TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.asyncai.tts.html">
    Pipecat's API methods for AsyncAI TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07zc-interruptible-asyncai.py">
    Complete example with WebSocket streaming
  </Card>

  <Card title="Async Documentation" icon="book" href="https://docs.async.ai/">
    Official Async API documentation
  </Card>

  <Card title="Voice Models" icon="microphone" href="https://async.ai/">
    Explore available voice models and features
  </Card>
</CardGroup>

## Installation

To use Async services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[asyncai]"
```

## Prerequisites

### Async Account Setup

Before using Async TTS services, you need:

1. **Async Account**: Sign up at [Async](https://async.ai)
2. **API Key**: Generate an API key from your account dashboard
3. **Voice Selection**: Choose from available voice models

### Required Environment Variables

* `ASYNCAI_API_KEY`: Your Async API key for authentication


# AWS Polly
Source: https://docs.pipecat.ai/server/services/tts/aws

Text-to-speech service using Amazon Polly

## Overview

`AWSPollyTTSService` provides high-quality text-to-speech synthesis through Amazon Polly with support for standard, neural, and generative engines. The service offers extensive language support, SSML features, and voice customization options including prosody controls for pitch, rate, and volume.

<CardGroup>
  <Card title="AWS Polly API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.aws.tts.html">
    Pipecat's API methods for AWS Polly integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07m-interruptible-aws.py">
    Complete example with generative engine
  </Card>

  <Card title="AWS Polly Documentation" icon="book" href="https://docs.aws.amazon.com/polly/">
    Official AWS Polly documentation and features
  </Card>

  <Card title="Voice Samples" icon="microphone" href="https://docs.aws.amazon.com/polly/latest/dg/voicelist.html">
    Browse available voices and languages
  </Card>
</CardGroup>

## Installation

To use AWS Polly services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[aws]"
```

## Prerequisites

### AWS Account Setup

Before using AWS Polly TTS services, you need:

1. **AWS Account**: Sign up at [AWS Console](https://aws.amazon.com/)
2. **IAM User**: Create an IAM user with Polly permissions
3. **Access Keys**: Generate access key ID and secret access key
4. **Voice Selection**: Choose from available voices in the [voice list](https://docs.aws.amazon.com/polly/latest/dg/voicelist.html)

### Required Environment Variables

* `AWS_ACCESS_KEY_ID`: Your AWS access key ID
* `AWS_SECRET_ACCESS_KEY`: Your AWS secret access key
* `AWS_SESSION_TOKEN`: Session token (if using temporary credentials)
* `AWS_REGION`: AWS region (defaults to "us-east-1")


# Azure
Source: https://docs.pipecat.ai/server/services/tts/azure

Text-to-speech service using Azure Cognitive Services Speech SDK

## Overview

Azure Cognitive Services provides high-quality text-to-speech synthesis with two service implementations: `AzureTTSService` (WebSocket-based) for real-time streaming with low latency, and `AzureHttpTTSService` (HTTP-based) for batch synthesis. `AzureTTSService` is recommended for interactive applications requiring streaming capabilities.

<CardGroup>
  <Card title="Azure TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.azure.tts.html">
    Pipecat's API methods for Azure TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07f-interruptible-azure.py">
    Complete example with streaming synthesis
  </Card>

  <Card title="Azure Speech Documentation" icon="book" href="https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/">
    Official Azure Speech Services documentation
  </Card>

  <Card title="Voice Gallery" icon="microphone" href="https://speech.microsoft.com/portal/voicegallery">
    Browse available voices and languages
  </Card>
</CardGroup>

## Installation

To use Azure services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[azure]"
```

## Prerequisites

### Azure Account Setup

Before using Azure TTS services, you need:

1. **Azure Account**: Sign up at [Azure Portal](https://portal.azure.com/)
2. **Speech Service**: Create a Speech resource in your Azure subscription
3. **API Key and Region**: Get your subscription key and service region
4. **Voice Selection**: Choose from available voices in the [Voice Gallery](https://speech.microsoft.com/portal/voicegallery)

### Required Environment Variables

* `AZURE_SPEECH_API_KEY`: Your Azure Speech service API key
* `AZURE_SPEECH_REGION`: Your Azure Speech service region (e.g., "eastus")


# Camb AI
Source: https://docs.pipecat.ai/server/services/tts/camb

Text-to-speech service using Camb AI's MARS models for high-quality speech synthesis

## Overview

`CambTTSService` provides high-quality text-to-speech synthesis using Camb AI's MARS model family with streaming capabilities. The service offers multiple model options optimized for different use cases: `mars-flash` for fast inference and `mars-pro` for high-quality output.

<CardGroup>
  <Card title="Camb AI TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.camb.tts.html">
    Pipecat's API methods for Camb AI TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07zg-interruptible-camb.py">
    Complete example with interruption handling
  </Card>

  <Card title="Camb AI Documentation" icon="book" href="https://docs.camb.ai/">
    Official Camb AI API documentation
  </Card>

  <Card title="Camb AI Platform" icon="microphone" href="https://studio.camb.ai/">
    Access the Camb AI studio platform
  </Card>
</CardGroup>

## Installation

To use Camb AI services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[camb]"
```

## Prerequisites

### Camb AI Account Setup

Before using Camb AI TTS services, you need:

1. **Camb AI Account**: Sign up at [Camb AI](https://studio.camb.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Voice Selection**: Choose voice IDs from the platform

### Required Environment Variables

* `CAMB_API_KEY`: Your Camb AI API key for authentication

## Notes

<Tip>
  Set the `audio_out_sample_rate` in `PipelineParams` to match the model's sample rate (22050 for `mars-flash`, 48000 for `mars-pro`) for optimal quality. See the [example implementation](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07zg-interruptible-camb.py) for usage.
</Tip>


# Cartesia
Source: https://docs.pipecat.ai/server/services/tts/cartesia

Text-to-speech services using Cartesia’s WebSocket and HTTP APIs

## Overview

Cartesia provides high-quality text-to-speech synthesis with two service implementations: `CartesiaTTSService` (WebSocket-based) for real-time streaming with word timestamps, and `CartesiaHttpTTSService` (HTTP-based) for simpler batch synthesis. `CartesiaTTSService` is recommended for interactive applications requiring low latency and interruption handling.

<CardGroup>
  <Card title="Cartesia TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.cartesia.tts.html">
    Pipecat's API methods for Cartesia TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07-interruptible.py">
    Complete example with interruption handling
  </Card>

  <Card title="Cartesia Documentation" icon="book" href="https://docs.cartesia.ai/get-started/overview">
    Official Cartesia API documentation and features
  </Card>

  <Card title="Voice Library" icon="microphone" href="https://play.cartesia.ai/">
    Browse and test available voices
  </Card>
</CardGroup>

## Installation

To use Cartesia services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[cartesia]"
```

## Prerequisites

### Cartesia Account Setup

Before using Cartesia TTS services, you need:

1. **Cartesia Account**: Sign up at [Cartesia](https://play.cartesia.ai/sign-up)
2. **API Key**: Generate an API key from your account dashboard
3. **Voice Selection**: Choose voice IDs from the [voice library](https://play.cartesia.ai/)

### Required Environment Variables

* `CARTESIA_API_KEY`: Your Cartesia API key for authentication

## Customizing Speech

`CartesiaTTSService` provides a set of helper methods for implementing Cartesia-specific customizations, meant to be used as part of text transformers. These include methods for spelling out text, adjusting speech rate, and modifying pitch. See the [Text Transformers for TTS](/guides/learn/text-to-speech#text-transformers-for-tts) section in the Text-to-Speech guide for usage examples.

### SPELL(text: str) -> str:

A convenience method to wrap text in [Cartesia's spell tag](https://docs.cartesia.ai/build-with-cartesia/sonic-3/ssml-tags#spelling-out-numbers-and-letters) for spelling out text character by character.

```python theme={null}
# Text transformers for TTS
# This will insert Cartesia's spell tags around the provided text.
async def spell_out_text(text: str, type: str) -> str:
    return CartesiaTTSService.SPELL(text)

tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    text_transforms=[
        ("phone_number", spell_out_text),
    ],
)
```

### EMOTION\_TAG(emotion: CartesiaEmotion) -> str:

A convenience method to create an [emotion tag](https://docs.cartesia.ai/build-with-cartesia/sonic-3/volume-speed-emotion#emotion-controls-beta) for expressing emotions in speech.

```python theme={null}
# Text transformers for TTS
# This will insert Cartesia's sarcasm tag in front of any sentence that is just "whatever".
async def maybe_insert_sarcasm(text: str, type: str) -> str:
    if text.strip(".!").lower() == "whatever":
        return CartesiaTTSService.EMOTION_TAG(CartesiaEmotion.SARCASM) + text + CartesiaTTSService.EMOTION_TAG(CartesiaEmotion.NEUTRAL)
    return text

tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    text_transforms=[
        ("sentence", maybe_insert_sarcasm),
    ],
)
```

### PAUSE\_TAG(seconds: float) -> str:

A convenience method to create Cartesia's [SSML tag for inserting pauses](https://docs.cartesia.ai/build-with-cartesia/sonic-3/ssml-tags#pauses-and-breaks) in speech.

```python theme={null}
# Text transformers for TTS
# This will insert a one second pause after questions.
async def pause_after_questions(text: str, type: str) -> str:
    if text.endswith("?"):
        return f"{text}{CartesiaTTSService.PAUSE_TAG(1.0)}"
    return text

tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    text_transforms=[
        ("sentence", pause_after_questions), # Only apply to sentence aggregations
    ],
)
```

### VOLUME\_TAG(volume: float) -> str:

A convenience method to create Cartesia's [SSML volume tag](https://docs.cartesia.ai/build-with-cartesia/sonic-3/ssml-tags#volume) for dynamically adjusting speech volume in situ.

```python theme={null}
# Text transformers for TTS
# This will increase the volume for any full text aggregation that is in all caps.
async def maybe_say_it_loud(text: str, type: str) -> str:
    if text.upper() == text:
        return f"{CartesiaTTSService.VOLUME_TAG(2.0)}{text}{CartesiaTTSService.VOLUME_TAG(1.0)}"
    return text

tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    text_transforms=[
        ("*", maybe_say_it_loud), # Apply to all text
    ],
)
```

### SPEED\_TAG(speed: float) -> str:

A convenience method to create Cartesia's [SSML speed tag](https://docs.cartesia.ai/build-with-cartesia/sonic-3/ssml-tags#speed) for dynamically adjusting the speech rate in situ.

```python theme={null}
# Text transformers for TTS
# This will make the word slow always be spoken more slowly.
async def slow_down_slow_words(text: str, type: str) -> str:
    return text.replace(
        "slow",
        f"{CartesiaTTSService.SPEED_TAG(0.6)}slow{CartesiaTTSService.SPEED_TAG(1.0)}"
    )

tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    text_transforms=[
        ("*", slow_down_slow_words), # Apply to all text
    ],
)
```


# Deepgram
Source: https://docs.pipecat.ai/server/services/tts/deepgram

Text-to-speech service implementation using Deepgram's Aura API

## Overview

`DeepgramTTSService` provides high-quality text-to-speech synthesis using Deepgram's Aura API with streaming capabilities and ultra-low latency. The service offers various voice models optimized for conversational AI applications with efficient audio streaming.

<CardGroup>
  <Card title="Deepgram TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.deepgram.tts.html">
    Pipecat's API methods for Deepgram TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07c-interruptible-deepgram.py">
    Complete example with Silero VAD
  </Card>

  <Card title="Deepgram Documentation" icon="book" href="https://developers.deepgram.com/reference/text-to-speech-api/speak">
    Official Deepgram Aura TTS API documentation
  </Card>

  <Card title="Voice Models" icon="microphone" href="https://developers.deepgram.com/docs/tts-models">
    Browse available Aura voice models
  </Card>
</CardGroup>

## Installation

To use Deepgram services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[deepgram]"
```

## Prerequisites

### Deepgram Account Setup

Before using Deepgram TTS services, you need:

1. **Deepgram Account**: Sign up at [Deepgram Console](https://console.deepgram.com/)
2. **API Key**: Generate an API key from your project dashboard
3. **Voice Selection**: Choose from available Aura voice models

### Required Environment Variables

* `DEEPGRAM_API_KEY`: Your Deepgram API key for authentication


# ElevenLabs
Source: https://docs.pipecat.ai/server/services/tts/elevenlabs

Text-to-speech service using ElevenLab's streaming API with word-level timing

## Overview

ElevenLabs provides high-quality text-to-speech synthesis with two service implementations: `ElevenLabsTTSService` (WebSocket-based) for real-time streaming with word timestamps and audio context management, and `ElevenLabsHttpTTSService` (HTTP-based) for simpler integration. `ElevenLabsTTSService` is recommended for interactive applications requiring low latency and precise timing.

<CardGroup>
  <Card title="ElevenLabs TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.elevenlabs.tts.html">
    Pipecat's API methods for ElevenLabs TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07d-interruptible-elevenlabs.py">
    Complete example with WebSocket streaming
  </Card>

  <Card title="ElevenLabs Documentation" icon="book" href="https://elevenlabs.io/docs/api-reference/text-to-speech/v-1-text-to-speech-voice-id-multi-stream-input">
    Official ElevenLabs TTS API documentation
  </Card>

  <Card title="Voice Library" icon="microphone" href="https://elevenlabs.io/voice-library">
    Browse and clone voices from the community
  </Card>
</CardGroup>

## Installation

To use ElevenLabs services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[elevenlabs]"
```

## Prerequisites

### ElevenLabs Account Setup

Before using ElevenLabs TTS services, you need:

1. **ElevenLabs Account**: Sign up at [ElevenLabs](https://elevenlabs.io/app/sign-up)
2. **API Key**: Generate an API key from your account dashboard
3. **Voice Selection**: Choose voice IDs from the [voice library](https://elevenlabs.io/voice-library)

### Required Environment Variables

* `ELEVENLABS_API_KEY`: Your ElevenLabs API key for authentication


# Fish Audio
Source: https://docs.pipecat.ai/server/services/tts/fish

Real-time text-to-speech service using Fish Audio's WebSocket API

## Overview

`FishTTSService` provides real-time text-to-speech synthesis through Fish Audio's WebSocket-based streaming API. The service offers custom voice models, prosody controls, and multiple audio formats optimized for conversational AI applications with low latency.

<CardGroup>
  <Card title="Fish TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.fish.tts.html">
    Pipecat's API methods for Fish Audio TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07t-interruptible-fish.py">
    Complete example with custom voice model
  </Card>

  <Card title="Fish Audio Documentation" icon="book" href="https://docs.fish.audio/developer-platform/getting-started/introduction">
    Official Fish Audio documentation
  </Card>

  <Card title="Voice Models" icon="microphone" href="https://fish.audio/app/my-voices/">
    Create and manage custom voice models
  </Card>
</CardGroup>

## Installation

To use Fish Audio services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[fish]"
```

## Prerequisites

### Fish Audio Account Setup

Before using Fish Audio TTS services, you need:

1. **Fish Audio Account**: Sign up at [Fish Audio Console](https://console.fish.audio/)
2. **API Key**: Generate an API key from your account dashboard
3. **Voice Models**: Create or select custom voice models for synthesis

### Required Environment Variables

* `FISH_API_KEY`: Your Fish Audio API key for authentication


# Google
Source: https://docs.pipecat.ai/server/services/tts/google

Text-to-speech service using Google's Cloud Text-to-Speech API

## Overview

Google Cloud Text-to-Speech provides high-quality speech synthesis with two service implementations: `GoogleTTSService` (WebSocket-based) for streaming with the lowest latency, and `GoogleHttpTTSService` (HTTP-based) for simpler integration. `GoogleTTSService` is recommended for real-time applications.

<CardGroup>
  <Card title="Google TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.google.tts.html">
    Pipecat's API methods for Google Cloud TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07n-interruptible-google.py">
    Complete example with Chirp 3 HD voice
  </Card>

  <Card title="Google Cloud Documentation" icon="book" href="https://cloud.google.com/text-to-speech/docs">
    Official Google Cloud Text-to-Speech documentation
  </Card>

  <Card title="Voice Gallery" icon="microphone" href="https://cloud.google.com/text-to-speech/docs/voices">
    Browse available voices and languages
  </Card>
</CardGroup>

## Installation

To use Google services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[google]"
```

## Prerequisites

### Google Cloud Setup

Before using Google Cloud TTS services, you need:

1. **Google Cloud Account**: Sign up at [Google Cloud Console](https://console.cloud.google.com/)
2. **Project Setup**: Create a project and enable the Text-to-Speech API
3. **Service Account**: Create a service account with TTS permissions
4. **Authentication**: Set up credentials via service account key or Application Default Credentials

### Required Environment Variables

* `GOOGLE_APPLICATION_CREDENTIALS`: Path to your service account key file (recommended)
* Or use Application Default Credentials for cloud deployments


# Groq
Source: https://docs.pipecat.ai/server/services/tts/groq

Text-to-speech service implementation using Groq's TTS API

## Overview

`GroqTTSService` provides fast text-to-speech synthesis using Groq's TTS API with multiple voice options. The service operates at a fixed 48kHz sample rate and offers efficient audio streaming for real-time applications with ultra-low latency.

<CardGroup>
  <Card title="Groq TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.groq.tts.html">
    Pipecat's API methods for Groq TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07l-interruptible-groq.py">
    Complete example with Groq STT and LLM
  </Card>

  <Card title="Groq Documentation" icon="book" href="https://console.groq.com/docs/api-reference#models">
    Official Groq API documentation and models
  </Card>

  <Card title="Voice Options" icon="microphone" href="https://console.groq.com/">
    Explore available voice models and features
  </Card>
</CardGroup>

## Installation

To use Groq services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[groq]"
```

## Prerequisites

### Groq Account Setup

Before using Groq TTS services, you need:

1. **Groq Account**: Sign up at [Groq Console](https://console.groq.com/login)
2. **API Key**: Generate an API key from your account dashboard
3. **Voice Selection**: Choose from available voice models

### Required Environment Variables

* `GROQ_API_KEY`: Your Groq API key for authentication


# Hathora
Source: https://docs.pipecat.ai/server/services/tts/hathora

Text-to-speech service implementations hosted on Hathora

## Overview

Hathora is a hosting provider for several models for voice AI, which can be utilized under the single `HathoraTTSService`.

<CardGroup>
  <Card title="Hathora TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.hahotra.tts.html">
    Pipecat's API methods for Hathora-hosted TTS models
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07ag-interruptible-hathora.py">
    Complete example using Hathora-hosted models
  </Card>

  <Card title="Hathora Models Documentation" icon="book" href="https://models.hathora.dev/">
    Official Hathora documentation and features
  </Card>
</CardGroup>

## Installation

To use Hathora services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[hathora]"
```

## Prerequisites

### Hathora Account Setup

Before using Hathora STT services, you need:

1. **Hathora Account**: Sign up at [Hathora Models Console](https://models.hathora.dev/)
2. **API Key**: Generate an API token from your [Tokens page](https://models.hathora.dev/tokens)

### Hathora Model Specifier

The `HathoraTTSService` accepts a `model: str` parameter which corresponds to the model you would like to use.

You can find available specifiers [here](https://models.hathora.dev/)


# Hume
Source: https://docs.pipecat.ai/server/services/tts/hume

Text-to-speech service using Hume AI's expressive Octave models with word timestamps

## Overview

Hume provides expressive text-to-speech synthesis using their Octave models, which adapt pronunciation, pitch, speed, and emotional style based on context. `HumeTTSService` offers real-time streaming with word-level timestamps, custom voice support, and advanced synthesis controls including acting instructions, speed adjustment, and trailing silence configuration.

<CardGroup>
  <Card title="Hume TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.hume.tts.html">
    Pipecat's API methods for Hume TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07ze-interruptible-hume.py">
    Complete example with word timestamps and interruption handling
  </Card>

  <Card title="Hume Documentation" icon="book" href="https://dev.hume.ai/docs/text-to-speech-tts/overview">
    Official Hume TTS API documentation and features
  </Card>

  <Card title="Voice Library" icon="microphone" href="https://dev.hume.ai/docs/text-to-speech-tts/voices">
    Browse and manage available voices
  </Card>
</CardGroup>

## Installation

To use Hume services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[hume]"
```

## Prerequisites

### Hume Account Setup

Before using Hume TTS services, you need:

1. **Hume Account**: Sign up at [Hume AI](https://www.hume.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Voice Selection**: Choose voice IDs from the voice library or create custom voices

### Required Environment Variables

* `HUME_API_KEY`: Your Hume API key for authentication


# Inworld
Source: https://docs.pipecat.ai/server/services/tts/inworld

Text-to-speech service using Inworld AI's TTS APIs

## Overview

Inworld provides high-quality, low-latency speech synthesis via two implementation types: `InworldTTSService` for real-time, minimal-latency use-cases through websockets and `InworldHttpTTSService` for streaming and non-streaming use-cases over HTTP. Featuring support for 12+ languages, timestamps, custom pronunciation and instant voice cloning.

<CardGroup>
  <Card title="Inworld TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.inworld.tts.html">
    Pipecat's API methods for Inworld TTS integration
  </Card>

  <Card title="Example Implementation (Websockets)" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07zb-interruptible-inworld.py">
    Complete example with Inworld TTS
  </Card>

  <Card title="Inworld Documentation" icon="book" href="https://docs.inworld.ai/docs/tts/tts">
    Official Inworld TTS API documentation
  </Card>

  <Card title="Inworld Portal" icon="microphone" href="https://platform.inworld.ai/">
    Create and manage voice models
  </Card>
</CardGroup>

## Installation

To use Inworld services, no additional dependencies are required beyond the base installation:

```bash theme={null}
pip install "pipecat-ai"
```

## Prerequisites

### Inworld Account Setup

Before using Inworld TTS services, you need:

1. **Inworld Account**: Sign up at [Inworld Studio](https://studio.inworld.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Voice Selection**: Choose from available voice models

### Required Environment Variables

* `INWORLD_API_KEY`: Your Inworld API key for authentication


# LMNT
Source: https://docs.pipecat.ai/server/services/tts/lmnt

Text-to-speech service implementation using LMNT's streaming API

## Overview

`LMNTTTSService` provides real-time text-to-speech synthesis through LMNT's WebSocket-based streaming API optimized for conversational AI. The service offers ultra-low latency with high-quality voice models and supports multiple languages with automatic interruption handling.

<CardGroup>
  <Card title="LMNT TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.lmnt.tts.html">
    Pipecat's API methods for LMNT TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07k-interruptible-lmnt.py">
    Complete example with voice synthesis
  </Card>

  <Card title="LMNT Documentation" icon="book" href="https://docs.lmnt.com/api-reference/speech/streaming">
    Official LMNT streaming speech API documentation
  </Card>

  <Card title="Voice Library" icon="microphone" href="https://app.lmnt.com/">
    Browse and create custom voices
  </Card>
</CardGroup>

## Installation

To use LMNT services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[lmnt]"
```

## Prerequisites

### LMNT Account Setup

Before using LMNT TTS services, you need:

1. **LMNT Account**: Sign up at [LMNT Console](https://app.lmnt.com/)
2. **API Key**: Generate an API key from your account dashboard
3. **Voice Selection**: Choose from available voice models or create custom voices

### Required Environment Variables

* `LMNT_API_KEY`: Your LMNT API key for authentication


# MiniMax
Source: https://docs.pipecat.ai/server/services/tts/minimax

Text-to-speech service implementation using MiniMax T2A API

## Overview

`MiniMaxTTSService` provides high-quality text-to-speech synthesis using MiniMax's T2A (Text-to-Audio) API with streaming capabilities, emotional voice control, and support for multiple languages. The service offers various models optimized for different use cases, from low-latency to high-definition audio quality.

<CardGroup>
  <Card title="MiniMax TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.minimax.tts.html">
    Pipecat's API methods for MiniMax TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07y-interruptible-minimax.py">
    Complete example with emotional voice settings
  </Card>

  <Card title="MiniMax Documentation" icon="book" href="https://www.minimax.io/platform/document/T2A%20V2?key=66719005a427f0c8a5701643">
    Official MiniMax T2A API documentation
  </Card>

  <Card title="MiniMax Platform" icon="microphone" href="https://www.minimax.io/platform/">
    Access voice models and API credentials
  </Card>
</CardGroup>

## Installation

To use MiniMax services, no additional dependencies are required beyond the base installation:

```bash theme={null}
pip install "pipecat-ai"
```

## Prerequisites

### MiniMax Account Setup

Before using MiniMax TTS services, you need:

1. **MiniMax Account**: Sign up at [MiniMax Platform](https://www.minimax.io/platform/)
2. **API Credentials**: Get your API key and Group ID from the platform
3. **Voice Selection**: Choose from available voice models and emotional settings

### Required Environment Variables

* `MINIMAX_API_KEY`: Your MiniMax API key for authentication
* `MINIMAX_GROUP_ID`: Your MiniMax group ID


# Neuphonic
Source: https://docs.pipecat.ai/server/services/tts/neuphonic

Text-to-speech service implementation using Neuphonic's API

## Overview

Neuphonic provides high-quality text-to-speech synthesis with two service implementations: `NeuphonicTTSService` (WebSocket-based) with real-time streaming and interruption support, and `NeuphonicHttpTTSService` (HTTP-based) with server-sent events. `NeuphonicTTSService` is recommended for interactive applications requiring low latency.

<CardGroup>
  <Card title="Neuphonic TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.neuphonic.tts.html">
    Pipecat's API methods for Neuphonic TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07v-interruptible-neuphonic.py">
    Complete example with WebSocket streaming
  </Card>

  <Card title="Neuphonic Documentation" icon="book" href="https://docs.neuphonic.com/api-reference/tts/websocket">
    Official Neuphonic TTS API documentation
  </Card>

  <Card title="Voice Library" icon="microphone" href="https://docs.neuphonic.com/">
    Browse available voices and features
  </Card>
</CardGroup>

## Installation

To use Neuphonic services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[neuphonic]"
```

## Prerequisites

### Neuphonic Account Setup

Before using Neuphonic TTS services, you need:

1. **Neuphonic Account**: Sign up at [Neuphonic](https://docs.neuphonic.com/)
2. **API Key**: Generate an API key from your account dashboard
3. **Voice Selection**: Choose from available voice models

### Required Environment Variables

* `NEUPHONIC_API_KEY`: Your Neuphonic API key for authentication


# OpenAI
Source: https://docs.pipecat.ai/server/services/tts/openai

Text-to-speech service using OpenAI's TTS API

## Overview

`OpenAITTSService` provides high-quality text-to-speech synthesis using OpenAI's TTS API with multiple voice models including traditional TTS models and advanced GPT-based models. The service outputs 24kHz PCM audio with streaming capabilities for real-time applications.

<CardGroup>
  <Card title="OpenAI TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.openai.tts.html">
    Pipecat's API methods for OpenAI TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07g-interruptible-openai.py">
    Complete example with voice customization
  </Card>

  <Card title="OpenAI Documentation" icon="book" href="https://platform.openai.com/docs/api-reference/audio/createSpeech">
    Official OpenAI TTS API documentation
  </Card>

  <Card title="Voice Samples" icon="microphone" href="https://platform.openai.com/docs/guides/text-to-speech/voice-options">
    Listen to available voice options
  </Card>
</CardGroup>

## Installation

To use OpenAI services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[openai]"
```

## Prerequisites

### OpenAI Account Setup

Before using OpenAI TTS services, you need:

1. **OpenAI Account**: Sign up at [OpenAI Platform](https://platform.openai.com/)
2. **API Key**: Generate an API key from your [API keys page](https://platform.openai.com/api-keys)
3. **Voice Selection**: Choose from available voice options (alloy, echo, fable, onyx, nova, shimmer, etc.)

### Required Environment Variables

* `OPENAI_API_KEY`: Your OpenAI API key for authentication


# Piper
Source: https://docs.pipecat.ai/server/services/tts/piper

Text-to-speech service implementation using the Piper TTS server

## Overview

`PiperTTSService` provides high-quality neural text-to-speech synthesis through a self-hosted HTTP server. The service offers complete privacy and control with no external API dependencies, making it ideal for on-premise deployments and applications requiring data sovereignty.

<CardGroup>
  <Card title="Piper TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.piper.tts.html">
    Pipecat's API methods for Piper TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/tree/main/examples">
    Browse examples using Piper TTS
  </Card>

  <Card title="Piper Documentation" icon="book" href="https://github.com/OHF-Voice/piper1-gpl">
    Official Piper TTS documentation and setup
  </Card>

  <Card title="HTTP Server Setup" icon="server" href="https://github.com/OHF-Voice/piper1-gpl/blob/main/docs/API_HTTP.md">
    Configure Piper HTTP server for Pipecat
  </Card>
</CardGroup>

## Installation

To use Piper services, no additional Pipecat dependencies are required:

```bash theme={null}
pip install "pipecat-ai"  # Base installation is sufficient
```

## Prerequisites

### Piper Server Setup

Before using PiperTTSService, you need:

1. **Piper TTS Server**: Set up a running Piper TTS server following the [HTTP server documentation](https://github.com/OHF-Voice/piper1-gpl/blob/main/docs/API_HTTP.md)
2. **Voice Models**: Download and configure voice models for your target languages
3. **Server Configuration**: Configure server endpoint and voice selection

### Required Configuration

* **Server URL**: Configure the Piper server endpoint in your service initialization
* **Voice Models**: Ensure required voice models are available on the server

<Tip>
  Piper runs entirely locally, providing complete privacy and eliminating API
  key requirements.
</Tip>


# PlayHT
Source: https://docs.pipecat.ai/server/services/tts/playht

Text-to-speech services using PlayHT's WebSocket and HTTP APIs

## Overview

PlayHT provides high-quality text-to-speech synthesis with two service implementations: `PlayHTTTSService` (WebSocket-based) for real-time streaming with ultra-low latency, and `PlayHTHttpTTSService` (HTTP-based) for simpler synthesis. `PlayHTTTSService` is recommended for interactive applications requiring low latency and advanced interruption handling.

<CardGroup>
  <Card title="PlayHT TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.playht.tts.html">
    Pipecat's API methods for PlayHT TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07e-interruptible-playht.py">
    Complete example with voice cloning
  </Card>

  <Card title="PlayHT Documentation" icon="book" href="https://docs.play.ht/reference/playht-tts-websocket-api">
    Official PlayHT WebSocket API documentation
  </Card>

  <Card title="Voice Cloning" icon="microphone" href="https://play.ht/app">
    Create and manage custom voices
  </Card>
</CardGroup>

## Installation

To use PlayHT services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[playht]"
```

## Prerequisites

### PlayHT Account Setup

Before using PlayHT TTS services, you need:

1. **PlayHT Account**: Sign up at [PlayHT Dashboard](https://play.ht/app)
2. **API Credentials**: Get your User ID and API Key from the dashboard
3. **Voice Selection**: Choose from standard voices or create custom cloned voices

### Required Environment Variables

* `PLAY_HT_USER_ID`: Your PlayHT user ID
* `PLAY_HT_API_KEY`: Your PlayHT API key for authentication


# Resemble AI
Source: https://docs.pipecat.ai/server/services/tts/resemble

Text-to-speech service using Resemble AI's WebSocket streaming API with word-level timing

## Overview

`ResembleAITTSService` provides high-quality text-to-speech synthesis using Resemble AI's streaming WebSocket API with word-level timestamps and audio context management for handling multiple simultaneous synthesis requests with proper interruption support.

<CardGroup>
  <Card title="Resemble AI TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.resembleai.tts.html">
    Pipecat's API methods for Resemble AI TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07zk-interruptible-resemble.py">
    Complete example with interruption handling
  </Card>

  <Card title="Resemble AI Documentation" icon="book" href="https://docs.resemble.ai/">
    Official Resemble AI API documentation
  </Card>

  <Card title="Sign up" icon="user" href="https://app.resemble.ai/">
    Sign up for a Resemble AI account
  </Card>
</CardGroup>

## Installation

To use Resemble AI services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[resemble]"
```

## Prerequisites

### Resemble AI Account Setup

Before using Resemble AI TTS services, you need:

1. **Resemble AI Account**: Sign up at [Resemble AI](https://app.resemble.ai)
2. **API Key**: Generate an API key from your [account settings](https://app.resemble.ai/account/api)
3. **Voice Selection**: Choose or create voice UUIDs from your [voice library](https://app.resemble.ai/hub/voices)

### Required Environment Variables

* `RESEMBLE_API_KEY`: Your Resemble AI API key for authentication


# Rime
Source: https://docs.pipecat.ai/server/services/tts/rime

Text-to-speech service implementations using Rime AI

## Overview

Rime AI provides two TTS service implementations: `RimeTTSService` (WebSocket-based) with word-level timing and interruption support, and `RimeHttpTTSService` (HTTP-based) for simpler use cases. `RimeTTSService` is recommended for real-time interactive applications.

<CardGroup>
  <Card title="Rime TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.rime.tts.html">
    Pipecat's API methods for Rime TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07q-interruptible-rime.py">
    Complete example with word timestamps
  </Card>

  <Card title="Rime Documentation" icon="book" href="https://docs.rime.ai/api-reference/endpoint/websockets-json">
    Official Rime WebSocket and HTTP API documentation
  </Card>

  <Card title="Voice Models" icon="microphone" href="https://docs.rime.ai/">
    Explore available voice models and features
  </Card>
</CardGroup>

## Installation

To use Rime services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[rime]"
```

## Prerequisites

### Rime Account Setup

Before using Rime TTS services, you need:

1. **Rime Account**: Sign up at [Rime AI](https://docs.rime.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Voice Selection**: Choose from available voice models

### Required Environment Variables

* `RIME_API_KEY`: Your Rime API key for authentication

## Customizing Speech

`RimeTTSService` provides a set of helper methods for implementing Rime-specific customizations, meant to be used as part of text transformers. These include methods for spelling out text, adjusting speech rate, and modifying pitch. See the [Text Transformers for TTS](/guides/learn/text-to-speech#text-transformers-for-tts) section in the Text-to-Speech guide for usage examples.

### SPELL(text: str) -> str:

Implements [Rime's spell function](https://docs.rime.ai/api-reference/spell) to spell out text character by character.

```python theme={null}
# Text transformers for TTS
# This will insert Rime's spell tags around the provided text.
async def spell_out_text(text: str, type: str) -> str:
    return RimeTTSService.SPELL(text)

tts = RimeTTSService(
    api_key=os.getenv("RIME_API_KEY"),
    text_transforms=[
        ("phone_number", spell_out_text),
    ],
)
```

### PAUSE\_TAG(seconds: float) -> str:

Implements [Rime's custom pause functionality](https://docs.rime.ai/api-reference/custom-pauses) to generate a properly formatted pause tag you can insert into the text.

```python theme={null}
# Text transformers for TTS
# This will insert a one second pause after questions.
async def pause_after_questions(text: str, type: str) -> str:
    if text.endswith("?"):
        return f"{text}{RimeTTSService.PAUSE_TAG(1.0)}"
    return text

tts = RimeTTSService(
    api_key=os.getenv("RIME_API_KEY"),
    text_transforms=[
        ("sentence", pause_after_questions), # Only apply to sentence aggregations
    ],
)
```

### PRONOUNCE(self, text: str, word: str, phoneme: str) -> str:

Convenience method to support Rime's [custom pronunciations feature](https://docs.rime.ai/api-reference/custom-pronunciation). It takes a word and its desired phoneme representation, returning the text with the provided word replaced by the appropriate phoneme tag.

```python theme={null}
# Text transformers for TTS
# This will a phoneme in place of the word "potato" to define how it
# should be pronounced.
async def maybe_say_potato_all_fancylike(text: str, type: str) -> str:
    if using_fancy_voice:
        return RimeTTSService.PRONOUNCE(text, "potato", "potato")
    else:
        return RimeTTSService.PRONOUNCE(text, "potato", "poteto")

tts = RimeTTSService(
    api_key=os.getenv("RIME_API_KEY"),
    text_transforms=[
        ("*", maybe_say_potato_all_fancylike), # Apply to all text
    ],
)
```

### INLINE\_SPEED(self, text: str, speed: float) -> str:

A convenience method to support Rime's [inline speed adjustment feature](https://docs.rime.ai/api-reference/speed). It will wrap the provided text in the `[]` tags and add the provided speed to the `inlineSpeedAlpha` field in the request metadata.

```python theme={null}
# Text transformers for TTS
# This will make the word slow always be spoken more slowly.
async def slow_down_slow_words(text: str, type: str) -> str:
    return text.replace(
        "slow",
        RimeTTSService.INLINE_SPEED("slow", speed=0.5)
    )

tts = RimeTTSService(
    api_key=os.getenv("RIME_API_KEY"),
    text_transforms=[
        ("*", slow_down_slow_words), # Apply to all text
    ],
)
```


# NVIDIA Riva
Source: https://docs.pipecat.ai/server/services/tts/riva

Text-to-speech service implementation using NVIDIA Riva

## Overview

`NvidiaTTSService` provides high-quality text-to-speech synthesis through NVIDIA Riva's cloud-based AI models accessible via gRPC API. The service offers multilingual support, configurable quality settings, and streaming audio generation optimized for real-time applications.

<CardGroup>
  <Card title="NVIDIA Riva TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.riva.tts.html">
    Pipecat's API methods for NVIDIA Riva TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07r-interruptible-riva-nim.py">
    Complete example with Riva NIM
  </Card>

  <Card title="NVIDIA Riva Documentation" icon="book" href="https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tts/tts-overview.html">
    Official NVIDIA Riva TTS documentation
  </Card>

  <Card title="NVIDIA Developer Portal" icon="microphone" href="https://developer.nvidia.com/">
    Access API keys and Riva services
  </Card>
</CardGroup>

## Installation

To use NVIDIA Riva services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[nvidia]"
```

## Prerequisites

### NVIDIA Riva Setup

Before using Riva TTS services, you need:

1. **NVIDIA Developer Account**: Sign up at [NVIDIA Developer Portal](https://developer.nvidia.com/)
2. **API Key**: Generate an NVIDIA API key for Riva services
3. **Riva Access**: Ensure access to NVIDIA Riva TTS services

### Required Environment Variables

* `NVIDIA_API_KEY`: Your NVIDIA API key for authentication


# Sarvam AI
Source: https://docs.pipecat.ai/server/services/tts/sarvam

Text-to-speech service implementation using Sarvam AI's TTS API

## Overview

`SarvamTTSService` provides text-to-speech synthesis specialized for Indian languages and voices. The service offers extensive voice customization options including pitch, pace, and loudness control, with support for multiple Indian languages and preprocessing for mixed-language content.

<CardGroup>
  <Card title="Sarvam TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.sarvam.tts.html">
    Pipecat's API methods for Sarvam AI TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07z-interruptible-sarvam.py">
    Complete example with Indian language support
  </Card>

  <Card title="Sarvam Documentation" icon="book" href="https://docs.sarvam.ai/api-reference-docs/text-to-speech/convert">
    Official Sarvam AI text-to-speech API documentation
  </Card>

  <Card title="Sarvam Console" icon="microphone" href="https://www.sarvam.ai/">
    Access Indian language voices and API keys
  </Card>
</CardGroup>

## Installation

To use Sarvam AI services, no additional dependencies are required beyond the base installation:

```bash theme={null}
pip install "pipecat-ai"
```

## Prerequisites

### Sarvam AI Account Setup

Before using Sarvam AI TTS services, you need:

1. **Sarvam AI Account**: Sign up at [Sarvam AI Console](https://www.sarvam.ai/)
2. **API Key**: Generate an API key from your account dashboard
3. **Language Selection**: Choose from available Indian language voices

### Required Environment Variables

* `SARVAM_API_KEY`: Your Sarvam AI API key for authentication


# Speechmatics
Source: https://docs.pipecat.ai/server/services/tts/speechmatics

Text-to-speech service using Speechmatics TTS API

## Overview

`SpeechmaticsTTSService` provides production-grade, low-latency synthesis optimized for telephony and voice agents. By streaming 16kHz mono audio, it ensures bandwidth efficiency and prioritizes pronunciation accuracy for natural, uninterrupted conversations at scale.

<CardGroup>
  <Card title="Speechmatics TTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.speechmatics.tts.html">
    Pipecat's API methods for Speechmatics TTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07a-interruptible-speechmatics.py">
    Complete example with Speechmatics TTS
  </Card>

  <Card title="Speechmatics Documentation" icon="book" href="https://docs.speechmatics.com/text-to-speech/quickstart">
    Official Speechmatics TTS API documentation
  </Card>

  <Card title="Speechmatics Portal" icon="microphone" href="https://portal.speechmatics.com">
    Browse and test available voices
  </Card>
</CardGroup>

## Installation

To use Speechmatics services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[speechmatics]"
```

## Prerequisites

### Speechmatics Account Setup

Before using Speechmatics TTS services, you need:

1. **Speechmatics Account**: Sign up at [Speechmatics Portal](https://portal.speechmatics.com)
2. **API Key**: Generate an API key from your dashboard
3. **Voice Selection**: Choose from available [voices](https://docs.speechmatics.com/text-to-speech/quickstart#voices)

### Required Environment Variables

* `SPEECHMATICS_API_KEY`: Your Speechmatics API key for authentication

### Basic Usage

The Speechmatics TTS service requires an `aiohttp` ClientSession to manage high-performance network requests.

```python theme={null}
import os
import aiohttp
from pipecat.services.speechmatics.tts import SpeechmaticsTTSService

async def create_tts_service():
    # create a session (usually done within your pipeline setup)
    async with aiohttp.ClientSession() as session:
        tts = SpeechmaticsTTSService(
            api_key=os.getenv("SPEECHMATICS_API_KEY"),
            voice_id="sarah", 
            aiohttp_session=session,
        )
        # ... add tts to your pipeline ...
```


# XTTS
Source: https://docs.pipecat.ai/server/services/tts/xtts

Text-to-speech service implementation using Coqui's XTTS streaming server

<Warning>
  Coqui, the XTTS maintainer, has shut down. XTTS may not receive future updates
  or support.
</Warning>

## Overview

`XTTSTTSService` provides multilingual voice synthesis with voice cloning capabilities through a locally hosted streaming server. The service supports real-time streaming and custom voice training using Coqui's XTTS-v2 model for cross-lingual text-to-speech.

<CardGroup>
  <Card title="XTTS API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.xtts.tts.html">
    Pipecat's API methods for XTTS integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07i-interruptible-xtts.py">
    Complete example with voice cloning
  </Card>

  <Card title="XTTS Repository" icon="book" href="https://github.com/coqui-ai/xtts-streaming-server">
    Official XTTS streaming server repository
  </Card>

  <Card title="Voice Cloning" icon="microphone" href="https://github.com/coqui-ai/xtts-streaming-server#voice-cloning">
    Learn about custom voice training
  </Card>
</CardGroup>

## Installation

XTTS requires a running streaming server. Start the server using Docker:

```bash theme={null}
docker run --gpus=all -e COQUI_TOS_AGREED=1 --rm -p 8000:80 \
  ghcr.io/coqui-ai/xtts-streaming-server:latest-cuda121
```

## Prerequisites

### XTTS Server Setup

Before using XTTSTTSService, you need:

1. **Docker Environment**: Set up Docker with GPU support for optimal performance
2. **XTTS Server**: Run the XTTS streaming server container
3. **Voice Models**: Configure voice models and cloning samples as needed

### Required Configuration

* **Server URL**: Configure the XTTS server endpoint (default: `http://localhost:8000`)
* **Voice Selection**: Set up voice models or voice cloning samples

<Note>
  GPU acceleration is recommended for optimal performance. The server requires
  CUDA support for best results.
</Note>


# HeyGen
Source: https://docs.pipecat.ai/server/services/video/heygen

AI avatar video generation service for creating interactive conversational avatars

## Overview

`HeyGenVideoService` integrates with HeyGen [LiveAvatar](https://www.liveavatar.com/) to create interactive AI-powered video avatars that respond naturally in real-time conversations. The service handles bidirectional audio/video streaming, avatar animations, voice activity detection, and conversation interruptions to deliver engaging conversational AI experiences with lifelike visual presence.

<CardGroup>
  <Card title="HeyGen Video API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.heygen.video.html">
    Pipecat's API methods for HeyGen video integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/43a-heygen-video-service.py">
    Complete example with interactive avatar
  </Card>

  <Card title="HeyGen Documentation" icon="book" href="https://docs.liveavatar.com/docs/getting-started">
    Official HeyGen API documentation and guides
  </Card>

  <Card title="HeyGen Platform" icon="microphone" href="https://www.liveavatar.com/">
    Access interactive avatars and API keys
  </Card>
</CardGroup>

## Installation

To use HeyGen services, install the required dependency:

```bash theme={null}
pip install "pipecat-ai[heygen]"
```

## Prerequisites

### HeyGen Account Setup

Before using HeyGen video services, you need:

1. **HeyGen Account**: Sign up at [HeyGen Platform](https://app.liveavatar.com/signin)
2. **API Key**: Generate an API key from your account dashboard
3. **Avatar Selection**: Choose from available interactive avatars
4. **Streaming Setup**: Configure real-time avatar streaming capabilities

### Required Environment Variables

* `HEYGEN_LIVE_AVATAR_API_KEY`: Your HeyGen LiveAvatar API key for authentication


# Simli
Source: https://docs.pipecat.ai/server/services/video/simli

Real-time AI avatar video generation service using WebRTC streaming

## Overview

`SimliVideoService` integrates with Simli to create real-time AI avatar video experiences using WebRTC streaming. The service processes audio input to generate synchronized avatar video and audio output, handling real-time streaming, audio resampling, and conversation interruptions for engaging conversational AI applications.

<CardGroup>
  <Card title="Simli Video API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.simli.video.html">
    Pipecat's API methods for Simli video integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/27-simli-layer.py">
    Complete example with avatar streaming
  </Card>

  <Card title="Simli Documentation" icon="book" href="https://docs.simli.com/overview">
    Official Simli API documentation and guides
  </Card>

  <Card title="Simli Platform" icon="microphone" href="https://www.simli.com/">
    Access avatar faces and manage API keys
  </Card>
</CardGroup>

## Installation

To use Simli services, install the required dependency:

```bash theme={null}
pip install "pipecat-ai[simli]"
```

## Prerequisites

### Simli Account Setup

Before using Simli video services, you need:

1. **Simli Account**: Sign up at [Simli Platform](https://www.simli.com/)
2. **API Key**: Generate an API key from your account dashboard
3. **Face Selection**: Choose or create avatar faces for video generation
4. **WebRTC Setup**: Configure real-time streaming capabilities

### Required Environment Variables

* `SIMLI_API_KEY`: Your Simli API key for authentication
* `SIMLI_FACE_ID`: ID of your avatar face


# Tavus
Source: https://docs.pipecat.ai/server/services/video/tavus

AI avatar video generation service for creating realistic talking avatars

## Overview

`TavusVideoService` integrates with Tavus to generate AI-powered video avatars that speak your text-to-speech output in real-time. The service takes audio input and produces synchronized video of a realistic avatar speaking, enabling engaging conversational AI experiences with visual presence.

<CardGroup>
  <Card title="Tavus Video API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.tavus.video.html">
    Pipecat's API methods for Tavus video integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/21a-tavus-video-service.py">
    Complete example with avatar video generation
  </Card>

  <Card title="Tavus Documentation" icon="book" href="https://docs.tavus.io/sections/replica/overview">
    Official Tavus replica and avatar documentation
  </Card>

  <Card title="Tavus Platform" icon="microphone" href="https://platform.tavus.io/auth/sign-up?plan=free">
    Create avatars and manage API keys
  </Card>
</CardGroup>

## Installation

To use Tavus services, install the required dependency:

```bash theme={null}
pip install "pipecat-ai[tavus]"
```

## Prerequisites

### Tavus Account Setup

Before using Tavus video services, you need:

1. **Tavus Account**: Sign up at [Tavus Platform](https://platform.tavus.io/auth/sign-up?plan=free)
2. **API Key**: Generate an API key from your account dashboard
3. **Replica Creation**: Create and train voice replicas for your avatars
4. **Avatar Selection**: Choose or create avatar models for video generation

### Required Environment Variables

* `TAVUS_API_KEY`: Your Tavus API key for authentication
* `TAVUS_REPLICA_ID`: ID of your trained voice replica


# Moondream
Source: https://docs.pipecat.ai/server/services/vision/moondream

Vision service implementation using Moondream for local image analysis and question answering

## Overview

`MoondreamService` provides local image analysis and question-answering capabilities using the Moondream model. It runs entirely on your local machine, supporting various hardware acceleration options including CUDA, Intel XPU, and Apple MPS for privacy-focused computer vision applications.

<CardGroup>
  <Card title="Moondream Vision API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.services.moondream.vision.html">
    Pipecat's API methods for Moondream vision integration
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/12-describe-video.py">
    Browse examples using Moondream vision
  </Card>

  <Card title="Moondream Documentation" icon="book" href="https://www.moondream.ai/">
    Official Moondream model documentation
  </Card>

  <Card title="Hugging Face Model" icon="microphone" href="https://huggingface.co/vikhyatk/moondream2">
    Access Moondream model on Hugging Face
  </Card>
</CardGroup>

## Installation

To use Moondream services, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[moondream]"
```

## Prerequisites

### Local Model Setup

Before using Moondream vision services, you need:

1. **Model Download**: First run will automatically download the Moondream model from Hugging Face
2. **Hardware Configuration**: Set up CUDA, Intel XPU, or Apple MPS for optimal performance
3. **Storage Space**: Ensure sufficient disk space for model files
4. **Memory Requirements**: Adequate RAM/VRAM for model inference

### Hardware Acceleration

The service automatically detects and uses the best available hardware:

* **Intel XPU**: Requires intel\_extension\_for\_pytorch
* **NVIDIA CUDA**: For GPU acceleration
* **Apple Metal (MPS)**: For Apple Silicon optimization
* **CPU**: Fallback option for any system

### Configuration Options

* **Model Selection**: Choose Moondream model version and revision
* **Hardware Override**: Force CPU usage if needed
* **Local Processing**: Complete privacy with no external API calls

<Tip>
  No API keys required - Moondream runs entirely locally for complete privacy
  and control.
</Tip>


# AICFilter
Source: https://docs.pipecat.ai/server/utilities/audio/aic-filter

Speech enhancement using ai-coustics' SDK

## Overview

`AICFilter` is an audio processor that enhances user speech by reducing background noise and improving speech clarity. It inherits from `BaseAudioFilter` and processes audio frames in real-time using ai-coustics' speech enhancement technology.

To use AIC, you need a license key. Get started at [ai-coustics.com](https://ai-coustics.com/pipecat).

<Note>
  This documentation covers **aic-sdk v2.x**. If you're using aic-sdk v1.x, please see the [Migration Guide](#migration-guide-v1-to-v2) section below for upgrading instructions.
</Note>

## Installation

The AIC filter requires additional dependencies:

```bash theme={null}
pip install "pipecat-ai[aic]"
```

## Constructor Parameters

<ParamField type="str">
  ai-coustics license key for authentication. Get your key at [developers.ai-coustics.io](https://developers.ai-coustics.io).
</ParamField>

<ParamField type="Optional[str]">
  Model identifier to download from CDN. Required if `model_path` is not provided.
  See [artifacts.ai-coustics.io](https://artifacts.ai-coustics.io/) for available models.
  See the [documentation](https://docs.ai-coustics.com/guides/models) for more detailed information about the models.

  Examples: `"quail-vf-l-16khz"`, `"quail-s-16khz"`, `"quail-l-8khz"`
</ParamField>

<ParamField type="Optional[str]">
  Path to a local `.aicmodel` file. If provided, `model_id` is ignored and no download occurs.
  Useful for offline deployments or custom models.
</ParamField>

<ParamField type="Optional[Path]">
  Directory for downloading and caching models. Defaults to a cache directory in the user's home folder.
</ParamField>

## Methods

### create\_vad\_analyzer

Creates an `AICVADAnalyzer` that uses the AIC model's built-in voice activity detection.

```python theme={null}
def create_vad_analyzer(
    *,
    speech_hold_duration: Optional[float] = None,
    minimum_speech_duration: Optional[float] = None,
    sensitivity: Optional[float] = None,
) -> AICVADAnalyzer
```

#### VAD Parameters

<ParamField type="Optional[float]">
  Controls for how long the VAD continues to detect speech after the audio signal no longer contains speech (in seconds).
  Range: `0.0` to `100x model window length`, Default (in SDK): `0.05s`
</ParamField>

<ParamField type="Optional[float]">
  Controls for how long speech needs to be present in the audio signal before the VAD considers it speech (in seconds).
  Range: `0.0` to `1.0`, Default (in SDK): `0.0s`
</ParamField>

<ParamField type="Optional[float]">
  Controls the sensitivity (energy threshold) of the VAD. This value is used by the VAD as the threshold a speech audio signal's energy has to exceed in order to be considered speech.
  Formula: `Energy threshold = 10 ** (-sensitivity)`
  Range: `1.0` to `15.0`, Default (in SDK): `6.0`
</ParamField>

### get\_vad\_context

Returns the VAD context once the processor is initialized. Can be used to dynamically adjust VAD parameters at runtime.

```python theme={null}
vad_ctx = aic_filter.get_vad_context()
vad_ctx.set_parameter(VadParameter.Sensitivity, 8.0)
```

## Input Frames

<ParamField type="Frame">
  Specific control frame to toggle filtering on/off

  ```python theme={null}
  from pipecat.frames.frames import FilterEnableFrame

  # Disable speech enhancement
  await task.queue_frame(FilterEnableFrame(False))

  # Re-enable speech enhancement
  await task.queue_frame(FilterEnableFrame(True))
  ```
</ParamField>

## Usage Examples

### Basic Usage with AIC VAD

The recommended approach is to use `AICFilter` with its built-in VAD analyzer:

```python theme={null}
from pipecat.audio.filters.aic_filter import AICFilter
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.transports.services.daily import DailyTransport, DailyParams

# Create the AIC filter
aic_filter = AICFilter(
    license_key=os.environ["AIC_SDK_LICENSE"],
    model_id="quail-vf-l-16khz",
)

# Use AIC's integrated VAD
transport = DailyTransport(
    room_url,
    token,
    "Bot",
    DailyParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
        audio_in_filter=aic_filter,
    ),
)

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        vad_analyzer=aic_filter.create_vad_analyzer(
            speech_hold_duration=0.05,
            minimum_speech_duration=0.0,
            sensitivity=6.0,
        ),
    ),
)
```

### Using a Local Model

For offline deployments or when you want to manage model files yourself:

```python theme={null}
from pipecat.audio.filters.aic_filter import AICFilter

aic_filter = AICFilter(
    license_key=os.environ["AIC_SDK_LICENSE"],
    model_path="/path/to/your/model.aicmodel",
)
```

### Custom Cache Directory

Specify a custom directory for model downloads:

```python theme={null}
from pipecat.audio.filters.aic_filter import AICFilter

aic_filter = AICFilter(
    license_key=os.environ["AIC_SDK_LICENSE"],
    model_id="quail-s-16khz",
    model_download_dir="/opt/aic-models",
)
```

### With Other Transports

The AIC filter works with any Pipecat transport:

```python theme={null}
from pipecat.audio.filters.aic_filter import AICFilter
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.transports.websocket import FastAPIWebsocketTransport, FastAPIWebsocketParams

aic_filter = AICFilter(
    license_key=os.environ["AIC_SDK_LICENSE"],
    model_id="quail-vf-l-16khz",
)

transport = FastAPIWebsocketTransport(
    params=FastAPIWebsocketParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
        audio_in_filter=aic_filter,
    ),
)

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        vad_analyzer=aic_filter.create_vad_analyzer(
            speech_hold_duration=0.05,
            sensitivity=6.0,
        ),
    ),
)
```

<Info>
  See the [AIC filter example](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07zd-interruptible-aicoustics.py) for a complete working example.
</Info>

## Models

For detailed information about the available models, take a look at the [Models documentation](https://docs.ai-coustics.com/guides/models).

## Audio Flow

```mermaid theme={null}
graph TD
    A[AudioRawFrame] --> B[AICFilter]
    B --> C[AICVADAnalyzer]
    C --> D[STT]
```

The AIC filter enhances audio before it reaches the VAD and STT stages, improving transcription accuracy in noisy environments.

## Migration Guide (v1 to v2)

<Note>
  For the complete aic-sdk migration guide including all API changes, see the official [Python 1.3 to 2.0 Migration Guide](https://docs.ai-coustics.com/guides/migrations/python-1-3-to-2-0#quick-migration-checklist).
</Note>

### Migration Steps

1. Update Pipecat to the latest version (aic-sdk v2.x is included automatically).
2. Remove deprecated constructor parameters (`model_type`, `enhancement_level`, `voice_gain`, `noise_gate_enable`).
3. Add `model_id` parameter with an appropriate model (e.g., `"quail-vf-l-16khz"`).
4. Update any runtime VAD adjustments to use the new VAD context API.
5. We recommend to use `aic_filter.create_vad_analyzer()` for improved accuracy.

### Breaking Changes

| v1 Parameter        | v2 Replacement                            |
| ------------------- | ----------------------------------------- |
| `model_type`        | `model_id` (string-based model selection) |
| `enhancement_level` | Removed (model-specific behavior)         |
| `voice_gain`        | Removed                                   |
| `noise_gate_enable` | Removed                                   |

## Notes

* Requires ai-coustics license key (get one at [developers.ai-coustics.io](https://developers.ai-coustics.io))
* Models are automatically downloaded and cached on first use
* Supports real-time audio processing with low latency
* Handles PCM\_16 audio format (int16 samples)
* Thread-safe for pipeline processing
* Can be dynamically enabled/disabled via `FilterEnableFrame`
* Integrated VAD provides better accuracy than standalone VAD when using enhancement
* For available models, visit [artifacts.ai-coustics.io](https://artifacts.ai-coustics.io/)


# AudioBufferProcessor
Source: https://docs.pipecat.ai/server/utilities/audio/audio-buffer-processor

Process and buffer audio frames from conversations with flexible event handling

## Overview

The `AudioBufferProcessor` captures and buffers audio frames from both input (user) and output (bot) sources during conversations. It provides synchronized audio streams with configurable sample rates, supports both mono and stereo output, and offers flexible event handlers for various audio processing workflows.

## Constructor

```python theme={null}
AudioBufferProcessor(
    sample_rate=None,
    num_channels=1,
    buffer_size=0,
    enable_turn_audio=False,
    **kwargs
)
```

### Parameters

<ParamField type="Optional[int]">
  The desired output sample rate in Hz. If `None`, uses the transport's sample
  rate from the `StartFrame`.
</ParamField>

<ParamField type="int">
  Number of output audio channels:

  * `1`: Mono output (user and bot audio are mixed together)
  * `2`: Stereo output (user audio on left channel, bot audio on right channel)
</ParamField>

<ParamField type="int">
  Buffer size in bytes that triggers audio data events:

  * `0`: Events only trigger when recording stops
  * `>0`: Events trigger whenever buffer reaches this size (useful for chunked processing)
</ParamField>

<ParamField type="bool">
  Whether to enable per-turn audio event handlers (`on_user_turn_audio_data` and
  `on_bot_turn_audio_data`).
</ParamField>

## Properties

### sample\_rate

```python theme={null}
@property
def sample_rate(self) -> int
```

The current sample rate of the audio processor in Hz.

### num\_channels

```python theme={null}
@property
def num_channels(self) -> int
```

The number of channels in the audio output (1 for mono, 2 for stereo).

## Methods

### start\_recording()

```python theme={null}
async def start_recording()
```

Start recording audio from both user and bot sources. Initializes recording state and resets audio buffers.

### stop\_recording()

```python theme={null}
async def stop_recording()
```

Stop recording and trigger final audio data handlers with any remaining buffered audio.

### has\_audio()

```python theme={null}
def has_audio() -> bool
```

Check if both user and bot audio buffers contain data.

**Returns:** `True` if both buffers contain audio data.

## Event Handlers

The processor supports multiple event handlers for different audio processing workflows. Register handlers using the `@processor.event_handler()` decorator.

### on\_audio\_data

Triggered when `buffer_size` is reached or recording stops, providing merged audio.

```python theme={null}
@audiobuffer.event_handler("on_audio_data")
async def on_audio_data(buffer, audio: bytes, sample_rate: int, num_channels: int):
    # Handle merged audio data
    pass
```

**Parameters:**

* `buffer`: The AudioBufferProcessor instance
* `audio`: Merged audio data (format depends on `num_channels` setting)
* `sample_rate`: Sample rate in Hz
* `num_channels`: Number of channels (1 or 2)

### on\_track\_audio\_data

Triggered alongside `on_audio_data`, providing separate user and bot audio tracks.

```python theme={null}
@audiobuffer.event_handler("on_track_audio_data")
async def on_track_audio_data(buffer, user_audio: bytes, bot_audio: bytes,
                             sample_rate: int, num_channels: int):
    # Handle separate audio tracks
    pass
```

**Parameters:**

* `buffer`: The AudioBufferProcessor instance
* `user_audio`: Raw user audio bytes (always mono)
* `bot_audio`: Raw bot audio bytes (always mono)
* `sample_rate`: Sample rate in Hz
* `num_channels`: Always 1 for individual tracks

### on\_user\_turn\_audio\_data

Triggered when a user speaking turn ends. Requires `enable_turn_audio=True`.

```python theme={null}
@audiobuffer.event_handler("on_user_turn_audio_data")
async def on_user_turn_audio_data(buffer, audio: bytes, sample_rate: int, num_channels: int):
    # Handle user turn audio
    pass
```

**Parameters:**

* `buffer`: The AudioBufferProcessor instance
* `audio`: Audio data from the user's speaking turn
* `sample_rate`: Sample rate in Hz
* `num_channels`: Always 1 (mono)

### on\_bot\_turn\_audio\_data

Triggered when a bot speaking turn ends. Requires `enable_turn_audio=True`.

```python theme={null}
@audiobuffer.event_handler("on_bot_turn_audio_data")
async def on_bot_turn_audio_data(buffer, audio: bytes, sample_rate: int, num_channels: int):
    # Handle bot turn audio
    pass
```

**Parameters:**

* `buffer`: The AudioBufferProcessor instance
* `audio`: Audio data from the bot's speaking turn
* `sample_rate`: Sample rate in Hz
* `num_channels`: Always 1 (mono)

## Audio Processing Features

* **Automatic resampling**: Converts incoming audio to the specified sample rate
* **Buffer synchronization**: Aligns user and bot audio streams temporally
* **Silence insertion**: Fills gaps in non-continuous audio streams to maintain timing
* **Turn tracking**: Monitors speaking turns when `enable_turn_audio=True`

## Integration Notes

### STT Audio Passthrough

If using an STT service in your pipeline, enable audio passthrough to make audio available to the AudioBufferProcessor:

```python theme={null}
stt = DeepgramSTTService(
    api_key=os.getenv("DEEPGRAM_API_KEY"),
    audio_passthrough=True,
)
```

<Note>`audio_passthrough` is enabled by default.</Note>

### Pipeline Placement

Add the AudioBufferProcessor after `transport.output()` to capture both user and bot audio:

```python theme={null}
pipeline = Pipeline([
    transport.input(),
    # ... other processors ...
    transport.output(),
    audiobuffer,  # Place after audio output
    # ... remaining processors ...
])
```


# KoalaFilter
Source: https://docs.pipecat.ai/server/utilities/audio/koala-filter

Audio noise reduction filter using Koala AI technology from Picovoice

## Overview

`KoalaFilter` is an audio processor that reduces background noise in real-time audio streams using Koala Noise Suppression technology from Picovoice. It inherits from `BaseAudioFilter` and processes audio frames to improve audio quality by removing unwanted noise.

To use Koala, you need a Picovoice access key. Get started at [Picovoice Console](https://console.picovoice.ai/signup).

## Installation

The Koala filter requires additional dependencies:

```bash theme={null}
pip install "pipecat-ai[koala]"
```

You'll also need to set up your Koala access key as an environment variable: `KOALA_ACCESS_KEY`

## Constructor Parameters

<ParamField type="str">
  Picovoice access key for using the Koala noise suppression service
</ParamField>

## Input Frames

<ParamField type="Frame">
  Specific control frame to toggle filtering on/off

  ```python theme={null}
  from pipecat.frames.frames import FilterEnableFrame

  # Disable noise reduction
  await task.queue_frame(FilterEnableFrame(False))

  # Re-enable noise reduction
  await task.queue_frame(FilterEnableFrame(True))
  ```
</ParamField>

## Usage Example

```python theme={null}
from pipecat.audio.filters.koala_filter import KoalaFilter

transport = DailyTransport(
    room_url,
    token,
    "Respond bot",
    DailyParams(
        audio_in_filter=KoalaFilter(access_key=os.getenv("KOALA_ACCESS_KEY")), # Enable Koala noise reduction
        audio_in_enabled=True,
        audio_out_enabled=True,
    ),
)
```

## Notes

* Requires Picovoice access key
* Supports real-time audio processing
* Handles 16-bit PCM audio format
* Can be dynamically enabled/disabled
* Maintains audio quality while reducing noise
* Efficient processing for low latency
* Automatically handles audio frame buffering
* Sample rate must match Koala's required sample rate


# KrispVivaFilter
Source: https://docs.pipecat.ai/server/utilities/audio/krisp-viva-filter

Audio noise reduction filter using Krisp VIVA model

## Overview

`KrispVivaFilter` is an audio processor that reduces background noise in real-time audio streams using Krisp VIVA SDK. It inherits from `BaseAudioFilter` and processes audio frames to improve audio quality by removing unwanted noise using Krisp's proprietary noise suppression algorithms.

To use Krisp, you need a Krisp SDK license. Get started at [Krisp.ai](https://krisp.ai/developers/).

## Installation

See the [Krisp guide](/guides/features/krisp-viva) to learn how to install the Krisp VIVA SDK.

## Environment Variables

You need to provide the path to the Krisp model file (.kef extension). This can either be done by setting the `KRISP_VIVA_MODEL_PATH` environment variable or by setting the `model_path` in the constructor.

## Constructor Parameters

<ParamField type="str">
  Path to the Krisp model file (.kef extension).

  You can set the `model_path` directly. Alternatively, you can set the `KRISP_VIVA_MODEL_PATH` environment variable to the model file path.
</ParamField>

<ParamField type="int">
  Noise suppression level for the filter
</ParamField>

## Supported Sample Rates

The filter supports the following sample rates:

* 8000 Hz
* 16000 Hz
* 24000 Hz
* 32000 Hz
* 44100 Hz
* 48000 Hz

## Input Frames

<ParamField type="Frame">
  Specific control frame to toggle filtering on/off

  ```python theme={null}
  from pipecat.frames.frames import FilterEnableFrame

  # Disable noise reduction
  await task.queue_frame(FilterEnableFrame(False))

  # Re-enable noise reduction
  await task.queue_frame(FilterEnableFrame(True))
  ```
</ParamField>

## Usage Example

```python theme={null}
from pipecat.audio.filters.krisp_viva_filter import KrispVivaFilter
from pipecat.transports.daily.transport import DailyParams, DailyTransport

transport = DailyTransport(
    room_url,
    token,
    "Respond bot",
    DailyParams(
        audio_in_enabled=True,
        audio_in_filter=KrispVivaFilter(), # Enable Krisp noise reduction
        audio_out_enabled=True,
    ),
)
```


# SileroVADAnalyzer
Source: https://docs.pipecat.ai/server/utilities/audio/silero-vad-analyzer

Voice Activity Detection analyzer using the Silero VAD ONNX model

## Overview

`SileroVADAnalyzer` is a Voice Activity Detection (VAD) analyzer that uses the Silero VAD ONNX model to detect speech in audio streams. It provides high-accuracy speech detection with efficient processing using ONNX runtime.

## Installation

The Silero VAD analyzer requires additional dependencies:

```bash theme={null}
pip install "pipecat-ai[silero]"
```

## Constructor Parameters

<ParamField type="int">
  Audio sample rate in Hz. Must be either 8000 or 16000.
</ParamField>

<ParamField type="VADParams">
  Voice Activity Detection parameters object

  <Expandable title="properties">
    <ParamField type="float">
      Confidence threshold for speech detection. Higher values make detection more strict. Must be between 0 and 1.
    </ParamField>

    <ParamField type="float">
      Time in seconds that speech must be detected before transitioning to SPEAKING state.
    </ParamField>

    <ParamField type="float">
      Time in seconds of silence required before transitioning back to QUIET state.
    </ParamField>

    <ParamField type="float">
      Minimum audio volume threshold for speech detection. Must be between 0 and 1.
    </ParamField>
  </Expandable>
</ParamField>

## Usage Example

```python theme={null}
context = LLMContext(messages)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
    ),
)
```

## Technical Details

### Sample Rate Requirements

The analyzer supports two sample rates:

* 8000 Hz (256 samples per frame)
* 16000 Hz (512 samples per frame)

Model Management

* Uses ONNX runtime for efficient inference
* Automatically resets model state every 5 seconds to manage memory
* Runs on CPU by default for consistent performance
* Includes built-in model file

## Notes

* High-accuracy speech detection
* Efficient ONNX-based processing
* Automatic memory management
* Thread-safe for pipeline processing
* Built-in model file included
* CPU-optimized inference
* Supports 8kHz and 16kHz audio


# SoundfileMixer
Source: https://docs.pipecat.ai/server/utilities/audio/soundfile-mixer

Audio mixer for combining real-time audio with sound files

## Overview

`SoundfileMixer` is an audio mixer that combines incoming audio with audio from files. It supports multiple audio file formats through the soundfile library and can handle runtime volume adjustments and sound switching.

## Installation

The soundfile mixer requires additional dependencies:

```bash theme={null}
pip install "pipecat-ai[soundfile]"
```

## Constructor Parameters

<ParamField type="Mapping[str, str]">
  Dictionary mapping sound names to file paths. Files must be mono (single channel).
</ParamField>

<ParamField type="str">
  Name of the default sound to play (must be a key in sound\_files).
</ParamField>

<ParamField type="float">
  Initial volume for the mixed sound. Values typically range from 0.0 to 1.0, but can go higher.
</ParamField>

<ParamField type="bool">
  Whether to loop the sound file when it reaches the end.
</ParamField>

## Control Frames

<ParamField type="Frame">
  Updates mixer settings at runtime

  <Expandable title="properties">
    <ParamField type="str">
      Changes the current playing sound (must be a key in sound\_files)
    </ParamField>

    <ParamField type="float">
      Updates the mixing volume
    </ParamField>

    <ParamField type="bool">
      Updates whether the sound should loop
    </ParamField>
  </Expandable>
</ParamField>

<ParamField type="Frame">
  Enables or disables the mixer

  <Expandable title="properties">
    <ParamField type="bool">
      Whether mixing should be enabled
    </ParamField>
  </Expandable>
</ParamField>

## Usage Example

```python theme={null}
# Initialize mixer with sound files
mixer = SoundfileMixer(
    sound_files={"office": "office_ambience.wav"},
    default_sound="office",
    volume=2.0,
)

# Add to transport
transport = DailyTransport(
    room_url,
    token,
    "Audio Bot",
    DailyParams(
        audio_out_enabled=True,
        audio_out_mixer=mixer,
    ),
)

# Control mixer at runtime
await task.queue_frame(MixerUpdateSettingsFrame({"volume": 0.5}))
await task.queue_frame(MixerEnableFrame(False))  # Disable mixing
await task.queue_frame(MixerEnableFrame(True))   # Enable mixing
```

## Notes

* Supports any audio format that soundfile can read
* Automatically resamples audio files to match output sample rate
* Files must be mono (single channel)
* Thread-safe for pipeline processing
* Can dynamically switch between multiple sound files
* Volume can be adjusted in real-time
* Mixing can be enabled/disabled on demand


# DailyRESTHelper
Source: https://docs.pipecat.ai/server/utilities/daily/rest-helper

Classes and methods for interacting with the Daily API to manage rooms and tokens

<Card title="Daily REST API Documentation" icon="link" href="https://docs.daily.co/reference/rest-api">
  For complete Daily REST API reference and additional details
</Card>

## Classes

### DailyRoomSipParams

Configuration for SIP (Session Initiation Protocol) parameters.

<ResponseField name="display_name" type="string">
  Display name for the SIP endpoint
</ResponseField>

<ResponseField name="video" type="boolean">
  Whether video is enabled for SIP
</ResponseField>

<ResponseField name="sip_mode" type="string">
  SIP connection mode
</ResponseField>

<ResponseField name="num_endpoints" type="integer">
  Number of SIP endpoints
</ResponseField>

```python theme={null}
from pipecat.transports.services.helpers.daily_rest import DailyRoomSipParams

sip_params = DailyRoomSipParams(
    display_name="conference-line",
    video=True,
    num_endpoints=2
)
```

### RecordingsBucketConfig

Configuration for storing Daily recordings in a custom S3 bucket.

<ResponseField name="bucket_name" type="string">
  Name of the S3 bucket for storing recordings
</ResponseField>

<ResponseField name="bucket_region" type="string">
  AWS region where the S3 bucket is located
</ResponseField>

<ResponseField name="assume_role_arn" type="string">
  ARN of the IAM role to assume for S3 access
</ResponseField>

<ResponseField name="allow_api_access" type="boolean">
  Whether to allow API access to the recordings
</ResponseField>

```python theme={null}
from pipecat.transports.services.helpers.daily_rest import RecordingsBucketConfig

bucket_config = RecordingsBucketConfig(
    bucket_name="my-recordings-bucket",
    bucket_region="us-west-2",
    assume_role_arn="arn:aws:iam::123456789012:role/DailyRecordingsRole",
    allow_api_access=True
)
```

### DailyRoomProperties

Properties that configure a Daily room's behavior and features.

<ResponseField name="exp" type="float">
  Room expiration time as Unix timestamp (e.g., time.time() + 300 for 5 minutes)
</ResponseField>

<ResponseField name="enable_chat" type="boolean">
  Whether chat is enabled in the room
</ResponseField>

<ResponseField name="enable_prejoin_ui" type="boolean">
  Whether the prejoin lobby UI is enabled
</ResponseField>

<ResponseField name="enable_emoji_reactions" type="boolean">
  Whether emoji reactions are enabled
</ResponseField>

<ResponseField name="eject_at_room_exp" type="boolean">
  Whether to eject participants when room expires
</ResponseField>

<ResponseField name="enable_dialout" type="boolean">
  Whether dial-out is enabled
</ResponseField>

<ResponseField name="enable_recording" type="string">
  Recording settings ("cloud", "cloud-audio-only", "local", or "raw-tracks")
</ResponseField>

<ResponseField name="geo" type="string">
  Geographic region for room
</ResponseField>

<ResponseField name="max_participants" type="number">
  Maximum number of participants allowed in the room
</ResponseField>

<ResponseField name="recordings_bucket" type="RecordingsBucketConfig">
  Configuration for custom S3 bucket recordings
</ResponseField>

<ResponseField name="sip" type="DailyRoomSipParams">
  SIP configuration parameters
</ResponseField>

<ResponseField name="sip_uri" type="dict">
  SIP URI configuration (returned by Daily)
</ResponseField>

<ResponseField name="start_video_off" type="boolean">
  Whether the camera video is turned off by default
</ResponseField>

The class also includes a `sip_endpoint` property that returns the SIP endpoint URI if available.

`enable_recording` also supports `cloud-audio-only`, which records the call server-side and produces an audio-only MPEG-4 file with `.m4a` file extension and content type as `audio/mp4`. This recording setting behaves like `cloud`, except the `layout` options do not apply because there are no video tracks. Note: you can retrieve the resulting `.m4a` recordings via the [Daily REST API](https://docs.daily.co/reference/rest-api/recordings), in the same way you fetch `cloud` recording assets.

<Tip>
  If you're already using `enable_recording="cloud"` and want to switch to audio-only without changing your code, you can set `force_audio_only_recording: 1` on your Daily domain. This forces all cloud recordings to be audio-only (`.m4a` instead of `.mp4`) and skips recording video tracks even if they are present. This is useful when you want to transition to audio-only recordings immediately while avoiding an application redeploy. Switching `force_audio_only_recording: 0` will allow recording video tracks.
</Tip>

```python theme={null}
import time
from pipecat.transports.services.helpers.daily_rest import (
    DailyRoomProperties,
    DailyRoomSipParams,
    RecordingsBucketConfig,
)

properties = DailyRoomProperties(
    exp=time.time() + 3600,  # 1 hour from now
    enable_chat=True,
    enable_emoji_reactions=True,
    enable_recording="cloud-audio-only",
    geo="us-west",
    max_participants=50,
    sip=DailyRoomSipParams(display_name="conference"),
    recordings_bucket=RecordingsBucketConfig(
        bucket_name="my-bucket",
        bucket_region="us-west-2",
        assume_role_arn="arn:aws:iam::123456789012:role/DailyRole"
    )
)

# Access SIP endpoint if available
if properties.sip_endpoint:
    print(f"SIP endpoint: {properties.sip_endpoint}")
```

### DailyRoomParams

Parameters for creating a new Daily room.

<ResponseField name="name" type="string">
  Room name (if not provided, one will be generated)
</ResponseField>

<ResponseField name="privacy" type="string">
  Room privacy setting ("private" or "public")
</ResponseField>

<ResponseField name="properties" type="DailyRoomProperties">
  Room configuration properties
</ResponseField>

```python theme={null}
import time
from pipecat.transports.services.helpers.daily_rest import (
    DailyRoomParams,
    DailyRoomProperties,
)

params = DailyRoomParams(
    name="team-meeting",
    privacy="private",
    properties=DailyRoomProperties(
        enable_chat=True,
        exp=time.time() + 7200  # 2 hours from now
    )
)
```

### DailyRoomObject

Response object representing a Daily room.

<ResponseField name="id" type="string">
  Unique room identifier
</ResponseField>

<ResponseField name="name" type="string">
  Room name
</ResponseField>

<ResponseField name="api_created" type="boolean">
  Whether the room was created via API
</ResponseField>

<ResponseField name="privacy" type="string">
  Room privacy setting
</ResponseField>

<ResponseField name="url" type="string">
  Complete room URL
</ResponseField>

<ResponseField name="created_at" type="string">
  Room creation timestamp in ISO 8601 format
</ResponseField>

<ResponseField name="config" type="DailyRoomProperties">
  Room configuration
</ResponseField>

```python theme={null}
from pipecat.transports.services.helpers.daily_rest import (
    DailyRoomObject,
    DailyRoomProperties,
)

# Example of what a DailyRoomObject looks like when received
room = DailyRoomObject(
    id="abc123",
    name="team-meeting",
    api_created=True,
    privacy="private",
    url="https://your-domain.daily.co/team-meeting",
    created_at="2024-01-20T10:00:00.000Z",
    config=DailyRoomProperties(
        enable_chat=True,
        exp=1705743600
    )
)
```

### DailyMeetingTokenProperties

Properties for configuring a Daily meeting token.

<ResponseField name="room_name" type="string">
  The room this token is valid for. If not set, token is valid for all rooms.
</ResponseField>

<ResponseField name="eject_at_token_exp" type="boolean">
  Whether to eject user when token expires
</ResponseField>

<ResponseField name="eject_after_elapsed" type="integer">
  Eject user after this many seconds
</ResponseField>

<ResponseField name="nbf" type="integer">
  "Not before" timestamp - users cannot join before this time
</ResponseField>

<ResponseField name="exp" type="integer">
  Expiration timestamp - users cannot join after this time
</ResponseField>

<ResponseField name="is_owner" type="boolean">
  Whether token grants owner privileges
</ResponseField>

<ResponseField name="user_name" type="string">
  User's display name in the meeting
</ResponseField>

<ResponseField name="user_id" type="string">
  Unique identifier for the user (36 char limit)
</ResponseField>

<ResponseField name="enable_screenshare" type="boolean">
  Whether user can share their screen
</ResponseField>

<ResponseField name="start_video_off" type="boolean">
  Whether to join with video off
</ResponseField>

<ResponseField name="start_audio_off" type="boolean">
  Whether to join with audio off
</ResponseField>

<ResponseField name="enable_recording" type="string">
  Recording settings ("cloud", "cloud-audio-only", "local", or "raw-tracks")
</ResponseField>

<ResponseField name="enable_prejoin_ui" type="boolean">
  Whether to show prejoin UI
</ResponseField>

<ResponseField name="start_cloud_recording" type="boolean">
  Whether to start cloud recording when user joins
</ResponseField>

<ResponseField name="permissions" type="dict">
  Initial default permissions for a non-meeting-owner participant
</ResponseField>

### DailyMeetingTokenParams

Parameters for creating a Daily meeting token.

<ResponseField name="properties" type="DailyMeetingTokenProperties">
  Token configuration properties
</ResponseField>

```python theme={null}
from pipecat.transports.services.helpers.daily_rest import (
    DailyMeetingTokenParams,
    DailyMeetingTokenProperties,
)

token_params = DailyMeetingTokenParams(
    properties=DailyMeetingTokenProperties(
        user_name="John Doe",
        enable_screenshare=True,
        start_video_off=True,
        enable_recording="cloud-audio-only",
        start_cloud_recording=True,
        permissions={"canSend": ["video", "audio"]}
    )
)
```

<Warning>
  Recording type: ‎`cloud` will produce files with a ‎`.mp4` extension, while ‎`cloud-audio-only` will produce files with a ‎`.m4a` extension.
</Warning>

## Initialize DailyRESTHelper

Create a new instance of the Daily REST helper.

<ParamField type="string">
  Your Daily API key
</ParamField>

<ParamField type="string">
  The Daily API base URL
</ParamField>

<ParamField type="aiohttp.ClientSession">
  An aiohttp client session for making HTTP requests
</ParamField>

```python theme={null}
helper = DailyRESTHelper(
    daily_api_key="your-api-key",
    aiohttp_session=session
)
```

## Create Room

Creates a new Daily room with specified parameters.

<ParamField type="DailyRoomParams">
  Room configuration parameters including name, privacy, and properties
</ParamField>

```python theme={null}
# Create a room that expires in 1 hour
params = DailyRoomParams(
    name="my-room",
    privacy="private",
    properties=DailyRoomProperties(
        exp=time.time() + 3600,
        enable_chat=True
    )
)
room = await helper.create_room(params)
print(f"Room URL: {room.url}")
```

## Get Room From URL

Retrieves room information using a Daily room URL.

<ParamField type="string">
  The complete Daily room URL
</ParamField>

```python theme={null}
room = await helper.get_room_from_url("https://your-domain.daily.co/my-room")
print(f"Room name: {room.name}")
```

## Get Token

Generates a meeting token for a specific room.

<ParamField type="string">
  The complete Daily room URL
</ParamField>

<ParamField type="float">
  Token expiration time in seconds
</ParamField>

<ParamField type="bool">
  Whether to eject user when token expires
</ParamField>

<ParamField type="bool">
  Whether the token should have owner privileges (overrides any setting in
  params)
</ParamField>

<ParamField type="DailyMeetingTokenParams">
  Additional token configuration. Note that `room_name`, `exp`,
  `eject_at_token_exp`, and `is_owner` will be set based on the other function
  parameters.
</ParamField>

```python theme={null}
# Basic token generation
token = await helper.get_token(
    room_url="https://your-domain.daily.co/my-room",
    expiry_time=1800,  # 30 minutes
    owner=True,
    eject_at_token_exp=True
)

# Advanced token generation with additional properties
token_params = DailyMeetingTokenParams(
    properties=DailyMeetingTokenProperties(
        user_name="John Doe",
        start_video_off=True
    )
)
token = await helper.get_token(
    room_url="https://your-domain.daily.co/my-room",
    expiry_time=1800,
    owner=False,
    eject_at_token_exp=True,
    params=token_params
)
```

## Delete Room By URL

Deletes a room using its URL.

<ParamField type="string">
  The complete Daily room URL
</ParamField>

```python theme={null}
success = await helper.delete_room_by_url("https://your-domain.daily.co/my-room")
if success:
    print("Room deleted successfully")
```

## Delete Room By Name

Deletes a room using its name.

<ParamField type="string">
  The name of the Daily room
</ParamField>

```python theme={null}
success = await helper.delete_room_by_name("my-room")
if success:
    print("Room deleted successfully")
```

## Get Name From URL

Extracts the room name from a Daily room URL.

<ParamField type="string">
  The complete Daily room URL
</ParamField>

```python theme={null}
room_name = helper.get_name_from_url("https://your-domain.daily.co/my-room")
print(f"Room name: {room_name}")  # Outputs: "my-room"
```


# DTMFAggregator
Source: https://docs.pipecat.ai/server/utilities/dtmf-aggregator

Aggregates DTMF (phone keypad) input into meaningful sequences for LLM processing

## Overview

`DTMFAggregator` processes incoming DTMF (Dual-Tone Multi-Frequency) frames from phone keypad input and aggregates them into complete sequences that can be understood by LLM services. It buffers individual digit presses and flushes them as transcription frames when a termination digit is pressed, a timeout occurs, or an interruption happens.

This aggregator is essential for telephony applications where users interact via phone keypad buttons, converting raw DTMF input into structured text that LLMs can process alongside voice transcriptions.

## Constructor

```python theme={null}
aggregator = DTMFAggregator(
    timeout=2.0,
    termination_digit=KeypadEntry.POUND,
    prefix="DTMF: "
)
```

<ParamField type="float">
  Idle timeout in seconds before flushing the aggregated digits
</ParamField>

<ParamField type="KeypadEntry">
  Digit that triggers immediate flush of the aggregation
</ParamField>

<ParamField type="str">
  Prefix added to DTMF sequence in the output transcription
</ParamField>

## Input Frames

<ParamField type="Frame">
  Contains a single keypad button press with a KeypadEntry value
</ParamField>

<ParamField type="Frame">
  Flushes any pending aggregation when user interruption begins
</ParamField>

<ParamField type="Frame">
  Flushes pending aggregation and stops the aggregation task
</ParamField>

## Output Frames

<ParamField type="Frame">
  Contains the aggregated DTMF sequence as text with the configured prefix
</ParamField>

All input frames are passed through downstream, including the original `InputDTMFFrame` instances.

## Keypad Entries

The aggregator processes these standard phone keypad entries:

| KeypadEntry           | Value         | Description       |
| --------------------- | ------------- | ----------------- |
| `ZERO` through `NINE` | `"0"` - `"9"` | Numeric digits    |
| `STAR`                | `"*"`         | Star/asterisk key |
| `POUND`               | `"#"`         | Pound/hash key    |

## Aggregation Behavior

The aggregator flushes (emits a TranscriptionFrame) when:

1. **Termination digit**: The configured termination digit is pressed (default: `#`)
2. **Timeout**: No new digits received within the timeout period (default: 2 seconds)
3. **Interruption**: A `StartInterruptionFrame` is received
4. **Pipeline end**: An `EndFrame` is received

## Usage Examples

### Basic Telephony Integration

```python theme={null}
from pipecat.processors.aggregators.dtmf_aggregator import DTMFAggregator
from pipecat.serializers.twilio import TwilioFrameSerializer

# Create DTMF aggregator with default settings
dtmf_aggregator = DTMFAggregator()

# Set up Twilio serializer for phone integration
serializer = TwilioFrameSerializer(
    stream_sid=stream_sid,
    call_sid=call_sid,
    account_sid=os.getenv("TWILIO_ACCOUNT_SID"),
    auth_token=os.getenv("TWILIO_AUTH_TOKEN")
)

# Create pipeline with DTMF processing
pipeline = Pipeline([
    transport.input(),      # Websocket input from Twilio
    dtmf_aggregator,       # Process DTMF before STT
    stt,                   # Speech-to-text service
    context_aggregator.user(),
    llm,                   # LLM processes both voice and DTMF
    tts,                   # Text-to-speech
    transport.output(),
    context_aggregator.assistant(),
])
```

### Custom Configuration for Menu Systems

```python theme={null}
# Configure for menu system with star termination
menu_dtmf = DTMFAggregator(
    timeout=5.0,                    # Longer timeout for menu selection
    termination_digit=KeypadEntry.STAR,  # Use * to confirm selection
    prefix="Menu selection: "       # Clear prefix for LLM
)

# Update system prompt to handle DTMF input
messages = [
    {
        "role": "system",
        "content": """You are a phone menu assistant.

When you receive input starting with "Menu selection:", this represents
button presses on the phone keypad:
- Single digits (1-9): Menu options
- 0: Often "speak to operator"
- *: Confirmation or "go back"
- #: Usually "repeat menu"

Respond appropriately to both voice and keypad input."""
    }
]
```

## Sequence Examples

| User Input         | Aggregation Trigger    | Output TranscriptionFrame |
| ------------------ | ---------------------- | ------------------------- |
| `1`, `2`, `3`, `#` | Termination digit      | `"DTMF: 123#"`            |
| `*`, `0`           | 2-second timeout       | `"DTMF: *0"`              |
| `5`, interruption  | StartInterruptionFrame | `"DTMF: 5"`               |
| `9`, `9`, EndFrame | Pipeline shutdown      | `"DTMF: 99"`              |

## Frame Flow

```mermaid theme={null}
graph TD
    A[Twilio WebSocket] --> B[TwilioFrameSerializer]
    B --> C[DTMFAggregator]
    C --> D[STT Service]
    C --> E[TranscriptionFrame]
    E --> F[LLM Context Aggregator]
    D --> F
    F --> G[LLM Service]
```

## Error Handling

The aggregator gracefully handles:

* Invalid DTMF digits (logged and ignored)
* Pipeline interruptions (flushes pending sequences)
* Rapid key presses (buffers efficiently)
* Mixed voice and DTMF input (processes independently)

## Best Practices

1. **System Prompt Design**: Train your LLM to recognize and respond to DTMF prefixed input
2. **Timeout Configuration**: Use shorter timeouts (1-2s) for rapid entry, longer (3-5s) for menu selection
3. **Termination Strategy**: Use `#` for confirmation, `*` for cancel/back operations
4. **Pipeline Placement**: Always place before the user context aggregator to ensure proper frame ordering


# IVRNavigator
Source: https://docs.pipecat.ai/server/utilities/extensions/ivr

AI-powered Interactive Voice Response system navigation with automatic classification and goal-oriented decision making

## Overview

The `IVRNavigator` is a pipeline component that provides intelligent navigation of IVR phone systems. It combines LLM-based decision making with DTMF tone generation to automatically traverse phone menus toward specified goals. The navigator includes automatic classification between IVR systems and human conversations, enabling flexible call handling scenarios.

<CardGroup>
  <Card title="IVR Navigator Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.extensions.ivr.ivr_navigator.html">
    Complete API documentation and method details
  </Card>
</CardGroup>

## Constructor

```python theme={null}
from pipecat.extensions.ivr.ivr_navigator import IVRNavigator, IVRStatus

IVRNavigator(
    llm,
    ivr_prompt,
    ivr_vad_params=None
)
```

### Parameters

<ParamField type="LLMService">
  The LLM service instance used for text generation and navigation decision
  making.
</ParamField>

<ParamField type="str">
  The navigation goal that will be integrated with the base IVR navigation
  instructions. This should clearly describe what you want to accomplish (e.g.,
  "Navigate to billing support to discuss account charges").
</ParamField>

<ParamField type="Optional[VADParams]">
  Voice Activity Detection parameters optimized for IVR navigation. The default
  2.0 second stop time allows the system to hear complete menu options before
  responding, improving navigation success rates.
</ParamField>

## Navigation Commands

The IVRNavigator processes several XML-tagged commands from LLM responses:

### DTMF Commands

```xml theme={null}
<dtmf>1</dtmf>        <!-- Single keypress -->
<dtmf>1</dtmf><dtmf>2</dtmf><dtmf>3</dtmf>  <!-- Sequence entry -->
```

Valid DTMF values: `0-9`, `*`, `#`

### IVR Status Commands

```xml theme={null}
<ivr>detected</ivr>   <!-- IVR system detected, switch to navigation mode -->
<ivr>completed</ivr>  <!-- Navigation goal achieved -->
<ivr>stuck</ivr>      <!-- Unable to proceed -->
<ivr>wait</ivr>       <!-- Waiting for more complete information -->
```

### Mode Classification

```xml theme={null}
<mode>ivr</mode>          <!-- Classified as IVR system -->
<mode>conversation</mode> <!-- Classified as human conversation -->
```

## Event Handlers

The navigator supports two primary event handlers for different interaction scenarios.

### on\_conversation\_detected

Triggered when the system classifies incoming audio as a human conversation rather than an IVR system.

```python theme={null}
@ivr_navigator.event_handler("on_conversation_detected")
async def on_conversation_detected(processor, conversation_history: List[dict]):
    # Handle human conversation scenario
    # conversation_history contains previous interaction context
    pass
```

**Parameters:**

* `processor`: The IVRProcessor instance
* `conversation_history`: List of message dictionaries from previous context (excluding system messages)

### on\_ivr\_status\_changed

Triggered when IVR navigation status changes during the navigation process.

```python theme={null}
@ivr_navigator.event_handler("on_ivr_status_changed")
async def on_ivr_status_changed(processor, status: IVRStatus):
    # Handle navigation status changes
    if status == IVRStatus.COMPLETED:
        # Navigation successful
        pass
    elif status == IVRStatus.STUCK:
        # Navigation failed
        pass
```

**Parameters:**

* `processor`: The IVRProcessor instance
* `status`: IVRStatus enum value (`DETECTED`, `COMPLETED`, `STUCK`)

## IVRStatus Enumeration

```python theme={null}
from pipecat.extensions.ivr.ivr_navigator import IVRStatus

class IVRStatus(Enum):
    DETECTED = "detected"    # IVR system detected and navigation started
    COMPLETED = "completed"  # Navigation goal successfully achieved
    STUCK = "stuck"         # Navigation unable to proceed
    WAIT = "wait"           # Waiting for more complete information
```

## Built-in Prompts

The IVRNavigator includes sophisticated built-in prompts that handle classification and navigation logic automatically.

### Classification Prompt

The navigator starts with an automatic classification system that distinguishes between:

**IVR System Indicators:**

* Menu options ("Press 1 for billing", "Press 2 for support")
* Automated instructions ("Please enter your account number")
* System prompts ("Thank you for calling \[company]")
* Hold messages ("Please continue to hold")

**Human Conversation Indicators:**

* Personal greetings ("Hello, this is Sarah")
* Interactive responses ("Who am I speaking with?")
* Natural speech patterns and conversational flow
* Direct engagement ("I can help with that")

### Navigation Prompt

Once IVR is detected, the system uses your provided goal with comprehensive navigation instructions that include:

* **Decision making logic** for menu option selection
* **DTMF sequence handling** for data entry (dates, account numbers, phone numbers)
* **Verbal response generation** for conversational prompts
* **Completion detection** for successful navigation
* **Stuck state recognition** for error scenarios
* **Wait state management** for incomplete transcriptions

## Advanced Usage

### Context Preservation

The navigator automatically preserves conversation context when switching between modes:

```python theme={null}
# Context is automatically saved and restored during mode transitions
# No manual intervention required for basic scenarios

# For advanced context manipulation:
@ivr_navigator.event_handler("on_conversation_detected")
async def handle_conversation(processor, conversation_history):
    # conversation_history contains preserved context
    # Add additional context as needed
    enhanced_context = [
        {"role": "system", "content": "You are a customer service representative."},
        *conversation_history,
        {"role": "assistant", "content": "How can I help you today?"}
    ]

    await task.queue_frame(LLMMessagesUpdateFrame(messages=enhanced_context, run_llm=True))
```

### VAD Parameter Management

```python theme={null}
# Switch to conversation-optimized timing
@ivr_navigator.event_handler("on_conversation_detected")
async def optimize_for_conversation(processor, conversation_history):
    # Reduce response delay for natural conversation flow
    conversation_vad = VADParams(stop_secs=0.8)
    await task.queue_frame(VADParamsUpdateFrame(params=conversation_vad))
```

## Integration Examples

### Basic Pipeline Integration

```python theme={null}
from pipecat.pipeline.pipeline import Pipeline
from pipecat.processors.ivr.ivr_navigator import IVRNavigator

pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    ivr_navigator,    # Add in place of LLM
    tts,
    transport.output(),
    context_aggregator.assistant()
])
```


# VoicemailDetector
Source: https://docs.pipecat.ai/server/utilities/extensions/voicemail

Technical implementation details and architecture for voicemail detection

## Overview

This document provides technical implementation details for the VoicemailDetector, including its parallel pipeline architecture and performance considerations for production use.

<CardGroup>
  <Card title="API Reference" icon="book" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.extensions.voicemail.voicemail_detector.html">
    Complete API documentation for all VoicemailDetector classes and methods
  </Card>

  <Card title="Example Implementation" icon="code" href="https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/44-voicemail-detection.py">
    Working example showing VoicemailDetector integration in a complete pipeline
  </Card>
</CardGroup>

## Architecture

The VoicemailDetector uses a parallel pipeline architecture with two processing branches:

* **Conversation Branch**: Handles normal conversation flow and can be blocked when voicemail is detected to prevent the main LLM from processing additional input.

* **Classification Branch**: Contains the LLM classifier and decision logic. This branch closes after making a classification decision to prevent unnecessary LLM calls.

The system coordinates between branches using a notification system and gates that control frame flow based on classification decisions. TTS frames are buffered during classification and either released (for conversations) or cleared (for voicemail) based on the decision.

## Performance Considerations

### LLM Selection

The VoicemailDetector works with two separate LLMs in your pipeline:

**Conversation LLM (main pipeline):**

* Must be text-based for compatibility with the VoicemailDetector's gating system
* Realtime LLMs are not compatible as they don't allow the required level of output control
* This is the LLM that handles normal conversation after classification

**Classifier LLM (VoicemailDetector parameter):**

* Can be either text-based or realtime LLMs
* For realtime LLMs: Set output modality to `text` to ensure text-based responses
* Must be able to output "CONVERSATION" or "VOICEMAIL" keywords for classification
* Recommended models: OpenAILLMService with `gpt-4o`, GoogleLLMService with `gemini-2.0-flash`

The two LLMs operate independently. You can use a realtime LLM for classification (with text output) while using a text-based LLM for conversation, or use text-based LLMs for both.

### Response Timing

The `voicemail_response_delay` parameter should be tuned based on your target voicemail systems. The default value of 2 seconds is a good starting point, but you may need to adjust it based on your target voicemail systems.

## Common Issues

**Classification not working:**

* Confirm both `detector()` and `gate()` are correctly placed in pipeline
* Check that STT service is producing text input for classification
* Ensure the LLM is a text-based LLM

**Timing problems:**

* Adjust `voicemail_response_delay` based on observed voicemail greeting patterns
* Monitor classification speed; slow LLM responses affect conversation latency

**Audio playback issues:**

* Ensure `gate()` is placed immediately after TTS service
* Verify TTS frames are being generated before classification completes

**Custom prompt validation:**
The system validates custom prompts and warns if they're missing required response keywords ("CONVERSATION" and "VOICEMAIL"). Include the `CLASSIFIER_RESPONSE_INSTRUCTION` constant to ensure proper functionality.


# FrameFilter
Source: https://docs.pipecat.ai/server/utilities/filters/frame-filter

Processor that selectively passes through only specified frame types

## Overview

`FrameFilter` is a processor that filters frames based on their types, only passing through frames that match specified types (plus some system frames like `EndFrame` and `SystemFrame`).

## Constructor Parameters

<ParamField type="Tuple[Type[Frame], ...]">
  Tuple of frame types that should be passed through the filter
</ParamField>

## Functionality

When a frame passes through the filter, it is checked against the provided types. Only frames that match one of the specified types (or are system frames) will be passed downstream. All other frames are dropped.

## Output Frames

The processor always passes through:

* Frames matching any of the specified types
* `EndFrame` and `SystemFrame` instances (always allowed, so as to not block the pipeline)

## Usage Example

```python theme={null}
from pipecat.frames.frames import TextFrame, AudioRawFrame, Frame
from pipecat.processors.filters import FrameFilter
from typing import Tuple, Type

# Create a filter that only passes TextFrames and AudioRawFrames
text_and_audio_filter = FrameFilter(
    types=(TextFrame, AudioRawFrame)
)

# Add to pipeline
pipeline = Pipeline([
    source,
    text_and_audio_filter,  # Filters out all other frame types
    destination
])
```

## Frame Flow

```mermaid theme={null}
graph TD
    A[Input Frames] --> B[FrameFilter]
    B --> C{Frame Type Check}
    C -->|Matches Allowed Types| D[Output Frame]
    C -->|System Frame| D
    C -->|Other Frame Types| E[Dropped]
```

## Notes

* Simple but powerful way to restrict which frame types flow through parts of your pipeline
* Always allows system frames to pass through for proper pipeline operation
* Can be used to isolate specific parts of your pipeline from certain frame types
* Efficient implementation with minimal overhead


# FunctionFilter
Source: https://docs.pipecat.ai/server/utilities/filters/function-filter

Processor that filters frames using a custom filter function

## Overview

`FunctionFilter` is a flexible processor that uses a custom async function to determine which frames to pass through. This allows for complex, dynamic filtering logic beyond simple type checking.

## Constructor Parameters

<ParamField type="Callable[[Frame], Awaitable[bool]]">
  Async function that examines each frame and returns True to allow it or False
  to filter it out
</ParamField>

<ParamField type="FrameDirection">
  Which direction of frames to filter (DOWNSTREAM or UPSTREAM)
</ParamField>

## Functionality

When a frame passes through the processor:

1. System frames and end frames are always passed through
2. Frames moving in a different direction than specified are always passed through
3. Other frames are passed to the filter function
4. If the filter function returns True, the frame is passed through

## Output Frames

The processor conditionally passes through frames based on:

* Frame type (system frames and end frames always pass)
* Frame direction (only filters in the specified direction)
* Result of the custom filter function

## Usage Example

```python theme={null}
from pipecat.frames.frames import TextFrame, Frame
from pipecat.processors.filters import FunctionFilter
from pipecat.processors.frame_processor import FrameDirection

# Create filter that only allows TextFrames with more than 10 characters
async def long_text_filter(frame: Frame) -> bool:
    if isinstance(frame, TextFrame):
        return len(frame.text) > 10
    return False

# Apply filter to downstream frames only
text_length_filter = FunctionFilter(
    filter=long_text_filter,
    direction=FrameDirection.DOWNSTREAM
)

# Add to pipeline
pipeline = Pipeline([
    source,
    text_length_filter,  # Filters out short text frames
    destination
])
```

## Frame Flow

```mermaid theme={null}
graph TD
    A[Input Frames] --> B[FunctionFilter]
    B --> C{System/End Frame?}
    C -->|Yes| F[Output Frame]
    C -->|No| D{Correct Direction?}
    D -->|No| F
    D -->|Yes| E{Filter Function}
    E -->|Returns True| F
    E -->|Returns False| G[Dropped]
```

## Notes

* Provides maximum flexibility for complex filtering logic
* Can incorporate dynamic conditions that change at runtime
* Only filters frames moving in the specified direction
* Always passes through system frames for proper pipeline operation
* Can be used to create sophisticated content-based filters
* Supports async filter functions for complex processing


# IdentityFilter
Source: https://docs.pipecat.ai/server/utilities/filters/identify-filter

Processor that passes all frames through without modification

## Overview

`IdentityFilter` is a simple pass-through processor that forwards all frames without any modification or filtering. It acts as a transparent layer in your pipeline, allowing all frames to flow through unchanged.

<Tip>
  Check out Observers for an option that delivers similar functionality but
  doesn't require a processor to reside in the Pipeline.
</Tip>

## Constructor Parameters

The `IdentityFilter` constructor accepts no specific parameters beyond those inherited from `FrameProcessor`.

## Functionality

When a frame passes through the processor, it is immediately forwarded in the same direction with no changes. This applies to all frame types and both directions (upstream and downstream).

## Use Cases

While functionally equivalent to having no filter at all, `IdentityFilter` can be useful in several scenarios:

* Testing `ParallelPipeline` configurations to ensure frames aren't duplicated
* Acting as a placeholder where a more complex filter might be added later
* Monitoring frame flow in pipelines by adding logging in subclasses
* Creating a base class for more complex conditional filters

## Usage Example

```python theme={null}
from pipecat.processors.filters import IdentityFilter

# Create an identity filter
pass_through = IdentityFilter()

# Add to pipeline
pipeline = Pipeline([
    source,
    pass_through,  # All frames pass through unchanged
    destination
])
```

## Frame Flow

```mermaid theme={null}
graph LR
    A[Input Frame] --> B[IdentityFilter] --> C[Output Frame]
```

## Notes

* Simplest possible filter implementation
* Passes all frames through without modification
* Useful in testing parallel pipelines
* Can serve as a placeholder or base class
* Zero overhead in normal operation


# NullFilter
Source: https://docs.pipecat.ai/server/utilities/filters/null-filter

Processor that blocks all frames except system frames

## Overview

`NullFilter` is a filtering processor that blocks all frames from passing through, with the exception of system frames and end frames which are required for proper pipeline operation.

## Constructor Parameters

The `NullFilter` constructor accepts no specific parameters beyond those inherited from `FrameProcessor`.

## Functionality

When a frame passes through the processor:

* If the frame is a `SystemFrame` or `EndFrame`, it is passed through
* All other frame types are blocked and do not continue through the pipeline

This filter effectively acts as a barrier that allows only the essential system frames required for pipeline initialization, shutdown, and management.

## Use Cases

`NullFilter` is useful in several scenarios:

* Temporarily disabling parts of a pipeline without removing components
* Creating dead-end branches in parallel pipelines
* Testing pipeline behavior with blocked communication
* Implementing conditional pipelines where certain paths should be blocked

## Usage Example

```python theme={null}
from pipecat.processors.filters import NullFilter

# Create a null filter that blocks all non-system frames
blocker = NullFilter()

# Add to pipeline
pipeline = Pipeline([
    source,
    blocker,  # Blocks all regular frames
    destination  # Will only receive system frames
])
```

## Frame Flow

```mermaid theme={null}
graph TD
    A[Input Frames] --> B[NullFilter]
    B --> C{System/End Frame?}
    C -->|Yes| D[Output Frame]
    C -->|No| E[Blocked]
```

## Notes

* Blocks all regular frames in both directions
* Only allows system frames and end frames to pass through
* Useful for testing, debugging, and creating conditional pipelines
* Minimal overhead as it performs simple type checking
* Can be used to temporarily disable parts of a pipeline


# STTMuteFilter
Source: https://docs.pipecat.ai/server/utilities/filters/stt-mute

Processor for controlling STT muting and interruption handling during bot speech and function calls

<Warning>
  DEPRECATED: STTMuteFilter has been deprecated in favor of [User Mute
  Strategies](/server/utilities/turn-management/user-mute-strategies). Configure
  `user_mute_strategies` on the `LLMUserAggregator` instead.
</Warning>

## Overview

`STTMuteFilter` is a general-purpose processor that combines STT muting and interruption control. When active, it prevents both transcription and interruptions during specified conditions (e.g., bot speech, function calls), providing a cleaner conversation flow.

The processor supports multiple simultaneous strategies for when to mute the STT service, making it flexible for different use cases.

<Tip>
  Want to try it out? Check out the [STTMuteFilter foundational
  demo](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/24-stt-mute-filter.py)
</Tip>

## Constructor Parameters

<ParamField type="STTMuteConfig">
  Configuration object that defines the muting strategies and optional custom
  logic
</ParamField>

<ParamField type="Optional[STTService]">
  The STT service to control (deprecated, will be removed in a future version)
</ParamField>

## Configuration

The processor is configured using `STTMuteConfig`, which determines when and how the STT service should be muted:

<ParamField type="set[STTMuteStrategy]">
  Set of muting strategies to apply
</ParamField>

<ParamField type="Callable[[STTMuteFilter], Awaitable[bool]]">
  Optional callback for custom muting logic (required when strategy is `CUSTOM`)
</ParamField>

### Muting Strategies

`STTMuteConfig` accepts a set of these `STTMuteStrategy` values:

<ParamField type="STTMuteStrategy">
  Mute only during the bot's first speech (typically during introduction)
</ParamField>

<ParamField type="STTMuteStrategy">
  Start muted and remain muted until first bot speech completes. Useful when bot
  speaks first and you want to ensure its first response cannot be interrupted.
</ParamField>

<ParamField type="STTMuteStrategy">
  Mute during LLM function calls (e.g., API requests, external service calls)
</ParamField>

<ParamField type="STTMuteStrategy">
  Mute during all bot speech
</ParamField>

<ParamField type="STTMuteStrategy">
  Use custom logic provided via callback to determine when to mute. The callback
  is invoked when the bot is speaking and can use application state to decide
  whether to mute. When the bot stops speaking, unmuting occurs automatically if
  no other strategy requires muting.
</ParamField>

<Note>
  `MUTE_UNTIL_FIRST_BOT_COMPLETE` and `FIRST_SPEECH` strategies should not be
  used together as they handle the first bot speech differently.
</Note>

## Input Frames

<ParamField type="Frame">
  Indicates bot has started speaking
</ParamField>

<ParamField type="Frame">
  Indicates bot has stopped speaking
</ParamField>

<ParamField type="Frame">
  Indicates a function call has started
</ParamField>

<ParamField type="Frame">
  Indicates a function call has completed
</ParamField>

<ParamField type="Frame">
  Indicates an interim transcription result (suppressed when muted)
</ParamField>

<ParamField type="Frame">
  User interruption start event (suppressed when muted)
</ParamField>

<ParamField type="Frame">
  User interruption stop event (suppressed when muted)
</ParamField>

<ParamField type="Frame">
  Indicates a transcription result (suppressed when muted)
</ParamField>

<ParamField type="Frame">
  Indicates user has started speaking (suppressed when muted)
</ParamField>

<ParamField type="Frame">
  Indicates user has stopped speaking (suppressed when muted)
</ParamField>

## Output Frames

<ParamField type="Frame">
  Control frame to mute/unmute the STT service
</ParamField>

All input frames are passed through except VAD-related frames (interruptions and user speaking events) when muted.

## Usage Examples

### Basic Usage (Mute During Bot's First Speech)

```python theme={null}
stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
stt_mute_filter = STTMuteFilter(
    config=STTMuteConfig(strategies={
        STTMuteStrategy.FIRST_SPEECH
    })
)

pipeline = Pipeline([
    transport.input(),
    stt,
    stt_mute_filter,  # Between the STT service and context aggregator
    context_aggregator.user(),
    # ... rest of pipeline
])
```

### Mute Until First Bot Response Completes

```python theme={null}
stt_mute_filter = STTMuteFilter(
    config=STTMuteConfig(strategies={STTMuteStrategy.MUTE_UNTIL_FIRST_BOT_COMPLETE})
)
```

This ensures no user speech is processed until after the bot's first complete response.

### Always Mute During Bot Speech

```python theme={null}
stt_mute_filter = STTMuteFilter(
    config=STTMuteConfig(strategies={STTMuteStrategy.ALWAYS})
)
```

### Custom Muting Logic

The `CUSTOM` strategy allows you to control muting based on application state when the bot is speaking. The callback will be invoked whenever the bot is speaking, and your logic decides whether to mute:

```python theme={null}
# Create a state manager
class SessionState:
    def __init__(self):
        self.session_ending = False

session_state = SessionState()

# Callback function that determines whether to mute
async def session_state_mute_logic(stt_filter: STTMuteFilter) -> bool:
    # Return True to mute, False otherwise
    # This is called when the bot is speaking
    return session_state.session_ending

# Configure filter with CUSTOM strategy
stt_mute_filter = STTMuteFilter(
    config=STTMuteConfig(
        strategies={STTMuteStrategy.CUSTOM},
        should_mute_callback=session_state_mute_logic
    )
)

# Later, when you want to trigger muting (e.g., during session timeout):
async def handle_session_timeout():
    # Update state that will be checked by the callback
    session_state.session_ending = True

    # Send goodbye message
    goodbye_message = "Thank you for using our service. This session is now ending."
    await pipeline.push_frame(TTSSpeakFrame(text=goodbye_message))

    # The system will automatically mute during this message because:
    # 1. Bot starts speaking, triggering the callback
    # 2. Callback returns True (session_ending is True)
    # 3. When bot stops speaking, unmuting happens automatically
```

### Combining Multiple Strategies

```python theme={null}
async def custom_mute_logic(processor: STTMuteFilter) -> bool:
    # Example: Mute during business hours only
    current_hour = datetime.now().hour
    return 9 <= current_hour < 17

stt_mute_filter = STTMuteFilter(
    config=STTMuteConfig(
        strategies={
            STTMuteStrategy.FUNCTION_CALL,       # Mute during function calls
            STTMuteStrategy.CUSTOM,              # And during business hours
            STTMuteStrategy.MUTE_UNTIL_FIRST_BOT_COMPLETE  # And until first bot speech completes
        },
        should_mute_callback=custom_mute_logic
    )
)
```

### Frame Flow

```mermaid theme={null}
graph TD
    A[Transport Input] --> B[STTMuteFilter]
    B --> C[STT Service]
    B -- "Suppressed when muted" --> D[VAD-related Frames]
    B -- "STTMuteFrame" --> C
```

## Notes

* Combines STT muting and interruption control into a single concept
* Muting prevents both transcription and interruptions
* Multiple strategies can be active simultaneously
* CUSTOM strategy callback is only invoked when the bot is speaking
* Unmuting happens automatically when bot speech ends (if no other strategy requires muting)
* Placed between the STT service and context aggregator in pipeline
* Maintains conversation flow during bot speech and function calls
* Efficient state tracking for minimal overhead


# WakeCheckFilter
Source: https://docs.pipecat.ai/server/utilities/filters/wake-check-filter

Processor that passes frames only after detecting wake phrases in transcriptions

## Overview

`WakeCheckFilter` monitors `TranscriptionFrame`s for specified wake phrases and only allows frames to pass through after a wake phrase has been detected. It includes a keepalive timeout to maintain the awake state for a period after detection, allowing continuous conversation without requiring repeated wake phrases.

## Constructor Parameters

<ParamField type="list[str]">
  List of wake phrases to detect in transcriptions
</ParamField>

<ParamField type="float">
  Number of seconds to remain in the awake state after each transcription
</ParamField>

## Functionality

The filter maintains state for each participant and processes frames as follows:

1. `TranscriptionFrame` objects are checked for wake phrases
2. If a wake phrase is detected, the filter enters the "AWAKE" state
3. While in the "AWAKE" state, all transcription frames pass through
4. After no activity for the keepalive timeout period, the filter returns to "IDLE"
5. All non-transcription frames pass through normally

Wake phrases are detected using regular expressions that match whole words with flexible spacing, making detection resilient to minor transcription variations.

## States

<ParamField type="WakeState">
  Default state - only non-transcription frames pass through
</ParamField>

<ParamField type="WakeState">
  Active state after wake phrase detection - all frames pass through
</ParamField>

## Output Frames

* All non-transcription frames pass through unchanged
* After wake phrase detection, transcription frames pass through
* When awake, transcription frames reset the keepalive timer

## Usage Example

```python theme={null}
from pipecat.processors.filters import WakeCheckFilter

# Create filter with wake phrases
wake_filter = WakeCheckFilter(
    wake_phrases=["hey assistant", "ok computer", "listen up"],
    keepalive_timeout=5.0  # Stay awake for 5 seconds after each transcription
)

# Add to pipeline
pipeline = Pipeline([
    transport.input(),
    stt_service,
    wake_filter,  # Only passes transcriptions after wake phrases
    llm_service,
    tts_service,
    transport.output()
])
```

## Frame Flow

```mermaid theme={null}
graph TD
    A[Input Frames] --> B[WakeCheckFilter]
    B --> C{Transcription Frame?}
    C -->|No| F[Output Frame]
    C -->|Yes| D{Wake State}
    D -->|AWAKE| E{Keepalive Expired?}
    E -->|No| F
    E -->|Yes| G[Return to IDLE]
    D -->|IDLE| H{Contains Wake Phrase?}
    H -->|Yes| I[Set AWAKE] --> F
    H -->|No| J[Filtered Out]
```

## Notes

* Maintains separate state for each participant ID
* Uses regex pattern matching for resilient wake phrase detection
* Accumulates transcription text to detect phrases across multiple frames
* Trims accumulated text when wake phrase is detected
* Supports multiple wake phrases
* Passes all non-transcription frames through unchanged
* Error handling produces ErrorFrames for robust operation
* Case-insensitive matching for natural language use


# WakeNotifierFilter
Source: https://docs.pipecat.ai/server/utilities/filters/wake-notifier-filter

Processor that triggers a notifier when specified frame types pass a custom filter

## Overview

`WakeNotifierFilter` monitors the pipeline for specific frame types and triggers a notification when those frames pass a custom filter condition. It passes all frames through unchanged while performing this notification side-effect.

## Constructor Parameters

<ParamField type="BaseNotifier">
  The notifier object to trigger when conditions are met
</ParamField>

<ParamField type="Tuple[Type[Frame]]">
  Tuple of frame types to monitor
</ParamField>

<ParamField type="Callable[[Frame], Awaitable[bool]]">
  Async function that examines each matching frame and returns True to trigger
  notification
</ParamField>

## Functionality

The processor operates as follows:

1. Checks if the incoming frame matches any of the specified types
2. If it's a matching type, calls the filter function with the frame
3. If the filter returns True, triggers the notifier
4. Passes all frames through unchanged, regardless of the filtering result

This allows for notification side-effects without modifying the pipeline's data flow.

## Output Frames

* All frames pass through unchanged in their original direction
* No frames are modified or filtered out

## Usage Example

```python theme={null}
from pipecat.frames.frames import TranscriptionFrame, UserStartedSpeakingFrame
from pipecat.processors.filters import WakeNotifierFilter
from pipecat.sync.event_notifier import EventNotifier

# Create an event notifier
wake_event = EventNotifier()

# Create filter that notifies when certain wake phrases are detected
async def wake_phrase_filter(frame):
    if isinstance(frame, TranscriptionFrame):
        return "hey assistant" in frame.text.lower()
    return False

# Add to pipeline
wake_notifier = WakeNotifierFilter(
    notifier=wake_event,
    types=(TranscriptionFrame, UserStartedSpeakingFrame),
    filter=wake_phrase_filter
)

# In another component, wait for the notification
async def handle_wake_event():
    await wake_event.wait()
    print("Wake phrase detected!")
```

## Frame Flow

```mermaid theme={null}
graph TD
    A[Input Frame] --> B[WakeNotifierFilter]
    B --> C{Matching Type?}
    C -->|Yes| D{Filter Function}
    D -->|Returns True| E[Notify]
    D -->|Returns False| F[Pass Through]
    C -->|No| F
    E --> F
```

## Notes

* Acts as a transparent pass-through for all frames
* Can trigger external events without modifying pipeline flow
* Useful for signaling between pipeline components
* Can monitor for multiple frame types simultaneously
* Uses async filter function for complex conditions
* Functions as a "listener" that doesn't affect the data stream
* Can be used for logging, analytics, or coordinating external systems


# LLMTextProcessor
Source: https://docs.pipecat.ai/server/utilities/frame/llm-text-processor

A processor for aggregating LLMTextFrames into logical units before passing them to downstream services

## Overview

`LLMTextProcessor` is a processor designed to aggregate `LLMTextFrame`s into coherent text units before passing them to downstream services, such as TTS. By utilizing text aggregators, it ensures that text is properly segmented and structured, enhancing the quality of subsequent processing. This processor expects `LLMTextFrame`s as input and outputs `AggregatedTextFrame`s containing the aggregated text.

<Note>
  If an `LLMTextProcessor` is in use, the `text_aggregator` parameter of TTS services will be ignored, as text aggregation is handled upstream.
</Note>

The benefit of pre-aggregating LLM text frames is that it allows for more controlled and meaningful text synthesis. Downstream services can operate on complete sentences or logical text blocks. For TTS services, this means being able to customize how certain types of text are spoken (e.g., spelling out phone numbers, stripping out url protocols, or inserting other tts-specific annotations) or even skipping over certain text segments entirely (e.g., code snippets or markup).  For other services, such as RTVI, it allows for sending these logical text units as separate `bot-output` messages, supporting custom client-side handling and rendering (e.g. collapsible code blocks, clickable links, etc.).

<CardGroup>
  <Card title="LLMTextProcessor API Reference" icon="code" href="https://reference-server.pipecat.ai/en/latest/api/pipecat.processors.aggregators.llm_text_processor.html">
    Pipecat's API methods for LLMTextProcessor
  </Card>

  <Card title="Example Implementation" icon="play" href="https://github.com/pipecat-ai/pipecat-examples/blob/main/code-helper">
    Complete example with LLMTextProcessor and custom TTS and RTVI handling
  </Card>
</CardGroup>

## Constructor Parameters

<ParamField type="BaseTextAggregator">
  An instance of a text aggregator (e.g., `PatternPairAggregator` or a custom aggregator type) used to aggregate incoming text from `LLMTextFrame`s.
  If `None`, a `SimpleTextAggregator` will be used by default, aggregating text based on sentence boundaries.
</ParamField>

## Usage

The `LLMTextProcessor` should be integrated into your pipeline after the LLM service and before any services that consume text, such as TTS. It processes incoming `LLMTextFrame`s, aggregates their text content, and outputs `AggregatedTextFrame`s.

<Tip>
  For more usage examples, check out the docs for the [PatternPairAggregator](/server/utilities/text/pattern-pair-aggregator#usage-examples).
</Tip>

```python theme={null}
from pipecat.processors.aggregators.llm_text_processor import LLMTextProcessor

...

llm_text_aggregator = PatternPairAggregator()
llm_text_aggregator.add_pattern(
    type="code",
    start_pattern="<code>",
    end_pattern="</code>",
    action=MatchAction.AGGREGATE,
)
llm_text_processor = LLMTextProcessor(text_aggregator=llm_text_aggregator)

...

# Pipeline - The following pipeline is typical for a STT->LLM->TTS bot + RTVI
#            with the addition of the LLMTextProcessor to handle special text segments.
pipeline = Pipeline(
    [
        transport.input(),
        rtvi,
        stt,
        transcript_processor.user(),
        context_aggregator.user(),
        llm,
        llm_text_processor,
        tts,
        transport.output(),
        transcript_processor.assistant(),
        context_aggregator.assistant(),
    ]
)
```


# Producer & Consumer Processors
Source: https://docs.pipecat.ai/server/utilities/frame/producer-consumer

Route frames between different parts of a pipeline, allowing selective frame sharing across parallel branches or within complex pipelines

## Overview

The Producer and Consumer processors work as a pair to route frames between different parts of a pipeline, particularly useful when working with [`ParallelPipeline`](/server/pipeline/parallel-pipeline). They allow you to selectively capture frames from one pipeline branch and inject them into another.

## ProducerProcessor

`ProducerProcessor` examines frames flowing through the pipeline, applies a filter to decide which frames to share, and optionally transforms these frames before sending them to connected consumers.

### Constructor Parameters

<ParamField type="Callable[[Frame], Awaitable[bool]]">
  An async function that determines which frames should be sent to consumers.
  Should return `True` for frames to be shared.
</ParamField>

<ParamField type="Callable[[Frame], Awaitable[Frame]]">
  Optional async function that transforms frames before sending to consumers. By
  default, passes frames unchanged.
</ParamField>

<ParamField type="bool">
  When `True`, passes all frames through the normal pipeline flow. When `False`,
  only passes through frames that don't match the filter.
</ParamField>

## ConsumerProcessor

`ConsumerProcessor` receives frames from a `ProducerProcessor` and injects them into its pipeline branch.

### Constructor Parameters

<ParamField type="ProducerProcessor">
  The producer processor that will send frames to this consumer.
</ParamField>

<ParamField type="Callable[[Frame], Awaitable[Frame]]">
  Optional async function that transforms frames before injecting them into the
  pipeline.
</ParamField>

<ParamField type="FrameDirection">
  The direction in which to push received frames. Usually `DOWNSTREAM` to send
  frames forward in the pipeline.
</ParamField>

## Usage Examples

### Basic Usage: Moving TTS Audio Between Branches

```python theme={null}
# Create a producer that captures TTS audio frames
async def is_tts_audio(frame: Frame) -> bool:
    return isinstance(frame, TTSAudioRawFrame)

# Define an async transformer function
async def tts_to_input_audio_transformer(frame: Frame) -> Frame:
    if isinstance(frame, TTSAudioRawFrame):
        # Convert TTS audio to input audio format
        return InputAudioRawFrame(
            audio=frame.audio,
            sample_rate=frame.sample_rate,
            num_channels=frame.num_channels
        )
    return frame

producer = ProducerProcessor(
    filter=is_tts_audio,
    transformer=tts_to_input_audio_transformer
    passthrough=True  # Keep these frames in original pipeline
)

# Create a consumer to receive the frames
consumer = ConsumerProcessor(
    producer=producer,
    direction=FrameDirection.DOWNSTREAM
)

# Use in a ParallelPipeline
pipeline = Pipeline([
    transport.input(),
    ParallelPipeline(
        # Branch 1: LLM for bot responses
        [
            llm,
            tts,
            producer,  # Capture TTS audio here
        ],
        # Branch 2: Audio processing branch
        [
            consumer,  # Receive TTS audio here
            llm, # Speech-to-Speech LLM (audio in)
        ]
    ),
    transport.output(),
])
```


# MCPClient
Source: https://docs.pipecat.ai/server/utilities/mcp/mcp

Service to connect to MCP (Model Context Protocol) servers

## Overview

MCP is an open standard for enabling AI agents to interact with external data and tools. `MCPClient` provides a way to access and call tools via MCP. For example, instead of writing bespoke function call implementations for an external API, you may use an MCP server that provides a bridge to the API. *Be aware there may be security implications.* See [MCP documenation](https://github.com/modelcontextprotocol) for more details.

## Installation

To use `MCPClient`, install the required dependencies:

```bash theme={null}
pip install "pipecat-ai[mcp]"
```

You may also need to set environment variables as required by the specific MCP server to which you are connecting.

## Configuration

### Constructor Parameters

You can connect to your MCP server via Stdio or SSE transport. See [here](https://modelcontextprotocol.io/docs/concepts/transports#built-in-transport-types) for more documentation on MCP transports.

<ParamField type="str | StdioServerParameters">
  You can provide either:

  * URL: "[https://your.mcp.server/sse](https://your.mcp.server/sse)"
  * StdioServerParameters, which are defined as:

  ```python theme={null}
    StdioServerParameters(
          command="python",  # Executable
          args=["example_server.py"],  # Optional command line arguments
          env=None,  # Optional environment variables
      )
  ```
</ParamField>

### Input Parameters

See more information regarding server params [here](https://github.com/modelcontextprotocol/python-sdk?tab=readme-ov-file#writing-mcp-clients).

## Usage Example

### MCP Stdio Transport Implementation

```python theme={null}

# Import MCPClient and StdioServerParameters
...
from mcp import StdioServerParameters
from pipecat.services.mcp_service import MCPClient
...

# Initialize an LLM
llm = ...

# Initialize and configure MCPClient with server parameters
mcp = MCPClient(
        server_params=StdioServerParameters(
            command=shutil.which("npx"),
            args=["-y", "@name/mcp-server-name@latest"],
            env={"ENV_API_KEY": "<env_api_key>"},
        )
    )

# Create tools schema from the MCP server and register them with llm
tools = await mcp.register_tools(llm)

# Create context with system message and tools
# Tip: Let the LLM know it has access to tools from an MCP server by including it in the system prompt.
context = LLMContext(
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant in a voice conversation. You have access to MCP tools. Keep responses concise."
        }
    ],
    tools=tools
)
```

### MCP SSE Transport Implementation

```python theme={null}

# Import MCPClient
...
from pipecat.services.mcp_service import MCPClient
...

# Initialize an LLM
llm = ...

# Initialize and configure MCPClient with MCP SSE server url
mcp = MCPClient(server_params="https://your.mcp.server/sse")

# Create tools schema from the MCP server and register them with llm
tools = await mcp.register_tools(llm)

# Create context with system message and tools
# Tip: Let the LLM know it has access to tools from an MCP server by including it in the system prompt.
context = LLMContext(
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant in a voice conversation. You have access to MCP tools. Keep responses concise."
        }
    ],
    tools=tools
)
```

## Methods

<ResponseField name="register_tools" type="async method">
  Converts MCP tools to Pipecat-friendly function definitions and registers the
  functions with the llm.
</ResponseField>

```python theme={null}
async def register_tools(self, llm) -> ToolsSchema:
```

## Additional documentation

<Note>
  See [MCP's docs](https://github.com/modelcontextprotocol/python-sdk) for MCP
  related updates.
</Note>


# Debug Log Observer
Source: https://docs.pipecat.ai/server/utilities/observers/debug-observer

Comprehensive frame logging with configurable filtering in Pipecat

The `DebugLogObserver` provides detailed logging of frame activity in your Pipecat pipeline, with full visibility into frame content and flexible filtering options.

## Features

* Log all frame types and their content
* Filter by specific frame types
* Filter by source or destination components
* Automatic formatting of frame fields
* Special handling for complex data structures

## Usage

### Log All Frames

Log all frames passing through the pipeline:

```python theme={null}
from pipecat.observers.loggers.debug_log_observer import DebugLogObserver

task = PipelineTask(
    pipeline,
    params=PipelineParams(
        observers=[DebugLogObserver()],
    ),
)
```

### Filter by Frame Types

Log only specific frame types:

```python theme={null}
from pipecat.frames.frames import TranscriptionFrame, InterimTranscriptionFrame
from pipecat.observers.loggers.debug_log_observer import DebugLogObserver

task = PipelineTask(
    pipeline,
    params=PipelineParams(
        observers=[
            DebugLogObserver(frame_types=(
                TranscriptionFrame,
                InterimTranscriptionFrame
            ))
        ],
    ),
)
```

### Advanced Source/Destination Filtering

Filter frames based on their type and source/destination:

```python theme={null}
from pipecat.frames.frames import StartInterruptionFrame, UserStartedSpeakingFrame, LLMTextFrame
from pipecat.observers.loggers.debug_log_observer import DebugLogObserver, FrameEndpoint
from pipecat.transports.base_output_transport import BaseOutputTransport
from pipecat.services.stt_service import STTService

task = PipelineTask(
    pipeline,
    params=PipelineParams(
        observers=[
            DebugLogObserver(frame_types={
                # Only log StartInterruptionFrame when source is BaseOutputTransport
                StartInterruptionFrame: (BaseOutputTransport, FrameEndpoint.SOURCE),

                # Only log UserStartedSpeakingFrame when destination is STTService
                UserStartedSpeakingFrame: (STTService, FrameEndpoint.DESTINATION),

                # Log LLMTextFrame regardless of source or destination
                LLMTextFrame: None
            })
        ],
    ),
)
```

## Log Output Format

The observer logs each frame with its complete details:

```
[Source] → [Destination]: [FrameType] [field1: value1, field2: value2, ...] at [timestamp]s
```

For example:

```
OpenAILLMService#0 → DailyTransport#0: LLMTextFrame text: 'Hello, how can I help you today?' at 1.24s
```

## Configuration Options

| Parameter        | Type                                                                                   | Description                                                     |
| ---------------- | -------------------------------------------------------------------------------------- | --------------------------------------------------------------- |
| `frame_types`    | `Tuple[Type[Frame], ...]` or `Dict[Type[Frame], Optional[Tuple[Type, FrameEndpoint]]]` | Frame types to log, with optional source/destination filtering  |
| `exclude_fields` | `Set[str]`                                                                             | Field names to exclude from logging (defaults to binary fields) |

## FrameEndpoint Enum

The `FrameEndpoint` enum is used for source/destination filtering:

* `FrameEndpoint.SOURCE`: Filter by source component
* `FrameEndpoint.DESTINATION`: Filter by destination component


# LLM Log Observer
Source: https://docs.pipecat.ai/server/utilities/observers/llm-observer

Logging LLM activity in Pipecat

The `LLMLogObserver` provides detailed logging of Large Language Model (LLM) activity within your Pipecat pipeline. It tracks the entire lifecycle of LLM interactions, from initial prompts to final responses.

## Frame Types Monitored

The observer tracks the following frame types (only from/to LLM service):

* **LLMFullResponseStartFrame**: When the LLM begins generating a response
* **LLMFullResponseEndFrame**: When the LLM completes its response
* **LLMTextFrame**: Individual text chunks generated by the LLM
* **FunctionCallInProgressFrame**: Function/tool calls made by the LLM
* **LLMMessagesFrame**: Input messages sent to the LLM (deprecated in favor of context frames)
* **LLMContextFrame**: Updated context sent to the LLM
* **FunctionCallResultFrame**: Results returned from function calls

## Usage

```python theme={null}
from pipecat.observers.loggers.llm_log_observer import LLMLogObserver

task = PipelineTask(
    pipeline,
    params=PipelineParams(
        observers=[LLMLogObserver()],
    ),
)
```

## Log Output Format

The observer uses emojis and consistent formatting for easy log reading:

* 🧠 \[Source] → LLM START/END RESPONSE
* 🧠 \[Source] → LLM GENERATING: \[text]
* 🧠 \[Source] → LLM FUNCTION CALL: \[details]
* 🧠 → \[Destination] LLM MESSAGES FRAME: \[messages]
* 🧠 → \[Destination] LLM CONTEXT FRAME: \[context]

All log entries include timestamps for precise timing analysis.


# Observer Pattern
Source: https://docs.pipecat.ai/server/utilities/observers/observer-pattern

Understanding and implementing observers in Pipecat

The Observer pattern in Pipecat allows non-intrusive monitoring of frames as they flow through the pipeline. Observers can watch frame traffic without affecting the pipeline's core functionality.

<Warning>DEPRECATED: The old observer pattern with individual parameters (`on_push_frame(src, dst, frame, direction, timestamp)`) is deprecated. Use the new pattern with data objects (`on_push_frame(data: FramePushed)`) instead.</Warning>

## Base Observer

All observers must inherit from `BaseObserver` and can implement two main methods:

* `on_push_frame(data: FramePushed)`: Called when a frame is pushed from one processor to another
* `on_process_frame(data: FrameProcessed)`: Called when a frame is being processed by a processor

```python theme={null}
from pipecat.observers.base_observer import BaseObserver, FramePushed, FrameProcessed

class CustomObserver(BaseObserver):
    async def on_push_frame(self, data: FramePushed):
        # Your frame observation logic here
        pass
    
    async def on_process_frame(self, data: FrameProcessed):
        # Your frame processing observation logic here
        pass
```

## Available Observers

Pipecat provides several built-in observers:

* **LLMLogObserver**: Logs LLM activity and responses
* **TranscriptionLogObserver**: Logs speech-to-text transcription events
* **RTVIObserver**: Converts internal frames to RTVI protocol messages for server to client messaging

## Using Multiple Observers

You can attach multiple observers to a pipeline task. Each observer will be notified of all frames:

```python theme={null}
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        observers=[LLMLogObserver(), TranscriptionLogObserver(), CustomObserver()],
    ),
)
```

## Example: Debug Observer

Here's an example observer that logs interruptions and bot speaking events:

```python theme={null}
from pipecat.observers.base_observer import BaseObserver, FramePushed, FrameProcessed
from pipecat.frames.frames import (
    StartInterruptionFrame,
    BotStartedSpeakingFrame,
    BotStoppedSpeakingFrame,
)
from pipecat.processors.frame_processor import FrameDirection
from loguru import logger

class DebugObserver(BaseObserver):
    """Observer to log interruptions and bot speaking events to the console.

    Logs all frame instances of:
    - StartInterruptionFrame
    - BotStartedSpeakingFrame
    - BotStoppedSpeakingFrame

    This allows you to see the frame flow from processor to processor through the pipeline for these frames.
    Log format: [EVENT TYPE]: [source processor] → [destination processor] at [timestamp]s
    """

    async def on_push_frame(self, data: FramePushed):
        time_sec = data.timestamp / 1_000_000_000
        arrow = "→" if data.direction == FrameDirection.DOWNSTREAM else "←"

        if isinstance(data.frame, StartInterruptionFrame):
            logger.info(f"⚡ INTERRUPTION START: {data.source} {arrow} {data.destination} at {time_sec:.2f}s")
        elif isinstance(data.frame, BotStartedSpeakingFrame):
            logger.info(f"🤖 BOT START SPEAKING: {data.source} {arrow} {data.destination} at {time_sec:.2f}s")
        elif isinstance(data.frame, BotStoppedSpeakingFrame):
            logger.info(f"🤖 BOT STOP SPEAKING: {data.source} {arrow} {data.destination} at {time_sec:.2f}s")
```

## Common Use Cases

Observers are particularly useful for:

* Debugging frame flow
* Logging specific events
* Monitoring pipeline behavior
* Collecting metrics
* Converting internal frames to external messages


# Transcription Log Observer
Source: https://docs.pipecat.ai/server/utilities/observers/transcription-observer

Logging speech-to-text transcription activity in Pipecat

The `TranscriptionLogObserver` logs all speech-to-text transcription activity in your Pipecat pipeline, providing visibility into both final and interim transcription results.

## Frame Types Monitored

The observer tracks the following frame types (only from STT service):

* **TranscriptionFrame**: Final transcription results
* **InterimTranscriptionFrame**: In-progress transcription results

## Usage

```python theme={null}
from pipecat.observers.loggers.transcription_log_observer import TranscriptionLogObserver

task = PipelineTask(
    pipeline,
    params=PipelineParams(
        observers=[TranscriptionLogObserver()],
    ),
)
```

## Log Output Format

The observer uses consistent formatting with emoji indicators:

* 💬 \[Source] → TRANSCRIPTION: \[text] from \[user\_id]
* 💬 \[Source] → INTERIM TRANSCRIPTION: \[text] from \[user\_id]

All log entries include timestamps for precise timing analysis.


# Turn Tracking Observer
Source: https://docs.pipecat.ai/server/utilities/observers/turn-tracking-observer

Track conversation turns and events in your Pipecat pipeline

The `TurnTrackingObserver` monitors and tracks conversational turns in your Pipecat pipeline, providing events when turns start and end. It intelligently identifies when a user-bot interaction cycle begins and completes.

## Turn Lifecycle

A turn represents a complete user-bot interaction cycle:

1. **Start**: When the user starts speaking (or pipeline starts for first turn)
2. **Processing**: User speaks, bot processes and responds
3. **End**: After the bot finishes speaking and either:
   * The user starts speaking again
   * A timeout period elapses with no further activity

## Events

The observer emits two main events:

* **`on_turn_started`**: When a new turn begins
  * Parameters: `turn_number` (int)
* **`on_turn_ended`**: When a turn completes
  * Parameters: `turn_number` (int), `duration` (float, in seconds), `was_interrupted` (bool)

## Usage

The observer is automatically created when you initialize a `PipelineTask` with `enable_turn_tracking=True` (which is the default):

```python theme={null}
task = PipelineTask(
    pipeline,
    # Turn tracking is enabled by default
)

# Access the observer
turn_observer = task.turn_tracking_observer

# Register event handlers
@turn_observer.event_handler("on_turn_started")
async def on_turn_started(observer, turn_number):
    logger.info(f"Turn {turn_number} started")

@turn_observer.event_handler("on_turn_ended")
async def on_turn_ended(observer, turn_number, duration, was_interrupted):
    status = "interrupted" if was_interrupted else "completed"
    logger.info(f"Turn {turn_number} {status} in {duration:.2f}s")
```

## Configuration

You can configure the observer's behavior when creating a `PipelineTask`:

```python theme={null}
from pipecat.observers.turn_tracking_observer import TurnTrackingObserver

# Create a custom observer instance
custom_turn_tracker = TurnTrackingObserver(
    turn_end_timeout_secs=3.5,     # Turn end timeout (default: 2.5)
)

# Add it as a regular observer
task = PipelineTask(
    pipeline,
    observers=[custom_turn_tracker],
    # Disable the default one if adding your own
    enable_turn_tracking=False,
)
```

## Interruptions

The observer automatically detects interruptions when the user starts speaking while the bot is still speaking. In this case:

* The current turn is marked as interrupted (`was_interrupted=True`)
* A new turn begins immediately

## How It Works

The observer monitors specific frame types to track conversation flow:

* **StartFrame**: Initiates the first turn
* **UserStartedSpeakingFrame**: Starts user speech or triggers a new turn
* **BotStartedSpeakingFrame**: Marks bot speech beginning
* **BotStoppedSpeakingFrame**: Starts the turn end timeout

After a bot stops speaking, the observer waits for the configured timeout period. If no further bot speech occurs, the turn ends; otherwise, it continues as part of the same turn.

## Use Cases

* **Analytics**: Measure turn durations, interruption rates, and conversation flow
* **Logging**: Record turn-based logs for diagnostics and analysis
* **Visualization**: Show turn-based conversation timelines in UIs
* **Tracing**: Group spans and metrics by conversation turns


# User-Bot Latency Observer
Source: https://docs.pipecat.ai/server/utilities/observers/user-bot-latency-observer

Measure response time between user speech and bot responses in Pipecat

The `UserBotLatencyLogObserver` measures the time between when a user stops speaking and when the bot starts responding, providing metrics for conversational AI performance optimization.

## Features

* Tracks user speech start/stop timing
* Measures bot response latency
* Calculates statistics: average, minimum, maximum
* Provides real-time latency logging
* Automatically resets between conversation turns

## Usage

### Basic Latency Monitoring

Add latency monitoring to your pipeline:

```python theme={null}
from pipecat.observers.loggers.user_bot_latency_log_observer import UserBotLatencyLogObserver

task = PipelineTask(
    pipeline,
    params=PipelineParams(observers=[UserBotLatencyLogObserver()]),
)
```

## How It Works

The observer tracks conversation flow through these key events:

1. **User starts speaking** → Resets latency tracking
2. **User stops speaking** → Records timestamp
3. **Bot starts speaking** → Calculates and logs latency
4. **Pipeline ends** → Reports session latency statistics

## Log Output

### Real-time Latency Logs

During conversation, each response latency is logged:

```
⏱️ LATENCY FROM USER STOPPED SPEAKING TO BOT STARTED SPEAKING: 1.234s
```

### Final Statistics

When the pipeline ends, comprehensive statistics are reported:

```
⏱️ LATENCY FROM USER STOPPED SPEAKING TO BOT STARTED SPEAKING - Avg: 1.456s, Min: 0.892s, Max: 2.103s
```

## Limitations

* Only measures speech-to-speech latency (not text processing time)
* Requires proper frame sequencing to work accurately


# OpenTelemetry Tracing
Source: https://docs.pipecat.ai/server/utilities/opentelemetry

Monitor and analyze your Pipecat conversational pipelines using OpenTelemetry

## Overview

Pipecat includes built-in support for OpenTelemetry tracing, allowing you to gain deep visibility into your voice applications. Tracing helps you:

* Track latency and performance across your conversation pipeline
* Monitor service health and identify bottlenecks
* Visualize conversation turns and service dependencies
* Collect usage metrics and operational analytics

## Installation

To use OpenTelemetry tracing with Pipecat, install the tracing dependencies:

```bash theme={null}
pip install "pipecat-ai[tracing]"
```

<Tip>
  For local development and testing, we recommend using Jaeger as a trace collector.
  You can run it with Docker:

  ```bash theme={null}
  docker run -d --name jaeger \
    -p 16686:16686 \
    -p 4317:4317 \
    -p 4318:4318 \
    jaegertracing/all-in-one:latest
  ```

  Then access the UI at [http://localhost:16686](http://localhost:16686)
</Tip>

## Basic Setup

Enabling tracing in your Pipecat application requires two steps:

1. **Initialize the OpenTelemetry SDK** with your preferred exporter
2. **Enable tracing in your PipelineTask**

```python theme={null}
import os
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from pipecat.utils.tracing.setup import setup_tracing
from pipecat.pipeline.task import PipelineTask, PipelineParams

# Step 1: Initialize OpenTelemetry with your chosen exporter
exporter = OTLPSpanExporter(
    endpoint="http://localhost:4317",  # Jaeger or other collector endpoint
    insecure=True,
)

setup_tracing(
    service_name="my-voice-app",
    exporter=exporter,
    console_export=False,  # Set to True for debug output
)

# Step 2: Enable tracing in your PipelineTask
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        enable_metrics=True,                              # Required for some service metrics
    ),
    enable_tracing=True,                                  # Enable tracing for this task
    enable_turn_tracking=True,                            # Enable turn tracking for this task
    conversation_id="customer-123",                       # Optional - will auto-generate if not provided
    additional_span_attributes={"session.id": "abc-123"} # Optional - additional attributes to attach to the otel span
)
```

<Info>
  For complete working examples, see our sample implementations:

  * [Jaeger Tracing Example](https://github.com/pipecat-ai/pipecat-examples/tree/main/open-telemetry/jaeger) - Uses gRPC exporter with Jaeger
  * [Langfuse Tracing Example](https://github.com/pipecat-ai/pipecat-examples/tree/main/open-telemetry/langfuse) - Uses HTTP exporter with Langfuse for LLM-focused observability
</Info>

## Trace Structure

Pipecat organizes traces hierarchically, following the natural structure of conversations:

```
Conversation (conversation)
├── turn
│   ├── stt
│   ├── llm
│   └── tts
└── turn
    ├── stt
    ├── llm
    └── tts
    turn...
```

For real-time multimodal services like Gemini Live and OpenAI Realtime, the structure adapts to their specific patterns:

```
Conversation (conversation)
├── turn
│   ├── llm_setup (session configuration)
│   ├── stt (user input)
│   ├── llm_response (complete response with usage)
│   └── llm_tool_call/llm_tool_result (for function calls)
└── turn
    ├── stt (user input)
    └── llm_response (complete response)
    turn...
```

This hierarchical structure makes it easy to:

* Track the full lifecycle of a conversation
* Measure latency for individual turns
* Identify which services are contributing to delays
* Compare performance across different conversations

## Exporter Options

Pipecat supports any OpenTelemetry-compatible exporter. Common options include:

### OTLP Exporter (for Jaeger, Grafana, etc.)

```python theme={null}
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(
    endpoint="http://localhost:4317",  # Your collector endpoint
    insecure=True,  # Use False for TLS connections
)
```

### HTTP OTLP Exporter (for Langfuse, etc.)

```python theme={null}
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(
    # Configure with environment variables:
    # OTEL_EXPORTER_OTLP_ENDPOINT
    # OTEL_EXPORTER_OTLP_HEADERS
)
```

See our [Langfuse example](https://github.com/pipecat-ai/pipecat-examples/tree/main/open-telemetry/langfuse) for details on configuring this exporter.

### Console Exporter (for debugging)

The console exporter can be enabled alongside any other exporter by setting `console_export=True`:

```python theme={null}
setup_tracing(
    service_name="my-voice-app",
    exporter=otlp_exporter,
    console_export=True,  # Prints traces to stdout
)
```

### Cloud Provider Exporters

Many cloud providers offer OpenTelemetry-compatible observability services:

* **AWS X-Ray**
* **Google Cloud Trace**
* **Azure Monitor**
* **Datadog APM**

Check the OpenTelemetry documentation for specific exporter configurations: [OpenTelemetry Vendors](https://opentelemetry.io/ecosystem/vendors/)

### OpenInference

Arize-ai provides OpenInference instrumentation, compatible with OpenTelemetry.

* [openinference-instrumentation-pipecat](https://github.com/Arize-ai/openinference/tree/main/python/instrumentation/openinference-instrumentation-pipecat)

See [Observability community integrations](https://docs.pipecat.ai/server/services/community-integrations#observability) for more details.

## Span Attributes

Pipecat enriches spans with detailed attributes about service operations:

### TTS Service Spans

* `gen_ai.system`: Service provider (e.g., "cartesia")
* `gen_ai.request.model`: Model ID/name
* `voice_id`: Voice identifier
* `text`: The text being synthesized
* `metrics.character_count`: Number of characters in the text
* `metrics.ttfb`: Time to first byte in seconds
* `settings.*`: Service-specific configuration parameters

### STT Service Spans

* `gen_ai.system`: Service provider (e.g., "deepgram")
* `gen_ai.request.model`: Model ID/name
* `transcript`: The transcribed text
* `is_final`: Whether the transcription is final
* `language`: Detected or configured language
* `vad_enabled`: Whether voice activity detection is enabled
* `metrics.ttfb`: Time to first byte in seconds
* `settings.*`: Service-specific configuration parameters

### LLM Service Spans

* `gen_ai.system`: Service provider (e.g., "openai", "gcp.gemini")
* `gen_ai.request.model`: Model ID/name
* `gen_ai.operation.name`: Operation type (e.g., "chat")
* `stream`: Whether streaming is enabled
* `input`: JSON-serialized input messages
* `output`: Complete response text
* `tools`: JSON-serialized tools configuration
* `tools.count`: Number of tools available
* `tools.names`: Comma-separated tool names
* `system`: System message content
* `gen_ai.usage.input_tokens`: Number of prompt tokens
* `gen_ai.usage.output_tokens`: Number of completion tokens
* `metrics.ttfb`: Time to first byte in seconds
* `gen_ai.request.*`: Standard parameters (temperature, max\_tokens, etc.)

### Multimodal Service Spans (Gemini Live & OpenAI Realtime)

#### Setup Spans

* `gen_ai.system`: "gcp.gemini" or "openai"
* `gen_ai.request.model`: Model identifier
* `tools.count`: Number of available tools
* `tools.definitions`: JSON-serialized tool schemas
* `system_instruction`: System prompt (truncated)
* `session.*`: Session configuration parameters

#### Request Spans (OpenAI Realtime)

* `input`: JSON-serialized context messages being sent
* `gen_ai.operation.name`: "llm\_request"

#### Response Spans

* `output`: Complete assistant response text
* `output_modality`: "TEXT" or "AUDIO" (Gemini Live)
* `gen_ai.usage.input_tokens`: Prompt tokens used
* `gen_ai.usage.output_tokens`: Completion tokens generated
* `function_calls.count`: Number of function calls made
* `function_calls.names`: Comma-separated function names
* `metrics.ttfb`: Time to first response in seconds

#### Tool Call/Result Spans (Gemini Live)

* `tool.function_name`: Name of the function being called
* `tool.call_id`: Unique identifier for the call
* `tool.arguments`: Function arguments (truncated)
* `tool.result`: Function execution result (truncated)
* `tool.result_status`: "completed", "error", or "parse\_error"

### Turn Spans

* `turn.number`: Sequential turn number
* `turn.type`: Type of turn (e.g., "conversation")
* `turn.duration_seconds`: Duration of the turn
* `turn.was_interrupted`: Whether the turn was interrupted
* `conversation.id`: ID of the parent conversation

### Conversation Spans

* `conversation.id`: Unique identifier for the conversation
* `conversation.type`: Type of conversation (e.g., "voice")

## Usage Metrics

Pipecat's tracing implementation automatically captures usage metrics for LLM and TTS services:

### LLM Token Usage

Token usage is captured in LLM spans as:

* `gen_ai.usage.input_tokens`
* `gen_ai.usage.output_tokens`

### TTS Character Count

Character count is captured in TTS spans as:

* `metrics.character_count`

## Performance Metrics

Pipecat traces capture key performance metrics for each service:

### Time To First Byte (TTFB)

The time it takes for a service to produce its first response:

* `metrics.ttfb` (in seconds)

### Processing Duration

The total time spent processing in each service is captured in the span duration.

## Configuration Options

### PipelineTask Parameters

<ParamField type="bool">
  Enable or disable tracing for the pipeline
</ParamField>

<ParamField type="bool">
  Whether to enable turn tracking.
</ParamField>

<ParamField type="Optional[str]">
  Custom ID for the conversation. If not provided, a UUID will be generated
</ParamField>

<ParamField type="Optional[dict]">
  Any additional attributes to add to top-level OpenTelemetry conversation span.
</ParamField>

### setup\_tracing() Parameters

<ParamField type="str">
  Name of the service for traces
</ParamField>

<ParamField type="Optional[SpanExporter]">
  A pre-configured OpenTelemetry span exporter instance
</ParamField>

<ParamField type="bool">
  Whether to also export traces to console (useful for debugging)
</ParamField>

## Example

Here's a complete example showing OpenTelemetry tracing setup with Jaeger:

```python theme={null}
import os
from dotenv import load_dotenv
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.services.deepgram.stt import DeepgramSTTService
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.services.cartesia.tts import CartesiaTTSService
from pipecat.utils.tracing.setup import setup_tracing

load_dotenv()

# Initialize tracing if enabled
if os.getenv("ENABLE_TRACING"):
    # Create the exporter
    otlp_exporter = OTLPSpanExporter(
        endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317"),
        insecure=True,
    )

    # Set up tracing with the exporter
    setup_tracing(
        service_name="pipecat-demo",
        exporter=otlp_exporter,
        console_export=bool(os.getenv("OTEL_CONSOLE_EXPORT")),
    )

# Create your services
stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"))
tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    voice_id="your-voice-id"
)

# Build pipeline
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),
    context_aggregator.assistant(),
])

# Create pipeline task with tracing enabled
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        enable_metrics=True,
        enable_usage_metrics=True,
    ),
    enable_tracing=True,
    enable_turn_tracking=True,
    conversation_id="customer-123",                       # Optional - will auto-generate if not provided
    additional_span_attributes={"session.id": "abc-123"} # Optional - additional attributes to attach to the otel span
)

# Run the pipeline
runner = PipelineRunner()
await runner.run(task)
```

## Troubleshooting

If you're having issues with tracing:

* **No Traces Visible**: Ensure the OpenTelemetry packages are installed and that your collector endpoint is correct
* **Missing Service Data**: Verify that `enable_metrics=True` is set in PipelineParams
* **Debugging Tracing**: Enable console export with `console_export=True` to view traces in your logs
* **Connection Errors**: Check network connectivity to your trace collector
* **Collector Configuration**: Verify your collector is properly set up to receive traces

## References

* [OpenTelemetry Python Documentation](https://opentelemetry-python.readthedocs.io/)
* [OpenTelemetry Tracing Specification](https://opentelemetry.io/docs/reference/specification/trace/api/)
* [Jaeger Documentation](https://www.jaegertracing.io/docs/latest/)
* [Langfuse OpenTelemetry Documentation](https://langfuse.com/docs/opentelemetry/get-started)


# Development Runner
Source: https://docs.pipecat.ai/server/utilities/runner/guide

Unified runner for building voice AI bots with Daily, WebRTC, and telephony transports

## Overview

The Pipecat development runner provides a unified way to run voice AI bots across multiple transport types. It handles infrastructure setup - creating Daily rooms, managing WebRTC connections, and routing telephony calls.

## Installation

```bash theme={null}
pip install pipecat-ai[runner]
```

## What is a Runner?

A runner in Pipecat refers to a "bot runner", an HTTP service that provides a gateway for spawning bots on-demand. It's the component that enables your bot to run by providing it with server infrastructure and connection details like rooms and tokens.

A bot runner typically creates transport sessions (like Daily WebRTC rooms), generates authentication tokens for both bots and users, spawns new bot processes when users request sessions, and manages bot lifecycle and cleanup. Think of it as the bridge between incoming user connections and your bot logic.

## How the Development Runner Works

The development runner operates as a FastAPI web server that automatically discovers and executes your bot code. When you start the runner, it creates the necessary web endpoints and infrastructure for your chosen transport type.

* **WebRTC connections**: It serves a built-in web interface where users can connect directly as well as an endpoint to create new WebRTC sessions
* **Daily integration**: It provides endpoints that create new rooms and tokens and redirect users to join them
* **Telephony providers**: For Twilio, it sets up webhook endpoints that handle incoming calls and establish WebSocket connections for audio streaming

The runner automatically detects which transport type you're using and configures the appropriate infrastructure. It then discovers your bot function and spawns new instances whenever users connect. This means you can focus on writing your bot logic while the runner handles all the server infrastructure, connection management, and transport-specific details.

Your bot code receives runner arguments that contain everything it needs, including Daily room URLs and tokens, WebRTC connections, or WebSocket streams for telephony. The runner abstracts away the complexity of managing these different connection types, providing a unified interface for building bots that work across multiple platforms.

## Pipecat Cloud Ready

The bot runner is designed to be cloud-ready, meaning that you can run the same bot code locally and deployed to Pipecat Cloud without any modifications. It automatically handles the differences in transport setup, providing you with the flexibility to test locally using an open source transport, like SmallWebRTCTransport, but run in production using Daily or telephony transports.

## Building with the Runner

Now let's build a practical example to see how this works. The key insight is that your bot code is structured into two parts: the core bot logic that works with any transport, and the entry point that creates the appropriate transport based on the runner arguments.

Here's the basic structure:

```python theme={null}
# Your imports
from pipecat.runner.types import RunnerArguments
from pipecat.transports.base_transport import BaseTransport, TransportParams
from pipecat.transports.network.small_webrtc import SmallWebRTCTransport

async def run_bot(transport: BaseTransport):
    """Your core bot logic here:
    - Define services (STT, TTS, LLM)
    - Initialize messages and context
    - Create the pipeline
    - Add event handlers
    - Run the pipeline
    """

    # Your bot logic goes here
    # Define STT, LLM, TTS...
    # ...
    # Run your pipeline
    await runner.run(task)

async def bot(runner_args: RunnerArguments):
    """Main bot entry point compatible for local dev and Pipecat Cloud."""
    transport = SmallWebRTCTransport(
        params=TransportParams(
            audio_in_enabled=True,
            audio_out_enabled=True,
        ),
        webrtc_connection=runner_args.webrtc_connection,
    )

    await run_bot(transport)

if __name__ == "__main__":
    from pipecat.runner.run import main
    main()
```

The `run_bot()` function contains your actual bot logic and is transport-agnostic. The `bot()` function is the entry point that the runner calls - it creates the appropriate transport and passes it to your bot logic. This separation allows the same bot code to work across different transports.

When you run this with `python bot.py`, the development runner starts a web server and opens a browser interface at `http://localhost:7860/client`. Each time someone connects, the runner calls your `bot()` function with WebRTC runner arguments.

## Supporting Multiple Transports

To make your bot work across different platforms, you can detect the transport type from the runner arguments and create the appropriate transport. Here's how to support both Daily and WebRTC:

```python theme={null}
from pipecat.runner.types import DailyRunnerArguments, RunnerArguments, SmallWebRTCRunnerArguments

async def bot(runner_args: RunnerArguments):
    """Main bot entry point compatible with Pipecat Cloud."""

    transport = None

    if isinstance(runner_args, DailyRunnerArguments):
        from pipecat.transports.daily.transport import DailyParams, DailyTransport

        transport = DailyTransport(
            runner_args.room_url,
            runner_args.token,
            "Pipecat Bot",
            params=DailyParams(
                audio_in_enabled=True,
                audio_out_enabled=True,
            ),
        )

    elif isinstance(runner_args, SmallWebRTCRunnerArguments):
        from pipecat.transports.base_transport import TransportParams
        from pipecat.transports.network.small_webrtc import SmallWebRTCTransport

        transport = SmallWebRTCTransport(
            params=TransportParams(
                audio_in_enabled=True,
                audio_out_enabled=True,
            ),
            webrtc_connection=runner_args.webrtc_connection,
        )

    else:
        logger.error(f"Unsupported runner arguments type: {type(runner_args)}")
        return

    if transport is None:
        logger.error("Failed to create transport")
        return

    await run_bot(transport)

if __name__ == "__main__":
    from pipecat.runner.run import main
    main()
```

Now you can run your bot with different transports:

```bash theme={null}
python bot.py -t webrtc  # Uses SmallWebRTCRunnerArguments
python bot.py -t daily   # Uses DailyRunnerArguments
```

### Understanding Runner Arguments

Runner arguments are how the runner communicates transport-specific information to your bot. The runner determines which transport to use based on the command-line arguments, then creates the appropriate runner arguments:

* **`DailyRunnerArguments`**: Contains `room_url`, `token` (Optional), `body` (Optional) for joining Daily rooms
* **`SmallWebRTCRunnerArguments`**: Contains `webrtc_connection` for local WebRTC sessions
* **`WebSocketRunnerArguments`**: Contains `websocket` for telephony connections

All runner arguments also include:

* **`handle_sigint`**: Whether the bot should handle SIGINT (Ctrl+C) signals (managed automatically)
* **`handle_sigterm`**: Whether the bot should handle SIGTERM signals (managed automatically)

<Warning>
  `handle_sigint` and `handle_sigterm` are development only features and cannot
  be used when deploying to Pipecat Cloud.
</Warning>

The runner handles all the complex setup - creating Daily rooms, generating tokens, establishing WebSocket connections - and provides your bot with everything it needs through these runner arguments.

Notice how we use lazy imports (`from pipecat.transports.daily.transport import ...`) inside the conditional blocks. This ensures that transport-specific dependencies are only required when that transport is actually used, making your bot more portable.

<Info>
  `RunnerArguments` is the base class for all runner arguments. It provides a
  common interface for the runner to pass transport-specific information to your
  bot.
</Info>

## Environment Detection

When building bots that work both locally and in production, you often need to detect the execution environment to enable different features. The development runner sets the `ENV` environment variable to help with this:

```python theme={null}
import os

async def bot(runner_args: RunnerArguments):
    # Check if running in local development environment
    is_local = os.environ.get("ENV") == "local"

    # Enable production features only when deployed
    if not is_local:
        from pipecat.audio.filters.krisp_viva_filter import KrispVivaFilter
        krisp_filter = KrispVivaFilter()
    else:
        krisp_filter = None

    transport = DailyTransport(
        runner_args.room_url,
        runner_args.token,
        "Pipecat Bot",
        params=DailyParams(
            audio_in_filter=krisp_filter,  # Krisp VIVA filter only in production
            audio_in_enabled=True,
            audio_out_enabled=True,
        ),
    )
```

### Environment Values

The development runner automatically sets environment variables based on how your bot is running:

* **Local development**: `ENV=local` (set by the development runner)
* **Production/Cloud deployment**: `ENV` is not set or has a different value

This allows you to easily customize behavior between development and production environments:

## All Supported Transports

The development runner supports six transport types, each designed for different use cases:

### WebRTC (`-t webrtc`)

Local WebRTC connections with a built-in browser interface. Perfect for development and testing.

```bash theme={null}
python bot.py -t webrtc
# Opens http://localhost:7860/client

# ESP32 compatibility mode
python bot.py -t webrtc --esp32 --host 192.168.1.100
```

**Runner Arguments**: `SmallWebRTCRunnerArguments`

* `webrtc_connection`: Pre-configured WebRTC peer connection

### Daily (`-t daily`)

Integration with Daily for production video conferencing with rooms, participant management, and Pipecat client compatibility.

```bash theme={null}
python bot.py -t daily
# Opens http://localhost:7860 with room creation interface
# Also provides POST /start endpoint for Pipecat clients

# Direct connection for testing (bypasses web server)
python bot.py -d

# Enable PSTN dial-in webhook handling
python bot.py -t daily --dialin
```

**Runner Arguments**: `DailyRunnerArguments`

* `room_url`: Daily room URL to join
* `token`: Authentication token for the room
* `body`: Request data from /start endpoint (dict) or dial-in webhook data

**Available Endpoints**:

* `GET /`: Web interface for creating rooms
* `POST /start`: RTVI-compatible endpoint for programmatic access
* `POST /daily-dialin-webhook`: PSTN dial-in webhook handler (requires `--dialin` flag)

### Telephony (`-t twilio|telnyx|plivo|exotel`)

Phone call integration through telephony providers. Requires a public webhook endpoint and provider credentials.

```bash theme={null}
# Set up environment variables first
export TWILIO_ACCOUNT_SID=your_account_sid
export TWILIO_AUTH_TOKEN=your_auth_token

# Then run with public proxy
python bot.py -t twilio -x yourproxy.ngrok.io
```

**Environment Variables**:

* Twilio: `TWILIO_ACCOUNT_SID`, `TWILIO_AUTH_TOKEN`
* Telnyx: `TELNYX_API_KEY`
* Plivo: `PLIVO_AUTH_ID`, `PLIVO_AUTH_TOKEN`
* Exotel: None required (no hang-up functionality available)

Environment variables are optional, but when provided will attempt to hang up the call when the session ends.

**Runner Arguments**: `WebSocketRunnerArguments`

* `websocket`: WebSocket connection for audio streaming

The runner automatically detects the telephony provider from incoming WebSocket messages and configures the appropriate serializers and audio settings. Your bot receives a pre-configured FastAPI WebSocket connection ready for telephony audio streaming.

## Command Line Options

The development runner accepts several command-line arguments to customize its behavior:

```bash theme={null}
python bot.py [OPTIONS]

Options:
  --host TEXT          Server host address (default: localhost)
  --port INTEGER       Server port (default: 7860)
  -t, --transport      Transport type: daily, webrtc, twilio, telnyx, plivo, exotel (default: webrtc)
  -x, --proxy TEXT     Public proxy hostname for telephony webhooks (required for telephony)
  --esp32              Enable SDP munging for ESP32 WebRTC compatibility
  -d, --direct         Connect directly to Daily room for testing (automatically sets transport to daily)
  --dialin             Enable Daily PSTN dial-in webhook handling (requires Daily transport)
  -v, --verbose        Increase logging verbosity
```

### Key Arguments

**`--transport` / `-t`**: Determines which transport infrastructure to set up

* `webrtc`: Local WebRTC with browser interface
* `daily`: Daily.co integration with room management
* `twilio`, `telnyx`, `plivo`, `exotel`: Telephony provider integration

**`--proxy` / `-x`**: Required for most telephony transports (Twilio, Telnyx, Plivo). This should be a publicly accessible hostname (like `yourbot.ngrok.io`) that can receive webhooks from telephony providers. The runner automatically strips protocol prefixes (http\://, https\://) if provided. Not required for Exotel, which uses direct WebSocket connections.

**`--direct` / `-d`**: Special mode for Daily that bypasses the web server and connects your bot directly to a Daily room. Useful for quick testing but not recommended for production use.

**`--dialin`**: Enables the `/daily-dialin-webhook` endpoint for handling Daily PSTN dial-in calls. Only works with Daily transport (`-t daily`). This endpoint receives webhook data from Daily when a phone call dials into your configured phone number, creates a SIP-enabled room, and spawns your bot.

**`--esp32`**: Enables SDP (Session Description Protocol) modifications needed for ESP32 WebRTC compatibility. Must be used with a specific IP address via `--host`.

### Environment Variables

Different transports require different environment variables:

**Daily**:

* `DAILY_API_KEY`: Daily API key for creating rooms and tokens (required for dial-in)
* `DAILY_SAMPLE_ROOM_URL` (Optional): Existing room URL to use
* `DAILY_API_URL` (Optional): Daily API URL (defaults to [https://api.daily.co/v1](https://api.daily.co/v1))

**Telephony**:

* `TWILIO_ACCOUNT_SID`, `TWILIO_AUTH_TOKEN`: Twilio credentials
* `PLIVO_AUTH_ID`, `PLIVO_AUTH_TOKEN`: Plivo credentials
* `TELNYX_API_KEY`: Telnyx API key

The runner automatically uses these environment variables when creating transport sessions and authentication tokens.

## Simplifying with the Transport Utility

While the manual approach gives you full control, the `create_transport` utility provides a much cleaner way to handle multiple transports. Instead of writing conditional logic for each transport type, you define transport configurations upfront and let the utility handle the selection:

```python theme={null}
from pipecat.runner.utils import create_transport

# Define transport configurations using factory functions
transport_params = {
    "daily": lambda: DailyParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
    ),
    "webrtc": lambda: TransportParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
    ),
    "twilio": lambda: FastAPIWebsocketParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
        # add_wav_header and serializer handled automatically
    ),
}

async def bot(runner_args):
    """Simplified bot entry point using the transport utility."""
    transport = await create_transport(runner_args, transport_params)
    await run_bot(transport)
```

The utility automatically:

* Detects the telephony provider from WebSocket messages
* Configures the appropriate serializer (Twilio, Telnyx, Plivo, or Exotel)
* Sets up authentication using environment variables
* Handles WebRTC and Daily transport creation

Now your bot supports all six transport types with just two lines of code in the bot() function.

## RTVI Integration

The development runner provides RTVI (Real-Time Voice Interface) compatible endpoints for the Pipecat client SDKs. This allows you to use the Pipecat client libraries locally during development as well as when deployed to Pipecat Cloud.

### The `/start` Endpoint

For Daily transports, the runner automatically creates a `/start` POST endpoint that:

1. **Creates Daily rooms and tokens** using your `DAILY_API_KEY`
2. **Spawns bot instances** with request `body` data
3. **Returns connection details** in RTVI-compatible format

```bash theme={null}
# Start the Daily runner
python bot.py -t daily

# The /start endpoint is now available at:
# POST http://localhost:7860/start
```

**Request Format:**

```json theme={null}
{
  "createDailyRoom": true,
  "dailyRoomProperties": {
    "start_video_off": true,
    "start_audio_off": false
  },
  "body": {
    "custom_data": "your_value",
    "user_id": "user123"
  }
}
```

<Note>`dailyRoomProperties` are not yet handled by the runner.</Note>

**Response Format:**

```json theme={null}
{
  "dailyRoom": "https://domain.daily.co/room-name",
  "dailyToken": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9..."
}
```

### Accessing Request Data in Your Bot

The `body` field from the `/start` request is passed to your bot through `DailyRunnerArguments.body`:

```python theme={null}
async def bot(runner_args: DailyRunnerArguments):
    # Access custom data from the /start request
    custom_data = runner_args.body.get("custom_data")
    user_id = runner_args.body.get("user_id")

    print(f"Bot started for user: {user_id}")
    print(f"Custom data: {custom_data}")

    # Your bot logic here
    transport = DailyTransport(runner_args.room_url, runner_args.token, "Bot")
    await run_bot(transport)
```

### RTVI Client Example

You can use RTVI client libraries to connect to your local development runner:

```javascript theme={null}
import { PipecatClient } from "@pipecat-ai/client-js";
import { DailyTransport } from "@pipecat-ai/daily-transport";

const client = new PipecatClient({
  transport: new DailyTransport(),
  enableMic: true,
  enableCam: false,
});

// Start a session with custom data
await client.connect({ endpoint: "http://localhost:7860/start" });
```

<Info>
  The `/start` endpoint is only available for Daily transports (`-t daily`).
  WebRTC and telephony transports use different connection methods.
</Info>

## Daily Dial-In Webhook

The development runner can handle Daily PSTN dial-in webhooks when started with the `--dialin` flag. This allows you to test phone call integrations locally before deploying to production.

### Enabling Dial-In Support

```bash theme={null}
python bot.py -t daily --dialin
# Webhook endpoint available at:
# POST http://localhost:7860/daily-dialin-webhook
```

### How It Works

When a phone call dials into your Daily phone number:

1. **Daily sends a webhook** to your configured endpoint with call details
2. **The runner creates a SIP-enabled room** with appropriate configuration
3. **Your bot is spawned** with dial-in context in `runner_args.body`
4. **The call is connected** to the room where your bot is waiting

### Webhook Payload

Daily sends the following data to the `/daily-dialin-webhook` endpoint:

```json theme={null}
{
  "From": "+15551234567",
  "To": "+15559876543",
  "callId": "uuid-call-id",
  "callDomain": "uuid-call-domain",
  "sipHeaders": {}
}
```

### Accessing Dial-In Data in Your Bot

The dial-in webhook data is passed to your bot through `DailyRunnerArguments.body`. You need to parse it and configure the Daily transport with dial-in settings:

```python theme={null}
from pipecat.runner.types import DailyDialinRequest, RunnerArguments
from pipecat.transports.daily.transport import DailyDialinSettings, DailyParams, DailyTransport

async def bot(runner_args: RunnerArguments):
    # Parse the dial-in request from the runner
    request = DailyDialinRequest.model_validate(runner_args.body)

    # Configure Daily transport with dial-in settings
    daily_dialin_settings = DailyDialinSettings(
        call_id=request.dialin_settings.call_id,
        call_domain=request.dialin_settings.call_domain,
    )

    transport = DailyTransport(
        runner_args.room_url,
        runner_args.token,
        "Daily PSTN Dial-in Bot",
        params=DailyParams(
            api_key=request.daily_api_key,
            api_url=request.daily_api_url,
            dialin_settings=daily_dialin_settings,
            audio_in_enabled=True,
            audio_out_enabled=True,
        ),
    )

    # Log caller information if available
    if request.dialin_settings.From:
        print(f"Handling call from: {request.dialin_settings.From}")

    # Your bot logic here
    await run_bot(transport)
```

### Configuring Daily Phone Numbers

To test dial-in locally, you need to:

1. **Configure a Daily phone number** in your Daily dashboard
2. **Set the webhook URL** to your public endpoint (e.g., via ngrok)
3. **Ensure `DAILY_API_KEY` is set** in your environment

```bash theme={null}
# Example with ngrok
ngrok http 7860

# Configure webhook URL in Daily dashboard:
# https://your-subdomain.ngrok.io/daily-dialin-webhook
```

<Note>
  The `--dialin` flag requires the Daily transport (`-t daily`) and a valid
  `DAILY_API_KEY` environment variable.
</Note>

## Quick Reference

| Transport     | Command                                             | Access                                                       | Environment Variables                               |
| ------------- | --------------------------------------------------- | ------------------------------------------------------------ | --------------------------------------------------- |
| WebRTC        | `python bot.py`                                     | [http://localhost:7860/client](http://localhost:7860/client) | None                                                |
| Daily         | `python bot.py -t daily`                            | [http://localhost:7860](http://localhost:7860)               | `DAILY_API_KEY`, `DAILY_SAMPLE_ROOM_URL` (Optional) |
| Daily Direct  | `python bot.py -d`                                  | Direct connection                                            | `DAILY_API_KEY`, `DAILY_SAMPLE_ROOM_URL` (Optional) |
| Daily Dial-In | `python bot.py -t daily --dialin`                   | Phone calls via Daily PSTN                                   | `DAILY_API_KEY` (Required)                          |
| Twilio        | `python bot.py -t twilio -x proxy.ngrok.io`         | Phone calls                                                  | `TWILIO_ACCOUNT_SID`, `TWILIO_AUTH_TOKEN`           |
| Telnyx        | `python bot.py -t telnyx -x proxy.ngrok.io`         | Phone calls                                                  | `TELNYX_API_KEY`                                    |
| Plivo         | `python bot.py -t plivo -x proxy.ngrok.io`          | Phone calls                                                  | `PLIVO_AUTH_ID`, `PLIVO_AUTH_TOKEN`                 |
| Exotel        | `python bot.py -t exotel`                           | Phone calls                                                  | None                                                |
| ESP32 WebRTC  | `python bot.py -t webrtc --esp32 --host <ESP32_IP>` | ESP32 WebRTC connection                                      | None                                                |

## Examples

For practical examples of using the development runner with different transports, check out the following:

<Card title="Runner Examples" icon="code" href="https://github.com/pipecat-ai/pipecat-examples/tree/main/runner-examples">
  Explore the examples for different ways to use the development runner with
  various transports.
</Card>


# Transport Utilities
Source: https://docs.pipecat.ai/server/utilities/runner/transport-utils

Configuration and helper utilities for Daily, LiveKit, telephony, and WebRTC transports

## Overview

Pipecat provides several utility modules for configuring transports and handling transport-specific operations. While the [development runner](/server/utilities/runner/guide) handles most of these automatically, these utilities are useful for custom setups, advanced configurations, or when building your own deployment infrastructure.

## Daily Configuration

The Daily utilities handle room creation, token generation, and authentication setup for Daily integration. They support both standard video/audio rooms and SIP-enabled rooms for telephony integration.

### Basic Configuration

Use `configure()` for simple room and token setup:

```python theme={null}
import aiohttp
from pipecat.runner.daily import configure

async with aiohttp.ClientSession() as session:
    # Returns a DailyRoomConfig object
    room_config = await configure(session)

    room_url = room_config.room_url
    token = room_config.token

    # Use with DailyTransport
    transport = DailyTransport(room_url, token, "Bot Name", params=DailyParams())
```

The `configure()` function returns a `DailyRoomConfig` object with:

* `room_url`: The Daily room URL for joining
* `token`: Authentication token for the bot
* `sip_endpoint`: SIP endpoint URI (None for standard rooms)

This function:

* Uses `DAILY_SAMPLE_ROOM_URL` environment variable if set, otherwise creates a new room
* Creates rooms with 2-hour expiration and automatic ejection by default
* Generates an authentication token with 2-hour expiration using `DAILY_API_KEY`

### Environment Variables

**Required**:

* `DAILY_API_KEY`: Daily API key for creating rooms and tokens

**Optional**:

* `DAILY_SAMPLE_ROOM_URL`: Use an existing room instead of creating one
* `DAILY_API_URL`: Override Daily API endpoint (defaults to [https://api.daily.co/v1](https://api.daily.co/v1))

### Room and Token Management

When no `DAILY_SAMPLE_ROOM_URL` is provided, rooms are created automatically with:

* 2-hour expiration from creation time
* Automatic participant ejection when the room expires
* Unique names using UUID prefixes (e.g. `pipecat-uuid`)

<Note>
  Expired rooms are automatically cleaned up by Daily, so you don't need to
  manage them manually.
</Note>

Tokens are generated with 2-hour expiration and include necessary permissions for bot participation. The utilities handle all the Daily REST API interactions automatically.

### SIP Configuration

Daily utilities support SIP-enabled rooms for telephony integration. When SIP parameters are provided, the function creates rooms with telephony capabilities:

```python theme={null}
import aiohttp
from pipecat.runner.daily import configure

async with aiohttp.ClientSession() as session:
    # Create SIP-enabled room
    sip_config = await configure(
        session,
        sip_caller_phone="+15551234567",
        sip_enable_video=False,  # Voice-only by default
        sip_num_endpoints=1,
        sip_codecs={"audio": ["OPUS"], "video": ["H264"]}
    )

    print(f"Room URL: {sip_config.room_url}")
    print(f"SIP Endpoint: {sip_config.sip_endpoint}")
    print(f"Token: {sip_config.token}")
```

**SIP Parameters**:

* `sip_caller_phone`: Phone number or identifier for SIP display name (enables SIP mode)
* `sip_enable_video`: Whether video is enabled for SIP calls (default: False)
* `sip_num_endpoints`: Number of allowed SIP endpoints (default: 1)
* `sip_codecs`: Audio/video codecs to support (optional, uses Daily defaults if not specified)

<Note>
  SIP-enabled rooms cannot use existing room URLs from `DAILY_SAMPLE_ROOM_URL`.
  They always create new temporary rooms with SIP configuration.
</Note>

### Backward Compatibility

The `configure()` function supports tuple unpacking:

```python theme={null}
# Tuple unpacking (legacy style)
room_url, token = await configure(session)

# Object access (recommended)
config = await configure(session)
room_url = config.room_url
token = config.token
```

## LiveKit Configuration

LiveKit utilities manage authentication tokens, room setup, and agent permissions for LiveKit server integration.

### Basic Configuration

Use `configure()` for standard setup:

```python theme={null}
from pipecat.runner.livekit import configure

url, token, room_name = await configure()

# Use with LiveKitTransport
transport = LiveKitTransport(url=url, token=token, room_name=room_name, params=LiveKitParams())
```

### Configuration with Arguments

For command-line integration:

```python theme={null}
import argparse
from pipecat.runner.livekit import configure_with_args

parser = argparse.ArgumentParser()
url, token, room_name, args = await configure_with_args(parser)
```

Supports these command-line options:

* `-r, --room`: Specify LiveKit room name
* `-u, --url`: Specify LiveKit server URL

### Token Generation

LiveKit provides two token generation functions:

**`generate_token(room_name, participant_name, api_key, api_secret)`**
Creates a standard participant token for users or testing.

**`generate_token_with_agent(room_name, participant_name, api_key, api_secret)`**
Creates an agent token with special permissions. Use this for your bots.

```python theme={null}
from pipecat.runner.livekit import generate_token_with_agent

# Generate agent token for your bot
agent_token = generate_token_with_agent("my-room", "Pipecat Bot", api_key, api_secret)

# Generate user token for testing
user_token = generate_token("my-room", "Test User", api_key, api_secret)
```

### Environment Variables

**Required**:

* `LIVEKIT_API_KEY`: LiveKit API key
* `LIVEKIT_API_SECRET`: LiveKit API secret
* `LIVEKIT_URL`: LiveKit server URL
* `LIVEKIT_ROOM_NAME`: Default room name

All environment variables are required for LiveKit to function properly.

## WebSocket and Transport Utilities

The transport utilities provide helper functions for WebSocket parsing, SDP manipulation, and transport management.

### Telephony WebSocket Parsing

Use `parse_telephony_websocket()` to auto-detect telephony providers and extract call data:

```python theme={null}
from pipecat.runner.utils import parse_telephony_websocket

transport_type, call_data = await parse_telephony_websocket(websocket)

if transport_type == "twilio":
    stream_id = call_data["stream_id"]
    call_id = call_data["call_id"]
elif transport_type == "telnyx":
    stream_id = call_data["stream_id"]
    call_control_id = call_data["call_control_id"]
    outbound_encoding = call_data["outbound_encoding"]
elif transport_type == "plivo":
    stream_id = call_data["stream_id"]
    call_id = call_data["call_id"]
elif transport_type == "exotel":
    stream_sid=call_data["stream_id"],
    call_sid=call_data["call_id"],
```

The function automatically:

* Reads and parses initial WebSocket messages
* Detects provider based on message structure
* Extracts provider-specific call information
* Returns structured data for transport configuration

### Transport Helper Functions

**Client ID Detection**:

```python theme={null}
from pipecat.runner.utils import get_transport_client_id

client_id = get_transport_client_id(transport, client)
# Returns pc_id for WebRTC or participant ID for Daily
```

**Video Capture** (Daily only):

```python theme={null}
from pipecat.runner.utils import maybe_capture_participant_camera, maybe_capture_participant_screen

# Capture participant's camera
await maybe_capture_participant_camera(transport, client, framerate=30)

# Capture participant's screen
await maybe_capture_participant_screen(transport, client, framerate=15)
```

These functions safely handle transport detection and only execute if the transport supports the operation.

## When to Use These Utilities

### Automatic Usage (Most Common)

The development runner and `create_transport` utility handle these automatically. Most users won't need to call these functions directly.

### Manual Usage (Advanced)

Use these utilities directly when:

**Custom deployment infrastructure**: Building your own bot runner or deployment system

**Advanced transport configuration**: Need specific room settings, token permissions, or custom authentication

**Non-runner scenarios**: Integrating Pipecat transports into existing applications

**Testing and debugging**: Need to create rooms/tokens independently for testing

### Integration Example

Here's how you might use these utilities in a custom deployment:

```python theme={null}
import aiohttp
from pipecat.runner.daily import configure
from pipecat.runner.utils import parse_telephony_websocket
from pipecat.transports.daily.transport import DailyTransport, DailyParams

async def create_custom_bot_session(transport_type: str):
    if transport_type == "daily":
        async with aiohttp.ClientSession() as session:
            room_config = await configure(session)

            room_url = room_config.room_url
            token = room_config.token

            return DailyTransport(room_url, token, "Custom Bot", DailyParams())

    elif transport_type == "telephony":
        # Handle custom telephony setup
        transport_type, call_data = await parse_telephony_websocket(websocket)
        # Configure based on detected provider...
```

These utilities provide the building blocks for any transport configuration scenario while maintaining the same reliability and functionality as the development runner.


# LLMSwitcher
Source: https://docs.pipecat.ai/server/utilities/service-switchers/llm-switcher

Dynamically switch between different LLM services with support for ad-hoc inference and unified function registration

## Overview

`LLMSwitcher` is a specialized version of `ServiceSwitcher` designed specifically for managing multiple LLM services. Beyond basic service switching, it provides convenient methods for running ad-hoc inferences and registering function handlers across all LLMs simultaneously.

This is particularly useful when you need to switch between different LLM providers based on task complexity, cost optimization, or specific model capabilities while maintaining a consistent function calling interface.

## Constructor

```python theme={null}
from pipecat.pipeline.llm_switcher import LLMSwitcher
from pipecat.pipeline.service_switcher import ServiceSwitcherStrategyManual

switcher = LLMSwitcher(
    llms=[openai_service, gemini_service],
    strategy_type=ServiceSwitcherStrategyManual
)
```

<ParamField type="List[LLMService]">
  List of LLM service instances to switch between.
</ParamField>

<ParamField type="Type[ServiceSwitcherStrategy]">
  The strategy class to use for switching logic. Pass the class itself, not an
  instance.
</ParamField>

## Properties

<ParamField type="List[LLMService]">
  The list of LLM services managed by this switcher.
</ParamField>

<ParamField type="Optional[LLMService]">
  The currently active LLM service, or None if no LLMs are configured.
</ParamField>

## Methods

### run\_inference()

Run a one-shot inference with the currently active LLM, outside of the normal pipeline flow.

```python theme={null}
result = await llm_switcher.run_inference(context=llm_context)
```

<ParamField type="LLMContext">
  The LLM context containing conversation history and messages.
</ParamField>

**Returns**: `Optional[str]` - The LLM's response as a string, or None if no LLM is active.

### register\_function()

Register a function handler with all LLMs in the switcher, regardless of which is currently active.

```python theme={null}
llm_switcher.register_function(
    function_name="get_weather",
    handler=handle_weather,
    cancel_on_interruption=True
)
```

<ParamField type="Optional[str]">
  The name of the function to handle. Use None for a catch-all handler that
  processes all function calls.
</ParamField>

<ParamField type="Callable">
  The async function handler. Should accept a single `FunctionCallParams`
  parameter.
</ParamField>

<ParamField type="Optional[Callable]">
  Legacy callback function (deprecated). Put initialization code at the top of
  your handler instead.
</ParamField>

<ParamField type="bool">
  Whether to cancel this function call when a user interruption occurs.
</ParamField>

## Usage Examples

### Basic LLM Switching

```python theme={null}
from pipecat.pipeline.llm_switcher import LLMSwitcher
from pipecat.pipeline.service_switcher import ServiceSwitcherStrategyManual
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.services.google.llm import GoogleLLMService
from pipecat.frames.frames import ManuallySwitchServiceFrame

# Create LLM services
openai = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"))
gemini = GoogleLLMService(api_key=os.getenv("GOOGLE_API_KEY"))

# Create switcher
llm_switcher = LLMSwitcher(
    llms=[openai, gemini],
    strategy_type=ServiceSwitcherStrategyManual
)

# Use in pipeline
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm_switcher,
    tts,
    transport.output(),
    context_aggregator.assistant()
])

# Switch to cheaper model for simple tasks
await task.queue_frame(ManuallySwitchServiceFrame(service=gpt35))
```


# ServiceSwitcher
Source: https://docs.pipecat.ai/server/utilities/service-switchers/service-switcher

Dynamically switch between different service instances at runtime using configurable strategies

## Overview

`ServiceSwitcher` is a specialized parallel pipeline that enables dynamic switching between multiple service instances at runtime. This is useful when you need to change between different STT providers, TTS providers, or other frame processors based on user preferences, costs, performance requirements, or other runtime conditions.

The switcher uses a strategy pattern to determine which service is active. Currently, a manual switching strategy is provided, allowing explicit control over which service handles frames at any given time.

## How It Works

`ServiceSwitcher` wraps multiple services in a parallel pipeline where only the active service processes frames. Each service is "sandwiched" between two filters that check if it's the currently active service before allowing frames to pass through. When you switch services, the filters update to redirect frame flow to the newly active service.

## Constructor

```python theme={null}
from pipecat.pipeline.service_switcher import ServiceSwitcher, ServiceSwitcherStrategyManual

switcher = ServiceSwitcher(
    services=[stt_service1, stt_service2],
    strategy_type=ServiceSwitcherStrategyManual
)
```

<ParamField type="List[FrameProcessor]">
  List of service instances to switch between. Can be any frame processors (STT,
  TTS, or custom processors).
</ParamField>

<ParamField type="Type[ServiceSwitcherStrategy]">
  The strategy class to use for switching logic. Pass the class itself, not an
  instance.
</ParamField>

## Switching Strategies

### ServiceSwitcherStrategyManual

The manual strategy allows explicit control over which service is active by pushing `ManuallySwitchServiceFrame` frames into the pipeline.

**Initial State**: The first service in the list is active by default.

**Switching**: Push a `ManuallySwitchServiceFrame` with the desired service instance.

## Usage Examples

### Switching Between TTS Services

```python theme={null}
from pipecat.frames.frames import ManuallySwitchServiceFrame
from pipecat.pipeline.service_switcher import ServiceSwitcher, ServiceSwitcherStrategyManual
from pipecat.services.elevenlabs.tts import ElevenLabsTTSService
from pipecat.services.cartesia.tts import CartesiaTTSService

# Create TTS services
elevenlabs = ElevenLabsTTSService(api_key=os.getenv("ELEVENLABS_API_KEY"), voice_id=os.getenv("ELEVENLABS_VOICE_ID"))
cartesia = CartesiaTTSService(api_key=os.getenv("CARTESIA_API_KEY"), voice_id=os.getenv("CARTESIA_VOICE_ID"))

# Create switcher with both services
tts_switcher = ServiceSwitcher(
    services=[elevenlabs, cartesia],
    strategy_type=ServiceSwitcherStrategyManual
)

# Use in pipeline
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    tts_switcher,
    transport.output(),
    context_aggregator.assistant()
])

# Later, switch to Cartesia
await task.queue_frame(ManuallySwitchServiceFrame(service=cartesia))
```


# Fal Smart Turn
Source: https://docs.pipecat.ai/server/utilities/smart-turn/fal-smart-turn

Cloud-hosted Smart Turn detection using Fal.ai

<Warning>
  DEPRECATED: `FalSmartTurnAnalyzer` is deprecated. Please use
  [LocalSmartTurnAnalyzerV3](/server/utilities/smart-turn/smart-turn-overview#local-smart-turn)
  instead, which provides fast CPU inference without requiring external API
  calls.
</Warning>

## Overview

`FalSmartTurnAnalyzer` provides an easy way to use Smart Turn detection via Fal.ai's cloud infrastructure. This implementation requires minimal setup - just an API key - and offers scalable inference without having to manage your own servers.

## Installation

```bash theme={null}
pip install "pipecat-ai[remote-smart-turn]"
```

## Requirements

* A Fal.ai account and API key (get one at [Fal.ai](https://fal.ai))
* Internet connectivity for making API calls

## Configuration

### Constructor Parameters

<ParamField type="Optional[str]">
  Your Fal.ai API key for authentication (required unless using a custom
  deployment)
</ParamField>

<ParamField type="str">
  URL endpoint for the Smart Turn API (defaults to the official Fal deployment)
</ParamField>

<ParamField type="aiohttp.ClientSession">
  An aiohttp client session for making HTTP requests
</ParamField>

<ParamField type="Optional[int]">
  Audio sample rate (will be set by the transport if not provided)
</ParamField>

<ParamField type="SmartTurnParams">
  Configuration parameters for turn detection. See
  [SmartTurnParams](/server/utilities/smart-turn/smart-turn-overview#configuration)
  for details.
</ParamField>

## Example

```python theme={null}
import os
import aiohttp
from pipecat.audio.turn.smart_turn.fal_smart_turn import FalSmartTurnAnalyzer
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.audio.vad.vad_analyzer import VADParams
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.transports.base_transport import TransportParams
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies

async def setup_transport():
    async with aiohttp.ClientSession() as session:
        transport = SmallWebRTCTransport(
            webrtc_connection=webrtc_connection,
            params=TransportParams(
                audio_in_enabled=True,
                audio_out_enabled=True,
            ),
        )

        # Configure Smart Turn Detection via user turn strategies
        user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
            context,
            user_params=LLMUserAggregatorParams(
                user_turn_strategies=UserTurnStrategies(
                    stop=[TurnAnalyzerUserTurnStopStrategy(
                        turn_analyzer=FalSmartTurnAnalyzer(
                            api_key=os.getenv("FAL_SMART_TURN_API_KEY"),
                            aiohttp_session=session
                        )
                    )]
                ),
                vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
            ),
        )

        # Continue with pipeline setup...
```

## Custom Deployment

You can also deploy the Smart Turn model yourself on Fal.ai and point to your custom deployment:

```python theme={null}
TurnAnalyzerUserTurnStopStrategy(
    turn_analyzer=FalSmartTurnAnalyzer(
        url="https://fal.run/your-username/your-deployment/raw",
        api_key=os.getenv("FAL_API_KEY"),
        aiohttp_session=session
    )
)
```

## Performance Considerations

* **Latency**: While Fal provides global infrastructure, there will be network latency compared to local inference
* **Reliability**: Depends on network connectivity and Fal.ai service availability
* **Scalability**: Handles scaling automatically based on your usage

## Notes

* Fal handles the model hosting, scaling, and infrastructure management
* The session timeout is controlled by the `stop_secs` parameter
* For high-throughput applications, consider deploying your own inference service


# Smart Turn Overview
Source: https://docs.pipecat.ai/server/utilities/smart-turn/smart-turn-overview

Advanced conversational turn detection powered by the smart-turn model

## Overview

Smart Turn Detection is an advanced feature in Pipecat that determines when a user has finished speaking and the bot should respond. Unlike basic Voice Activity Detection (VAD) which only detects speech vs. non-speech, Smart Turn Detection uses a machine learning model to recognize natural conversational cues like intonation patterns and linguistic signals.

<CardGroup>
  <Card title="Smart Turn Model" icon="github" href="https://github.com/pipecat-ai/smart-turn">
    Open source model for advanced conversational turn detection. Contribute to
    model training and development.
  </Card>

  <Card title="Data Collector" icon="microphone" href="https://turn-training.pipecat.ai/">
    Contribute conversational data to improve the smart-turn model
  </Card>

  <Card title="Data Classifier" icon="check-circle" href="https://smart-turn-dataset.pipecat.ai/">
    Help classify turn completion patterns in conversations
  </Card>
</CardGroup>

Pipecat provides `LocalSmartTurnAnalyzerV3` which runs inference locally using ONNX. This is the recommended approach due to the fast CPU inference times in Smart Turn v3.

## Installation

```bash theme={null}
pip install "pipecat-ai[local-smart-turn-v3]"
```

The Smart Turn model weights are bundled with Pipecat, so no need to download these separately.

## Integration with User Turn Strategies

Smart Turn Detection is integrated into your application by configuring a `TurnAnalyzerUserTurnStopStrategy` with `LocalSmartTurnAnalyzerV3` in your context aggregator:

```python theme={null}
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.audio.vad.vad_analyzer import VADParams
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.transports.base_transport import TransportParams
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies

transport = SmallWebRTCTransport(
    webrtc_connection=webrtc_connection,
    params=TransportParams(
        audio_in_enabled=True,
    ),
)

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_turn_strategies=UserTurnStrategies(
            stop=[TurnAnalyzerUserTurnStopStrategy(
                turn_analyzer=LocalSmartTurnAnalyzerV3()
            )]
        ),
        vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
    ),
)
```

<Tip>
  Smart Turn Detection requires VAD to be enabled and works best when the VAD analyzer is set to a short `stop_secs` value. We recommend 0.2 seconds.

  ```python theme={null}
  vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2))
  ```
</Tip>

## Configuration

The `SmartTurnParams` class configures turn detection behavior:

<ParamField type="float">
  Duration of silence in seconds required before triggering a silence-based end
  of turn
</ParamField>

<ParamField type="float">
  Amount of audio (in milliseconds) to include before speech is detected
</ParamField>

<ParamField type="float">
  Maximum allowed segment duration in seconds. For segments longer than this
  value, a rolling window is used.
</ParamField>

## Local Smart Turn

The `LocalSmartTurnAnalyzerV3` runs inference locally. Version 3 of the model supports fast CPU inference on ordinary cloud instances.

### Constructor Parameters

<ParamField type="Optional[str]">
  Path to the Smart Turn v3 ONNX file containing the model weights. Download this from
  [https://huggingface.co/pipecat-ai/smart-turn-v3/tree/main](https://huggingface.co/pipecat-ai/smart-turn-v3/tree/main)

  This parameter is optional, as Pipecat includes a copy of the model internally, and this
  is used if the path is unset.
</ParamField>

<ParamField type="Optional[int]">
  Audio sample rate (will be set by the transport if not provided)
</ParamField>

<ParamField type="SmartTurnParams">
  Configuration parameters for turn detection
</ParamField>

### Example

```python theme={null}
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.audio.vad.vad_analyzer import VADParams
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.transports.base_transport import TransportParams
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies

# Create the transport
transport = SmallWebRTCTransport(
    webrtc_connection=webrtc_connection,
    params=TransportParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
    ),
)

# Configure Smart Turn Detection via user turn strategies
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_turn_strategies=UserTurnStrategies(
            stop=[TurnAnalyzerUserTurnStopStrategy(
                turn_analyzer=LocalSmartTurnAnalyzerV3()
            )]
        ),
        vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
    ),
)
```

## How It Works

Smart Turn Detection continuously analyzes audio streams to identify natural turn completion points:

1. **Audio Buffering**: The system continuously buffers audio with timestamps, maintaining a small buffer of pre-speech audio.
2. **VAD Processing**: Voice Activity Detection (using the Silero model) detects when there is a pause in the user's speech.
3. **Smart Turn Analysis**: When VAD detects a pause in speech, the Smart Turn model analyzes the audio from the most recent 8 seconds of the user's turn, and makes a decision about whether the turn is complete or incomplete.

The system includes a fallback mechanism: if a turn is classified as incomplete but silence continues for longer than `stop_secs`, the turn is automatically marked as complete.

## Notes

* The model supports 23 languages, see the [source repository](https://github.com/pipecat-ai/smart-turn) for more details
* You can adjust the `stop_secs` parameter based on your application's needs for responsiveness
* Smart Turn generally provides a more natural conversational experience but is computationally more intensive than simple VAD
* `LocalSmartTurnAnalyzerV3` is designed to run on CPU, and inference can be performed on low-cost cloud instances in under 100ms. However, by installing the `onnxruntime-gpu` dependency, you can achieve higher performance by making use of GPU inference.


# MarkdownTextFilter
Source: https://docs.pipecat.ai/server/utilities/text/markdown-text-filter

Converts Markdown-formatted text to TTS-friendly plain text while preserving structure

## Overview

`MarkdownTextFilter` transforms Markdown-formatted text into plain text that's suitable for text-to-speech (TTS) systems. It intelligently removes formatting elements while preserving the content structure, including proper spacing and list formatting.

This filter is especially valuable for LLM-generated content, which often includes Markdown formatting that would sound unnatural if read aloud by a TTS system.

## Constructor

```python theme={null}
filter = MarkdownTextFilter(params=InputParams())
```

<ParamField type="InputParams">
  Configuration parameters for the filter
</ParamField>

### Input Parameters

Configure the filter behavior with these options:

<ParamField type="bool">
  Whether the filter is active (when False, text passes through unchanged)
</ParamField>

<ParamField type="bool">
  Whether to remove code blocks from the output
</ParamField>

<ParamField type="bool">
  Whether to remove Markdown tables from the output
</ParamField>

## Features

The filter handles these Markdown elements:

* **Basic Formatting**: Removes `*italic*`, `**bold**`, and other formatting markers
* **Code**: Removes inline code ticks and optionally removes code blocks
* **Lists**: Preserves numbered lists while removing Markdown formatting
* **Tables**: Optionally removes Markdown tables
* **Whitespace**: Carefully preserves meaningful whitespace for natural speech
* **HTML**: Removes HTML tags and converts entities to their plain text equivalents

## Usage Examples

### Basic Usage with TTS Service

```python theme={null}
from pipecat.utils.text.markdown_text_filter import MarkdownTextFilter
from pipecat.services.cartesia.tts import CartesiaTTSService

# Create the filter
md_filter = MarkdownTextFilter()

# Use with TTS service
tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    voice_id="voice_id_here",
    text_filter=md_filter
)
```

### Custom Configuration

```python theme={null}
# Create filter that removes code blocks and tables
md_filter = MarkdownTextFilter(
    params=MarkdownTextFilter.InputParams(
        filter_code=True,
        filter_tables=True
    )
)
```

## What Gets Removed

| Markdown Feature           | Example                  | Result       |
| -------------------------- | ------------------------ | ------------ |
| Bold                       | `**important**`          | `important`  |
| Italic                     | `*emphasized*`           | `emphasized` |
| Headers                    | `## Section`             | `Section`    |
| Code (inline)              | `` `code` ``             | `code`       |
| Code blocks (when enabled) | ` ```python\ncode\n``` ` | ` `          |
| Tables (when enabled)      | `\|A\|B\|\n\|--\|--\|`   | ` `          |
| HTML tags                  | `<em>text</em>`          | `text`       |
| Repeated characters        | `!!!!!!!`                | `!`          |

## Notes

* Preserves sentence structure and readability
* Maintains whitespace that affects speech prosody
* Handles streaming text with partial Markdown elements
* Efficiently converts HTML entities to plain text characters
* Smart handling of code blocks and tables with state tracking
* Integrates directly with TTS services in the Pipecat pipeline


# Text Aggregators and Filters
Source: https://docs.pipecat.ai/server/utilities/text/overview

An overview of text aggregators and filters available in the server utilities

## Overview

The terms "aggregators" and "filters" are overloaded in Pipecat and can refer to different components depending on the context. This document provides a high-level overview of the low-level text utilities often used by various pipeline services, like the TTS. These utilities operate on streaming text, allowing for dynamic processing, transformation, and aggregation of text data as it flows through the system. They are essentially text in -> text out (but maybe with some metadata about the text too).

## Aggregators

Aggregators are components that collect and combine text data over time, often buffering input until certain conditions are met (like sentence boundaries or specific patterns). They can modify, enhance, or restructure the text before passing it along. Examples include:

* SimpleTextAggregator: Buffers text until sentence boundaries are detected. This is the default aggregator used by most TTS services.
* SkipTagsAggregator: Buffers text until sentence boundaries, while skipping over specified tags so that sentences are not prematurely detected due to characters in between those tags. This is the default aggregator used by Cartesia and Rime in order to skip over custom spell tags.
* [PatternPairAggregator](./pattern-pair-aggregator): Buffers text until either a sentence boundary is detected or a complete pattern pair (like XML tags or custom delimiters) is found. Patterns found can either be removed, left in but trigger a registered callback, or aggregated separately. Useful for voice switching, structured content processing, and extracting metadata from LLM outputs.

### Usage Pattern

After initial setup, aggregators are used by repeatedly calling their `aggregate()` method with incoming text chunks. The aggregator processes the text according to its logic and returns aggregated text when appropriate. If no complete aggregation is ready, it will return `None`, otherwise it will return an `Aggregation` object containing the aggregated text along with a `type` string meant to convey the nature of the aggregation (e.g., "sentence", "xml", etc.).

```python theme={null}
from pipecat.utils.text.simple_text_aggregator import SimpleTextAggregator
aggregator = SimpleTextAggregator()
while streaming_text:
    chunk = get_next_text_chunk()
    aggregation = aggregator.aggregate(chunk)
    if aggregation:
        process_aggregated_text(aggregation.text, aggregation.type)
```

### Custom Aggregators

You can create custom aggregators by subclassing the `BaseTextAggregator` class and implementing your own aggregation logic in the `aggregate()` method. This allows for tailored text processing to meet specific application needs.

## Filters

Filters are components that process text data in a streaming fashion, transforming or modifying it as it passes through. Unlike aggregators, filters typically do not buffer text but instead operate on each chunk of text individually. Currently, the only built-in filter is the [`MarkdownTextFilter`](./markdown-text-filter), which processes markdown syntax in streaming text, converting it into TTS-friendly plain text.

### Usage Pattern

Filters are used by calling their `filter()` method with incoming text chunks. The filter processes the text and returns the transformed text immediately.

```python theme={null}
from pipecat.utils.text.markdown_text_filter import MarkdownTextFilter
filter = MarkdownTextFilter()
while streaming_text:
    chunk = get_next_text_chunk()
    filtered_text = filter.filter(chunk)
    process_filtered_text(filtered_text)
```

### Custom Filters

You can create custom filters by subclassing the `BaseTextFilter` class and implementing your own filtering logic in the `filter()` method. This allows for specialized text transformations to suit your application's requirements.


# PatternPairAggregator
Source: https://docs.pipecat.ai/server/utilities/text/pattern-pair-aggregator

Text aggregator that identifies and processes content between pattern pairs in streaming text

## Overview

`PatternPairAggregator` is a specialized text aggregator that buffers streaming text until it can identify complete pattern pairs (like XML tags, markdown formatting, or custom delimiters). It processes the content between these patterns using a set of pre-defined actions (remove, keep, or aggregate) and returns text outside those patterns at sentence boundaries. The aggregator supports registering callback functions that are invoked when specific pattern pairs are matched, allowing for custom processing when matches occur. Note: These callbacks do not support modifying the text being aggregated; they are intended for side effects like logging or updating state.

This aggregator is particularly useful for applications like voice switching, structured content processing, and extracting metadata from LLM outputs, ensuring that patterns spanning multiple text chunks are correctly identified or categorizing text based on embedded markers for downstream services and processing to treat different segments appropriately. For example: identifying URL patterns, code blocks, or special formatting in LLM responses that may need special speech handling in the TTS or client-side handling via RTVI.

<Tip>
  Want to see it in action? Check out the [voice switching
  demo](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/35-pattern-pair-voice-switching.py) or the [bot output demo](https://github.com/pipecat-ai/pipecat-examples/blob/main/code-helper).
</Tip>

## Constructor

```python theme={null}
pattern_aggregator = PatternPairAggregator()
```

No parameters are required for initialization. The aggregator starts with an empty buffer and no registered patterns.

## Methods

### add\_pattern

```python theme={null}
pattern_aggregator.add_pattern(
  type,
  start_pattern,
  end_pattern,
  action
)
```

Registers a new pattern pair to detect in the text.

<ParamField type="str">
  Unique identifier for this pattern pair that should also represent what the text between the tags represents (e.g., "voice", "xml", "credit\_card", etc.). This value will be returned as part of both PatternMatch provided to callbacks and the Aggregation object returned from `aggregate()`.

  <Warning>
    This type may not be set to "sentence" or "word" as those are reserved for standard aggregations.
  </Warning>

  <Tip>
    Choose descriptive and unique type names to avoid confusion when handling multiple patterns.
    This type will also be referenced in the TTS for optionally skipping these types or providing custom text transformations before speaking these types. It's also optionally referenced in RTVI for similar purposes of not sending certain types or transforming them before sending to the client.
  </Tip>
</ParamField>

<ParamField type="str">
  Pattern that marks the beginning of content
</ParamField>

<ParamField type="str">
  Pattern that marks the end of content
</ParamField>

<ParamField type="MatchAction">
  What to do with the matched pattern and its content:

  * `MatchAction.REMOVE`: The text along with its delimiters will be removed from the streaming text. Sentence aggregation will continue on as if this text did not exist.
  * `MatchAction.KEEP`: The delimiters will be removed, but the content between them will be kept. Sentence aggregation will continue on with the internal text included. This is helpful if you want to keep the content but be notified when it occurs via a callback.
  * `MatchAction.AGGREGATE`: Aggregate the matched pattern and its content as a separate aggregation. The matched content will be returned in an `Aggregation` object with the specified type when the pattern is completed. When the start of this pattern is detected, any buffered text up to that point will be returned as a standard "sentence" aggregation.
</ParamField>

<ResponseField name="Returns">Self for method chaining</ResponseField>

### add\_pattern\_pair

<Warning>
  This method is deprecated. Use `add_pattern` instead.
</Warning>

```python theme={null}
pattern_aggregator.add_pattern_pair(
  pattern_id,
  start_pattern,
  end_pattern,
  remove_match=True
)
```

### on\_pattern\_match

```python theme={null}
pattern_aggregator.on_pattern_match(type, handler)
```

Registers a handler function to be called when a specific pattern pair is matched.

<ParamField type="str">
  The pattern pair type to listen for (as defined in `add_pattern`)
</ParamField>

<ParamField type="Callable[[PatternMatch], Awaitable[None]]">
  Function to call when the pattern is matched. The function should accept a
  PatternMatch object.
</ParamField>

<ResponseField name="Returns">Self for method chaining</ResponseField>

## Pattern Match Object

When a pattern is matched, the handler function receives a `PatternMatch` object which is a subclass of the `Aggregation` object. It contains the following fields:

<ResponseField name="type" type="str">
  The identifier and descriptor of the matched pattern pair. This field is part of the `Aggregation` base class.
</ResponseField>

<ResponseField name="text" type="str">
  The text content between the start and end patterns. This field is part of the `Aggregation` base class.
</ResponseField>

<ResponseField name="full_match" type="str">
  The complete text including start and end patterns.
</ResponseField>

## Usage Examples

### Voice Switching in TTS

This example demonstrates finding custom `<voice>` tags in streaming text to switch voices dynamically in a TTS service like Cartesia. It removes the tags and the content between them, such that the content is treated as if it does not exist. It will not be spoken by the TTS, it will not be added to the context, and it will not be sent to clients via RTVI. Instead, it simply triggers a voice switch side effect.

```python theme={null}
# Define voice IDs
VOICE_IDS = {
    "narrator": "c45bc5ec-dc68-4feb-8829-6e6b2748095d",
    "female": "71a7ad14-091c-4e8e-a314-022ece01c121",
    "male": "7cf0e2b1-8daf-4fe4-89ad-f6039398f359",
}

# Initialize TTS service, starting with default narrator voice
tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    voice_id=VOICE_IDS["narrator"],
)

# Create pattern aggregator
pattern_aggregator = PatternPairAggregator()

# Add pattern for voice tags
pattern_aggregator.add_pattern(
    type="voice",
    start_pattern="<voice>",
    end_pattern="</voice>",
    action=MatchAction.REMOVE
)

# Register handler for voice switching
def on_voice_tag(match: PatternMatch):
    voice_name = match.content.strip().lower()
    if voice_name in VOICE_IDS:
        voice_id = VOICE_IDS[voice_name]
        tts.set_voice(voice_id)
        logger.info(f"Switched to {voice_name} voice")

pattern_aggregator.on_pattern_match("voice", on_voice_tag)

# Set the aggregator on an LLMTextProcessor
llm_text_processor = LLMTextProcessor(text_aggregator=pattern_aggregator)

# add the llm_text_processor to your pipeline after the llm and before the tts
# llm -> llm_text_processor -> tts
```

### Extracting Structured Data from LLM Outputs

This example shows how to extract JSON data blocks from LLM outputs, aggregating them separately to be removed from the spoken text, but not from the context or client display.

````python theme={null}
# Create pattern aggregator
pattern_aggregator = PatternPairAggregator()

# Add pattern for JSON data
pattern_aggregator.add_pattern(
    type="json",
    start_pattern="```json",
    end_pattern="```",
    action=MatchAction.AGGREGATE
)

# Set the aggregator on an LLMTextProcessor
llm_text_processor = LLMTextProcessor(text_aggregator=pattern_aggregator)

# Initialize TTS service, and don't speak JSON data
tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    skip_aggregator_types=["json"],
)

# add the llm_text_processor to your pipeline after the llm and before the tts
# llm -> llm_text_processor -> tts
````

### Handling Special Values in LLM Output

This example demonstrates how to identify and process custom tags in LLM output that denote special content, such as credit cards that should be handled differently by downstream services.
In this case, the TTS should spell it out, while RTVI should obfuscate the number.

```python theme={null}
from pipecat.utils.text.pattern_pair_aggregator import MatchAction, PatternPairAggregator

# Create pattern aggregator
pattern_aggregator = PatternPairAggregator()

# Add patterns for different parts of an explanation
pattern_aggregator.add_pattern(
    type="credit_card",
    start_pattern="<card>",
    end_pattern="</card>",
    action=MatchAction.AGGREGATE
)

# Set the aggregator on an LLMTextProcessor
llm_text_processor = LLMTextProcessor(text_aggregator=pattern_aggregator)

# Text-to-Speech service
tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
)

# Text transformers for TTS
# This will insert Cartesia's spell tags around the provided text.
async def spell_out_text(text: str, type: str) -> str:
    return CartesiaTTSService.SPELL(text)

# Setup the text transformers in TTS to spell out credit card numbers.
# The string below matches the type defined in the PatternPairAggregator
# above so that whenever those segments are encountered, this transform
# is applied
tts.add_text_transformer(spell_out_text, "credit_card")

# RTVI is automatically enabled. Use rtvi_observer_params to add
# a text transformer that obfuscates credit card numbers in client output.
from pipecat.processors.frameworks.rtvi import RTVIObserverParams

def obfuscate_credit_card(text: str, type: str) -> str:
    return "XXXX-XXXX-XXXX-" + text[-4:]

task = PipelineTask(
    pipeline,  # llm -> llm_text_processor -> tts
    rtvi_observer_params=RTVIObserverParams(
        bot_output_transforms=[("credit_card", obfuscate_credit_card)],
    ),
)
```

## How It Works

```mermaid theme={null}
flowchart TD
  A[New Text] --> B[Add to Buffer]
  B --> C{Start & End Patterns Found?}
    C --> |Yes| C1[Call Handlers]
    C1 --> D{Pattern Action?}
      D --> E{{AGGREGATE}}
        E --> E1[Reset buffer with trailing text]
        E1 --> E2[RETURN contents as Aggregation with specified type]
      D --> F{{REMOVE}}
        F --> F1[Remove pattern from buffer]
        F1 --> S{End of Sentence?}
      D --> G{{KEEP}}
        G --> S
    C --> |No| H{Start Pattern Found?}
      H --> |No| S
      H --> |Yes| I{Pattern Action == AGGREGATE?}
        I --> |Yes| I1[Reset buffer to pattern start]
        I1 --> I2[RETURN prior buffer as Aggregation of type 'sentence']
    S --> |Yes| S1[return Aggregation of type 'sentence']
    S --> | No| S2[return None]
```

## Notes

* Patterns are processed in the order they appear in the text
* Handlers are called when complete patterns are found
* Patterns can span multiple sentences of text, but be aware that encoding many "reasoning" tokens may slow down the LLM response


# TranscriptProcessor
Source: https://docs.pipecat.ai/server/utilities/transcript-processor

Factory for creating and managing conversation transcript processors with shared event handling

<Warning>
  DEPRECATED: TranscriptProcessor has been deprecated. Use
  `on_user_turn_stopped` and `on_assistant_turn_stopped` events on the context
  aggregators to collect transcriptions, see
  [Transcriptions](/server/utilities/turn-management/transcriptions) for
  details.
</Warning>

## Overview

The `TranscriptProcessor` is a factory class that creates and manages processors for handling conversation transcripts from both users and assistants. It provides unified access to transcript processors with shared event handling, making it easy to track and respond to conversation updates in real-time.

The processor normalizes messages from various sources into a consistent `TranscriptionMessage` format and emits events when new messages are added to the conversation.

## Constructor

```python theme={null}
TranscriptProcessor()
```

Creates a new transcript processor factory with no parameters.

## Methods

### user()

```python theme={null}
def user(**kwargs) -> UserTranscriptProcessor
```

Get or create the user transcript processor instance. This processor handles `TranscriptionFrame`s from STT services.

**Parameters:**

* `**kwargs`: Arguments passed to the `UserTranscriptProcessor` constructor

**Returns:** `UserTranscriptProcessor` instance for processing user messages.

### assistant()

```python theme={null}
def assistant(**kwargs) -> AssistantTranscriptProcessor
```

Get or create the assistant transcript processor instance. This processor handles `TTSTextFrame`s from TTS services and aggregates them into complete utterances.

**Parameters:**

* `**kwargs`: Arguments passed to the `AssistantTranscriptProcessor` constructor

**Returns:** `AssistantTranscriptProcessor` instance for processing assistant messages.

### event\_handler()

```python theme={null}
def event_handler(event_name: str)
```

Decorator that registers event handlers for both user and assistant processors.

**Parameters:**

* `event_name`: Name of the event to handle

**Returns:** Decorator function that registers the handler with both processors.

## Event Handlers

### on\_transcript\_update

Triggered when new messages are added to the conversation transcript.

```python theme={null}
@transcript.event_handler("on_transcript_update")
async def handle_transcript_update(processor, frame):
    # Handle transcript updates
    pass
```

**Parameters:**

* `processor`: The specific processor instance that emitted the event (UserTranscriptProcessor or AssistantTranscriptProcessor)
* `frame`: `TranscriptionUpdateFrame` containing the new messages

## Data Structures

### TranscriptionMessage

```python theme={null}
@dataclass
class TranscriptionMessage:
    role: Literal["user", "assistant"]
    content: str
    timestamp: str | None = None
    user_id: str | None = None
```

**Fields:**

* `role`: The message sender type ("user" or "assistant")
* `content`: The transcribed text content
* `timestamp`: ISO 8601 timestamp when the message was created
* `user_id`: Optional user identifier (for user messages only)

### TranscriptionUpdateFrame

Frame containing new transcript messages, emitted by the `on_transcript_update` event.

**Properties:**

* `messages`: List of `TranscriptionMessage` objects containing the new transcript content

## Frames

### UserTranscriptProcessor

* **Input:** `TranscriptionFrame` from STT services
* **Output:** `TranscriptionMessage` with role "user"

### AssistantTranscriptProcessor

* **Input:** `TTSTextFrame` from TTS services
* **Output:** `TranscriptionMessage` with role "assistant"

## Integration Notes

### Pipeline Placement

Place the processors at specific positions in your pipeline for accurate transcript collection:

```python theme={null}
pipeline = Pipeline([
    transport.input(),
    stt,                        # Speech-to-text service
    transcript.user(),          # Place after STT
    context_aggregator.user(),
    llm,
    tts,                        # Text-to-speech service
    transport.output(),
    transcript.assistant(),     # Place after transport.output()
    context_aggregator.assistant(),
])
```

### Event Handler Registration

Event handlers are automatically applied to both user and assistant processors:

```python theme={null}
transcript = TranscriptProcessor()

# This handler will receive events from both processors
@transcript.event_handler("on_transcript_update")
async def handle_update(processor, frame):
    for message in frame.messages:
        print(f"{message.role}: {message.content}")
```


# External Turn Management
Source: https://docs.pipecat.ai/server/utilities/turn-management/external-turn-management

Handle turn detection externally using UserTurnProcessor or external services

## Overview

In some scenarios, turn detection happens externally, either through a dedicated processor or an external service. Pipecat provides `ExternalUserTurnStrategies`, a [user turn strategy](/server/utilities/turn-management/user-turn-strategies) that defers turn handling to these external sources.

External turn management might be needed when:

* **Multiple context aggregators**: Parallel pipelines with multiple LLMs need a single, shared source of turn events
* **External services with turn detection**: Services like [Deepgram Flux](/server/services/stt/deepgram) or [Speechmatics](/server/services/stt/speechmatics) provide their own turn detection

In both cases, you need to configure your context aggregators with `ExternalUserTurnStrategies` to defer turn handling to the external source.

## External Services

Some speech-to-text services provide built-in turn detection. When using these services, configure your context aggregator with `ExternalUserTurnStrategies` to let the service handle turn management:

```python theme={null}
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.turns.user_turn_strategies import ExternalUserTurnStrategies

# Configure aggregator to use external turn strategies
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_turn_strategies=ExternalUserTurnStrategies()
    ),
)
```

## UserTurnProcessor

`UserTurnProcessor` is a frame processor for managing user turn lifecycle when you need a single source of turn events shared across multiple context aggregators. It emits `UserStartedSpeakingFrame` and `UserStoppedSpeakingFrame` frames and handles interruptions.

<Note>
  `UserTurnProcessor` only manages user turn start and end events. It does not
  handle transcription aggregation, that remains the responsibility of the
  context aggregators.
</Note>

### Constructor Parameters

<ParamField type="UserTurnStrategies">
  Configured strategies for starting and stopping user turns. See [User Turn
  Strategies](/server/utilities/turn-management/user-turn-strategies) for
  available options.
</ParamField>

<ParamField type="float">
  Timeout in seconds to automatically stop a user turn if no stop strategy
  triggers.
</ParamField>

### Event Handlers

`UserTurnProcessor` provides event handlers for turn lifecycle events:

```python theme={null}
@user_turn_processor.event_handler("on_user_turn_started")
async def on_user_turn_started(processor, strategy):
    # Called when a user turn starts
    pass

@user_turn_processor.event_handler("on_user_turn_stopped")
async def on_user_turn_stopped(processor, strategy):
    # Called when a user turn stops
    pass

@user_turn_processor.event_handler("on_user_turn_stop_timeout")
async def on_user_turn_stop_timeout(processor):
    # Called if no stop strategy triggers before timeout
    pass
```

### Usage with Parallel Pipelines

When using parallel pipelines with multiple context aggregators, place `UserTurnProcessor` before the parallel pipeline and configure each context aggregator with `ExternalUserTurnStrategies`:

```python theme={null}
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
from pipecat.pipeline.parallel_pipeline import ParallelPipeline
from pipecat.pipeline.pipeline import Pipeline
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_processor import UserTurnProcessor
from pipecat.turns.user_turn_strategies import ExternalUserTurnStrategies, UserTurnStrategies

# Create the external user turn processor with your preferred strategies
user_turn_processor = UserTurnProcessor(
    user_turn_strategies=UserTurnStrategies(
        stop=[
            TurnAnalyzerUserTurnStopStrategy(
                turn_analyzer=LocalSmartTurnAnalyzerV3()
            )
        ]
    ),
)

# Create contexts for each LLM
openai_context = LLMContext(openai_messages)
groq_context = LLMContext(groq_messages)

# Configure aggregators to use external turn strategies
openai_context_aggregator = LLMContextAggregatorPair(
    openai_context,
    user_params=LLMUserAggregatorParams(
        user_turn_strategies=ExternalUserTurnStrategies()
    ),
)
groq_context_aggregator = LLMContextAggregatorPair(
    groq_context,
    user_params=LLMUserAggregatorParams(
        user_turn_strategies=ExternalUserTurnStrategies()
    ),
)

# Build the pipeline with UserTurnProcessor before the parallel branches
pipeline = Pipeline(
    [
        transport.input(),
        stt,
        user_turn_processor,  # Handles turn management for all branches
        ParallelPipeline(
            [
                openai_context_aggregator.user(),
                openai_llm,
                transport.output(),
                openai_context_aggregator.assistant(),
            ],
            [
                groq_context_aggregator.user(),
                groq_llm,
                groq_context_aggregator.assistant(),
            ],
        ),
    ]
)
```

## Related

* [User Turn Strategies](/server/utilities/turn-management/user-turn-strategies) - Configure turn detection strategies
* [Parallel Pipeline](/server/pipeline/parallel-pipeline) - Run multiple pipeline branches concurrently
* [Turn Events](/server/utilities/turn-management/turn-events) - Handle turn lifecycle events


# Filter Incomplete User Turns
Source: https://docs.pipecat.ai/server/utilities/turn-management/filter-incomplete-turns

Use LLM-based detection to suppress responses when users are cut off mid-thought

## Overview

Filter Incomplete Turns is an LLM-powered feature that detects when a user's conversational turn was incomplete (they were cut off or need time to think) and suppresses the bot's response accordingly. Instead of responding to partial input, the bot waits for the user to continue, then automatically re-engages if they remain silent.

This creates more natural conversations by:

* Preventing the bot from responding to incomplete thoughts
* Giving users time to finish speaking without interruption
* Automatically prompting users to continue after pauses

## How It Works

When enabled, the LLM outputs a turn completion marker as the first character of every response:

| Marker | Meaning                                              | Bot Behavior                             |
| ------ | ---------------------------------------------------- | ---------------------------------------- |
| `✓`    | **Complete** - User finished their thought           | Respond normally                         |
| `○`    | **Incomplete Short** - User was cut off mid-sentence | Suppress response, wait 5s, then prompt  |
| `◐`    | **Incomplete Long** - User needs time to think       | Suppress response, wait 10s, then prompt |

The system automatically:

1. Injects turn completion instructions into the LLM's system prompt
2. Detects markers in the LLM's streaming response
3. Suppresses bot speech for incomplete turns
4. Starts a timeout based on the incomplete type
5. Re-prompts the LLM when the timeout expires

## Configuration

Enable the feature via `LLMUserAggregatorParams` when creating an `LLMContextAggregatorPair`:

```python theme={null}
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)

context = LLMContext(messages)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        filter_incomplete_user_turns=True,
    ),
)
```

### LLMUserAggregatorParams

<ParamField type="bool">
  Enable LLM-based turn completion detection. When True, the system
  automatically injects turn completion instructions into the LLM context and
  configures the LLM service to process turn markers.
</ParamField>

<ParamField type="UserTurnCompletionConfig">
  Optional configuration object for customizing turn completion behavior. If not
  provided, default values are used.
</ParamField>

## UserTurnCompletionConfig

Use `UserTurnCompletionConfig` to customize timeouts, prompts, and instructions:

```python theme={null}
from pipecat.turns.user_turn_completion_mixin import UserTurnCompletionConfig

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        filter_incomplete_user_turns=True,
        user_turn_completion_config=UserTurnCompletionConfig(
            incomplete_short_timeout=5.0,
            incomplete_long_timeout=10.0,
            incomplete_short_prompt="Custom prompt for short pauses...",
            incomplete_long_prompt="Custom prompt for long pauses...",
            instructions="Custom turn completion instructions...",
        ),
    ),
)
```

### Parameters

<ParamField type="float">
  Seconds to wait after detecting `○` (incomplete short) before re-prompting the
  LLM. Use shorter values for more responsive re-engagement.
</ParamField>

<ParamField type="float">
  Seconds to wait after detecting `◐` (incomplete long) before re-prompting the
  LLM. Use longer values to give users more time to think.
</ParamField>

<ParamField type="str">
  System prompt sent to the LLM when the short timeout expires. Should instruct
  the LLM to generate a brief, natural prompt encouraging the user to continue.
</ParamField>

<ParamField type="str">
  System prompt sent to the LLM when the long timeout expires. Should instruct
  the LLM to generate a friendly check-in message.
</ParamField>

<ParamField type="str">
  Complete turn completion instructions appended to the system prompt. Override
  this to customize how the LLM determines turn completeness.
</ParamField>

## Markers Explained

### Complete (✓)

The user has provided enough information for a meaningful response:

```
User: "I'd go to Japan because I love the culture and food."
LLM: "✓ Japan is a wonderful choice! The blend of ancient traditions..."
```

The `✓` marker tells the system to push the response normally. The marker itself is not spoken (marked with `skip_tts`).

### Incomplete Short (○)

The user was cut off mid-sentence and will likely continue soon:

```
User: "I'd go to Japan because I love"
LLM: "○"
```

The `○` marker suppresses the bot's response entirely. After 5 seconds (configurable), the LLM is prompted to re-engage with something like "Go ahead, I'm listening."

### Incomplete Long (◐)

The user needs more time to think or explicitly asked for time:

```
User: "That's a good question. Let me think..."
LLM: "◐"
```

The `◐` marker also suppresses the response, but waits 15 seconds (configurable) before prompting. This handles cases like:

* "Hold on a second"
* "Let me think about that"
* "Hmm, that's interesting..."

## Usage Examples

### Basic Usage

Enable turn completion with default settings:

```python theme={null}
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)

messages = [
    {
        "role": "system",
        "content": "You are a helpful assistant...",
    }
]

context = LLMContext(messages)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        filter_incomplete_user_turns=True,
    ),
)
```

<Note>
  You don't need to modify your system prompt. Turn completion instructions are
  automatically appended when `filter_incomplete_user_turns` is enabled.
</Note>

### Custom Timeouts

Adjust timeouts for your use case:

```python theme={null}
from pipecat.turns.user_turn_completion_mixin import UserTurnCompletionConfig

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        filter_incomplete_user_turns=True,
        user_turn_completion_config=UserTurnCompletionConfig(
            incomplete_short_timeout=3.0,  # More responsive
            incomplete_long_timeout=20.0,  # More patient
        ),
    ),
)
```

### Custom Prompts

Customize what the LLM says when re-engaging:

```python theme={null}
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        filter_incomplete_user_turns=True,
        user_turn_completion_config=UserTurnCompletionConfig(
            incomplete_short_prompt="""The user paused briefly.
Generate a contextual prompt to encourage them to continue.
Respond with ✓ followed by your message.""",
            incomplete_long_prompt="""The user has been quiet for a while.
Generate a friendly check-in.
Respond with ✓ followed by your message.""",
        ),
    ),
)
```

<Warning>
  Custom prompts must instruct the LLM to respond with `✓` followed by the
  message. This ensures the re-engagement message is spoken normally.
</Warning>

### With Smart Turn Detection

Combine with smart turn detection for better end-of-turn detection:

```python theme={null}
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_turn_strategies=UserTurnStrategies(
            stop=[
                TurnAnalyzerUserTurnStopStrategy(
                    turn_analyzer=LocalSmartTurnAnalyzerV3()
                )
            ]
        ),
        filter_incomplete_user_turns=True,
    ),
)
```

<Tip>
  Smart turn detection helps determine when the user stops speaking, while turn
  completion filtering determines whether to respond. They work well together
  for natural conversations.
</Tip>

## Transcripts

Turn completion markers are automatically stripped from assistant transcripts emitted via the `on_assistant_turn_stopped` event. Your transcript handlers will receive clean text without markers:

```python theme={null}
@assistant_aggregator.event_handler("on_assistant_turn_stopped")
async def on_assistant_turn_stopped(aggregator, message: AssistantTurnStoppedMessage):
    # message.content will be "Japan is a wonderful choice!"
    # NOT "✓ Japan is a wonderful choice!"
    print(f"Assistant: {message.content}")
```

## Supported LLM Services

Turn completion detection works with any LLM service that inherits from `LLMService`:

* OpenAI (`OpenAILLMService`)
* Anthropic (`AnthropicLLMService`)
* Google Gemini (`GoogleLLMService`)
* AWS Bedrock (`AWSLLMService`)
* And other compatible services

## Graceful Degradation

If the LLM fails to output a turn marker:

1. The system logs a warning indicating markers were expected but not found
2. The buffered text is pushed normally to avoid losing the response
3. The conversation continues without interruption

This ensures the feature doesn't break conversations if the LLM occasionally disobeys instructions.

## Related

* [User Turn Strategies](/server/utilities/turn-management/user-turn-strategies) - Configure turn detection
* [Smart Turn Detection](/server/utilities/smart-turn/smart-turn-overview) - AI-powered end-of-turn detection
* [Transcriptions](/server/utilities/turn-management/transcriptions) - Working with conversation transcripts


# Interruption Strategies
Source: https://docs.pipecat.ai/server/utilities/turn-management/interruption-strategies

Configure when users can interrupt the bot to prevent unwanted interruptions from brief affirmations

<Warning>
  DEPRECATED Interruption strategies have been deprecated in favor of [User Turn
  Strategies](/server/utilities/turn-management/user-turn-strategies).
</Warning>

## Overview

Interruption strategies allow you to control when users can interrupt the bot during speech. By default, any user speech immediately interrupts the bot, but this can be problematic when users engage in backchanneling—brief vocal responses like "yeah", "okay", or "mm-hmm" that indicate they're listening without intending to interrupt.

With interruption strategies, you can require users to meet specific criteria (such as speaking a minimum number of words or reaching a certain audio volume) before their speech will interrupt the bot, creating a more natural conversation flow.

<Tip>
  Want to try it out? Check out the [interruption strategies foundational
  demo](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/42-interruption-config.py)
</Tip>

## Configuration

Interruption strategies are configured via the `interruption_strategies` parameter in `PipelineParams`. When specified, the normal immediate interruption behavior is replaced with conditional interruption based on your criteria.

<ParamField type="List[BaseInterruptionStrategy]">
  List of interruption strategies to apply. When multiple strategies are
  provided, the first one that evaluates to true will trigger the interruption.
  If empty, normal interruption behavior applies.
</ParamField>

## Base Strategy Interface

All interruption strategies inherit from `BaseInterruptionStrategy`, which provides a common interface for evaluating interruption conditions.

<ParamField type="async method">
  Appends audio data to the strategy for analysis. Not all strategies handle
  audio.
</ParamField>

<ParamField type="async method">
  Appends text to the strategy for analysis. Not all strategies handle text.
</ParamField>

<ParamField type="async method">
  Called when the user stops speaking to determine if interruption should occur
  based on accumulated audio and/or text.
</ParamField>

<ParamField type="async method">
  Resets accumulated text and/or audio data.
</ParamField>

## Available Strategies

### MinWordsInterruptionStrategy

Requires users to speak a minimum number of words before interrupting the bot.

<ParamField type="int">
  Minimum number of words the user must speak to interrupt the bot. Must be
  greater than 0.
</ParamField>

```python theme={null}
from pipecat.audio.interruptions.min_words_interruption_strategy import MinWordsInterruptionStrategy

strategy = MinWordsInterruptionStrategy(min_words=3)
```

## How It Works

When interruption strategies are configured:

1. **Bot not speaking**: User speech interrupts immediately (normal behavior)
2. **Bot speaking**: User speech and audio are collected and fed to strategies
3. **User stops speaking**: Strategies are evaluated in order
4. **First match wins**: The first strategy that returns `True` triggers interruption
5. **No matches**: User speech is discarded

The system automatically handles both audio and text input:

* Audio frames (`InputAudioRawFrame`) are fed to `append_audio()`
* Transcription text is fed to `append_text()`
* Strategies can use either or both data types

## Usage Examples

### Basic Word Count Interruption

Require users to speak at least 3 words to interrupt the bot:

```python theme={null}
from pipecat.audio.interruptions.min_words_interruption_strategy import MinWordsInterruptionStrategy
from pipecat.pipeline.task import PipelineParams, PipelineTask

task = PipelineTask(
    pipeline,
    params=PipelineParams(
        allow_interruptions=True,
        interruption_strategies=[MinWordsInterruptionStrategy(min_words=3)]
    )
)
```

### Multiple Strategies with Priority

Strategies are evaluated in order, with the first match triggering interruption:

```python theme={null}
# Prioritize word count, then volume (hypothetical future strategy)
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        allow_interruptions=True,
        interruption_strategies=[
            MinWordsInterruptionStrategy(min_words=2),        # Check first
            # VolumeInterruptionStrategy(min_volume=0.8),     # Your custom strategy
        ]
    )
)
```

## Behavior Comparison

| Scenario                                      | Without Strategy         | With MinWordsInterruptionStrategy(min\_words=3) |
| --------------------------------------------- | ------------------------ | ----------------------------------------------- |
| User says "okay" while bot speaks             | ✅ Interrupts immediately | ❌ Ignored (only 1 word)                         |
| User says "yes that's right" while bot speaks | ✅ Interrupts immediately | ✅ Interrupts (3 words)                          |
| User speaks while bot is silent               | ✅ Processed immediately  | ✅ Processed immediately                         |

## Notes

* Interruption strategies only affect behavior when the bot is actively speaking
* When the bot is not speaking, user input is processed immediately regardless of strategy configuration
* The `allow_interruptions` parameter must be `True` for interruption strategies to work
* User speech that doesn't meet interruption criteria is discarded, not queued
* Strategies are evaluated in order - first match wins
* Both audio and text data are automatically fed to strategies based on their implementation
* Word counting uses simple whitespace splitting for word boundaries


# Transcriptions
Source: https://docs.pipecat.ai/server/utilities/turn-management/transcriptions

Collect user and assistant conversation transcripts using turn events

## Overview

Pipecat provides a straightforward way to collect conversation transcriptions using [turn events](/server/utilities/turn-management/turn-events). When a user or assistant turn ends, the corresponding event includes the complete transcript for that turn.

The key events for transcription collection are:

* **`on_user_turn_stopped`** - Provides the user's complete transcript via `UserTurnStoppedMessage`
* **`on_assistant_turn_stopped`** - Provides the assistant's complete transcript via `AssistantTurnStoppedMessage`

## Basic Example

```python theme={null}
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    UserTurnStoppedMessage,
    AssistantTurnStoppedMessage,
)

# Create context aggregator
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(context)

# Handle user transcriptions
@user_aggregator.event_handler("on_user_turn_stopped")
async def on_user_turn_stopped(aggregator, strategy, message: UserTurnStoppedMessage):
    print(f"[USER] {message.content}")

# Handle assistant transcriptions
@assistant_aggregator.event_handler("on_assistant_turn_stopped")
async def on_assistant_turn_stopped(aggregator, message: AssistantTurnStoppedMessage):
    print(f"[ASSISTANT] {message.content}")
```

## Saving Transcripts to a File

```python theme={null}
import json
from datetime import datetime

transcript_log = []

@user_aggregator.event_handler("on_user_turn_stopped")
async def on_user_turn_stopped(aggregator, strategy, message: UserTurnStoppedMessage):
    transcript_log.append({
        "role": "user",
        "content": message.content,
        "timestamp": message.timestamp,
        "user_id": message.user_id,
    })

@assistant_aggregator.event_handler("on_assistant_turn_stopped")
async def on_assistant_turn_stopped(aggregator, message: AssistantTurnStoppedMessage):
    transcript_log.append({
        "role": "assistant",
        "content": message.content,
        "timestamp": message.timestamp,
    })

# Save transcript when session ends
async def save_transcript():
    with open(f"transcript_{datetime.now().isoformat()}.json", "w") as f:
        json.dump(transcript_log, f, indent=2)
```

## Sending Transcripts to an External Service

```python theme={null}
import aiohttp

async def send_to_service(role: str, content: str, timestamp: str):
    async with aiohttp.ClientSession() as session:
        await session.post(
            "https://api.example.com/transcripts",
            json={"role": role, "content": content, "timestamp": timestamp}
        )

@user_aggregator.event_handler("on_user_turn_stopped")
async def on_user_turn_stopped(aggregator, strategy, message: UserTurnStoppedMessage):
    await send_to_service("user", message.content, message.timestamp)

@assistant_aggregator.event_handler("on_assistant_turn_stopped")
async def on_assistant_turn_stopped(aggregator, message: AssistantTurnStoppedMessage):
    await send_to_service("assistant", message.content, message.timestamp)
```

## Message Types

For details on `UserTurnStoppedMessage` and `AssistantTurnStoppedMessage` fields, see [Turn Events - Message Types](/server/utilities/turn-management/turn-events#message-types).

## Related

* [Turn Events](/server/utilities/turn-management/turn-events) - Complete reference for turn lifecycle events
* [Saving Transcripts Guide](/guides/fundamentals/saving-transcripts) - Guide for saving conversation transcripts


# Turn Events
Source: https://docs.pipecat.ai/server/utilities/turn-management/turn-events

Handle user and assistant turn lifecycle events for transcriptions and turn tracking

## Overview

Turn events provide hooks into the conversation turn lifecycle, allowing you to know when users and assistants start or stop speaking. These events are emitted by the context aggregators (`LLMUserAggregator` and `LLMAssistantAggregator`) and are particularly useful for:

* **Collecting transcriptions** - Get complete user and assistant transcripts when turns end
* **Turn tracking** - Monitor conversation flow and timing
* **Analytics** - Measure turn durations, detect timeouts, and track conversation patterns

## Events Summary

| Event                       | Emitter                | Description                                         |
| --------------------------- | ---------------------- | --------------------------------------------------- |
| `on_user_turn_started`      | `user_aggregator`      | User begins speaking                                |
| `on_user_turn_stopped`      | `user_aggregator`      | User finishes speaking (includes transcript)        |
| `on_user_turn_stop_timeout` | `user_aggregator`      | User turn ended due to timeout                      |
| `on_user_turn_idle`         | `user_aggregator`      | User has been idle (not speaking) for timeout       |
| `on_assistant_turn_started` | `assistant_aggregator` | Assistant begins responding                         |
| `on_assistant_turn_stopped` | `assistant_aggregator` | Assistant finishes responding (includes transcript) |

## User Turn Events

User turn events are registered on the `user_aggregator` from an `LLMContextAggregatorPair`.

### on\_user\_turn\_started

Fired when a user turn is detected to have started, based on the configured [start strategies](/server/utilities/turn-management/user-turn-strategies#start-strategies).

```python theme={null}
@user_aggregator.event_handler("on_user_turn_started")
async def on_user_turn_started(aggregator, strategy):
    print(f"User started speaking (detected by {strategy})")
```

**Parameters:**

| Parameter    | Type                        | Description                                |
| ------------ | --------------------------- | ------------------------------------------ |
| `aggregator` | `LLMUserAggregator`         | The user aggregator instance               |
| `strategy`   | `BaseUserTurnStartStrategy` | The strategy that triggered the turn start |

### on\_user\_turn\_stopped

Fired when a user turn is detected to have ended, based on the configured [stop strategies](/server/utilities/turn-management/user-turn-strategies#stop-strategies). This event includes the complete user transcript for the turn.

```python theme={null}
@user_aggregator.event_handler("on_user_turn_stopped")
async def on_user_turn_stopped(aggregator, strategy, message: UserTurnStoppedMessage):
    print(f"User said: {message.content}")
    print(f"Turn started at: {message.timestamp}")
    if message.user_id:
        print(f"User ID: {message.user_id}")
```

**Parameters:**

| Parameter    | Type                       | Description                                 |
| ------------ | -------------------------- | ------------------------------------------- |
| `aggregator` | `LLMUserAggregator`        | The user aggregator instance                |
| `strategy`   | `BaseUserTurnStopStrategy` | The strategy that triggered the turn stop   |
| `message`    | `UserTurnStoppedMessage`   | Contains the user's transcript and metadata |

### on\_user\_turn\_stop\_timeout

Fired when a user turn times out without any stop strategy triggering. This is a fallback mechanism that ends the turn after a configurable timeout period (default: 5.0 seconds) when the user has stopped speaking according to VAD but no transcription-based stop has occurred. Commonly, this event is used to retrigger the LLM response after the user has stopped speaking.

```python theme={null}
@user_aggregator.event_handler("on_user_turn_stop_timeout")
async def on_user_turn_stop_timeout(aggregator):
    message = {
        "role": "system",
        "content": "Continue.",
    }
    await user_aggregator.queue_frame(LLMMessagesAppendFrame([message], run_llm=True))
```

**Parameters:**

| Parameter    | Type                | Description                  |
| ------------ | ------------------- | ---------------------------- |
| `aggregator` | `LLMUserAggregator` | The user aggregator instance |

<Note>
  After `on_user_turn_stop_timeout` fires, `on_user_turn_stopped` will also be
  called with the accumulated transcript.
</Note>

### on\_user\_turn\_idle

Fired when the user has been idle (not speaking) for a configured timeout period. This event is useful for re-engaging users who may have stepped away or need a prompt to continue the conversation. The idle timer only starts after the first conversation activity and does not trigger while the bot is speaking or function calls are in progress.

```python theme={null}
@user_aggregator.event_handler("on_user_turn_idle")
async def on_user_turn_idle(aggregator):
    # Re-engage the user with a contextual prompt
    message = {
        "role": "system",
        "content": "The user has been quiet. Politely and briefly ask if they're still there.",
    }
    await aggregator.push_frame(LLMMessagesAppendFrame([message], run_llm=True))
```

**Parameters:**

| Parameter    | Type                | Description                  |
| ------------ | ------------------- | ---------------------------- |
| `aggregator` | `LLMUserAggregator` | The user aggregator instance |

<Note>
  The idle timer resets on user or bot speaking activity, and does not trigger
  during function call execution. This event can fire multiple times if the user
  remains idle.
</Note>

## Assistant Turn Events

Assistant turn events are registered on the `assistant_aggregator` from an `LLMContextAggregatorPair`.

### on\_assistant\_turn\_started

Fired when the assistant begins generating a response.

```python theme={null}
@assistant_aggregator.event_handler("on_assistant_turn_started")
async def on_assistant_turn_started(aggregator):
    print("Assistant started responding")
```

**Parameters:**

| Parameter    | Type                     | Description                       |
| ------------ | ------------------------ | --------------------------------- |
| `aggregator` | `LLMAssistantAggregator` | The assistant aggregator instance |

### on\_assistant\_turn\_stopped

Fired when the assistant finishes responding or is interrupted. This event includes the complete assistant transcript for the turn.

```python theme={null}
@assistant_aggregator.event_handler("on_assistant_turn_stopped")
async def on_assistant_turn_stopped(aggregator, message: AssistantTurnStoppedMessage):
    print(f"Assistant said: {message.content}")
    print(f"Turn started at: {message.timestamp}")
```

**Parameters:**

| Parameter    | Type                          | Description                                      |
| ------------ | ----------------------------- | ------------------------------------------------ |
| `aggregator` | `LLMAssistantAggregator`      | The assistant aggregator instance                |
| `message`    | `AssistantTurnStoppedMessage` | Contains the assistant's transcript and metadata |

<Note>
  This event fires when the LLM response completes, when the user interrupts, or
  when a user image is appended to context.
</Note>

## Message Types

### UserTurnStoppedMessage

Contains the user's complete transcript when their turn ends.

```python theme={null}
from pipecat.processors.aggregators.llm_response_universal import UserTurnStoppedMessage
```

<ParamField type="str">
  The complete transcribed text from the user's turn.
</ParamField>

<ParamField type="str">
  ISO 8601 timestamp indicating when the user turn started.
</ParamField>

<ParamField type="Optional[str]">
  Optional identifier for the user, if available from the transport.
</ParamField>

### AssistantTurnStoppedMessage

Contains the assistant's complete transcript when their turn ends.

```python theme={null}
from pipecat.processors.aggregators.llm_response_universal import AssistantTurnStoppedMessage
```

<ParamField type="str">
  The complete text content from the assistant's turn.
</ParamField>

<ParamField type="str">
  ISO 8601 timestamp indicating when the assistant turn started.
</ParamField>

## Related

* [Transcriptions](/server/utilities/turn-management/transcriptions) - Collect conversation transcripts using turn events
* [User Turn Strategies](/server/utilities/turn-management/user-turn-strategies) - Configure turn detection behavior
* [Turn Tracking Observer](/server/utilities/observers/turn-tracking-observer) - Track complete turn cycles with timing


# User Mute Strategies
Source: https://docs.pipecat.ai/server/utilities/turn-management/user-mute-strategies

Control when user input is suppressed during bot operations

## Overview

User mute strategies control whether incoming user input should be suppressed based on the current system state. They determine when user audio and transcriptions should be muted to prevent interruptions during critical bot operations like initial responses or function calls.

By default, user input is never muted. You can configure mute strategies to automatically suppress user input in specific scenarios, such as while the bot is speaking or during function execution. Custom strategies can also be implemented for specific use cases.

## Configuration

User mute strategies are configured via `LLMUserAggregatorParams` when creating an `LLMContextAggregatorPair`:

```python theme={null}
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.turns.user_mute import (
    MuteUntilFirstBotCompleteUserMuteStrategy,
    FunctionCallUserMuteStrategy,
)

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_mute_strategies=[
            MuteUntilFirstBotCompleteUserMuteStrategy(),
            FunctionCallUserMuteStrategy(),
        ],
    ),
)
```

## Available Strategies

### AlwaysUserMuteStrategy

Mutes user input whenever the bot is speaking. This prevents any interruptions during bot speech.

```python theme={null}
from pipecat.turns.user_mute import AlwaysUserMuteStrategy

strategy = AlwaysUserMuteStrategy()
```

**Behavior:**

* Mutes when `BotStartedSpeakingFrame` is received
* Unmutes when `BotStoppedSpeakingFrame` is received

### FirstSpeechUserMuteStrategy

Mutes user input only during the bot's first speech. After the initial response completes, user input is allowed even while the bot is speaking.

```python theme={null}
from pipecat.turns.user_mute import FirstSpeechUserMuteStrategy

strategy = FirstSpeechUserMuteStrategy()
```

**Behavior:**

* Allows user input before bot speaks
* Mutes during the first bot speech only
* Unmutes permanently after first speech completes

<Note>
  Use this strategy when you want to ensure the bot's greeting or initial
  response isn't interrupted, but allow normal interruptions afterward.
</Note>

### MuteUntilFirstBotCompleteUserMuteStrategy

Mutes user input from the start of the interaction until the bot completes its first speech. This ensures the bot maintains full control at the beginning of a conversation.

```python theme={null}
from pipecat.turns.user_mute import MuteUntilFirstBotCompleteUserMuteStrategy

strategy = MuteUntilFirstBotCompleteUserMuteStrategy()
```

**Behavior:**

* Mutes immediately when the pipeline starts (before bot speaks)
* Remains muted until first `BotStoppedSpeakingFrame` is received
* Unmutes permanently after first speech completes

<Note>
  Unlike `FirstSpeechUserMuteStrategy`, this strategy mutes user input even
  before the bot starts speaking. Use this when you don't want to process any
  user input until the bot has delivered its initial message.
</Note>

### FunctionCallUserMuteStrategy

Mutes user input while function calls are executing. This prevents user interruptions during potentially long-running tool operations.

```python theme={null}
from pipecat.turns.user_mute import FunctionCallUserMuteStrategy

strategy = FunctionCallUserMuteStrategy()
```

**Behavior:**

* Mutes when `FunctionCallsStartedFrame` is received
* Tracks multiple concurrent function calls
* Unmutes when all function calls complete (via `FunctionCallResultFrame` or `FunctionCallCancelFrame`)

<Tip>
  This strategy is particularly useful when function calls trigger external API
  requests or database operations that may take several seconds to complete and
  you don't want to the user to interrupt the output.
</Tip>

## Combining Multiple Strategies

Multiple strategies can be combined in a list. The strategies are combined with OR logic—if **any** strategy indicates the user should be muted, user input is suppressed.

```python theme={null}
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_mute_strategies=[
            MuteUntilFirstBotCompleteUserMuteStrategy(),  # Mute until first response
            FunctionCallUserMuteStrategy(),               # Mute during function calls
        ],
    ),
)
```

In this example, user input is muted:

* From pipeline start until the bot completes its first speech
* Whenever function calls are executing (even after first speech)

## Usage Examples

### Prevent Interruptions During Greeting

Ensure the bot's greeting plays completely before accepting user input:

```python theme={null}
from pipecat.turns.user_mute import MuteUntilFirstBotCompleteUserMuteStrategy

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_mute_strategies=[
            MuteUntilFirstBotCompleteUserMuteStrategy(),
        ],
    ),
)
```

### Mute During Function Calls Only

Allow normal interruptions but prevent them during tool execution:

```python theme={null}
from pipecat.turns.user_mute import FunctionCallUserMuteStrategy

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_mute_strategies=[
            FunctionCallUserMuteStrategy(),
        ],
    ),
)
```

### Never Allow Interruptions

Always mute user input while the bot is speaking:

```python theme={null}
from pipecat.turns.user_mute import AlwaysUserMuteStrategy

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_mute_strategies=[
            AlwaysUserMuteStrategy(),
        ],
    ),
)
```

## Event Handlers

You can register event handlers to be notified when user muting starts or stops. This is useful for observability or providing feedback to users.

### Available Events

#### on\_user\_mute\_started

Called when user input becomes muted due to any active mute strategy.

```python theme={null}
@user_aggregator.event_handler("on_user_mute_started")
async def on_user_mute_started(aggregator):
    logger.info("User mute started")
```

#### on\_user\_mute\_stopped

Called when user input is unmuted (no active mute strategies).

```python theme={null}
@user_aggregator.event_handler("on_user_mute_stopped")
async def on_user_mute_stopped(aggregator):
    logger.info("User mute stopped")
```

<Note>
  These events fire whenever the mute state changes, regardless of which
  strategy triggered the change. Use them to provide consistent feedback across
  all mute scenarios.
</Note>

## Related

* [User Turn Strategies](/server/utilities/turn-management/user-turn-strategies) - Configure turn detection behavior
* [User Input Muting Guide](/guides/fundamentals/user-input-muting) - Guide for controlling user input


# User Turn Strategies
Source: https://docs.pipecat.ai/server/utilities/turn-management/user-turn-strategies

Configure how user turns are detected and managed in conversations

## Overview

User turn strategies provide fine-grained control over how user speaking turns are detected in conversations. They determine when a user's turn starts (user begins speaking) and when it stops (user finishes speaking and expects a response).

By default, Pipecat uses a combination of VAD (Voice Activity Detection) and transcription-based detection:

* **Start**: VAD detection or transcription received
* **Stop**: Transcription received after VAD indicates silence

You can customize this behavior by providing your own strategies for more sophisticated turn detection, such as requiring a minimum number of words before triggering a turn, or using AI-powered turn detection models.

## How It Works

1. **Turn Start Detection**: When any start strategy triggers, the user aggregator:

   * Marks the start of a user turn
   * Optionally emits `UserStartedSpeakingFrame`
   * Optionally emits an interruption frame (if the bot is speaking)

2. **During User Turn**: The aggregator collects transcriptions and audio frames.

3. **Turn Stop Detection**: When a stop strategy triggers, the user aggregator:

   * Marks the end of the user turn
   * Emits `UserStoppedSpeakingFrame`
   * Pushes the aggregated user message to the LLM context

4. **Timeout Handling**: If no stop strategy triggers within `user_turn_stop_timeout` seconds (default: 5.0), the turn is automatically ended.

## Configuration

User turn strategies are configured via `LLMUserAggregatorParams` when creating an `LLMContextAggregatorPair`:

```python theme={null}
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.turns.user_turn_strategies import UserTurnStrategies

context = LLMContext(messages)
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_turn_strategies=UserTurnStrategies(
            start=[...],  # List of start strategies
            stop=[...],   # List of stop strategies
        ),
    ),
)
```

## Start Strategies

Start strategies determine when a user's turn begins. Multiple strategies can be provided, and the first one to trigger will signal the start of a user turn.

### Base Parameters

All start strategies inherit these parameters:

<ParamField type="bool">
  If True, the user aggregator will emit an interruption frame when the user
  turn starts, allowing the user to interrupt the bot.
</ParamField>

<ParamField type="bool">
  If True, the user aggregator will emit frames indicating when the user starts
  speaking. Disable this if another component (e.g., an STT service) already
  generates these frames.
</ParamField>

### VADUserTurnStartStrategy

Triggers a user turn start based on Voice Activity Detection. This is the most responsive strategy, detecting speech as soon as the VAD indicates the user has started speaking.

```python theme={null}
from pipecat.turns.user_start import VADUserTurnStartStrategy

strategy = VADUserTurnStartStrategy()
```

### TranscriptionUserTurnStartStrategy

Triggers a user turn start when a transcription is received. This serves as a fallback for scenarios where VAD-based detection fails (e.g., when the user speaks very softly) but the STT service still produces transcriptions.

<ParamField type="bool">
  Whether to trigger on interim (partial) transcription frames for earlier
  detection.
</ParamField>

```python theme={null}
from pipecat.turns.user_start import TranscriptionUserTurnStartStrategy

strategy = TranscriptionUserTurnStartStrategy(use_interim=True)
```

### MinWordsUserTurnStartStrategy

Requires the user to speak a minimum number of words before triggering a turn start. This is useful for preventing brief utterances like "okay" or "yeah" from triggering responses.

<ParamField type="int">
  Minimum number of spoken words required to trigger the start of a user turn.
</ParamField>

<ParamField type="bool">
  Whether to consider interim transcription frames for earlier detection.
</ParamField>

```python theme={null}
from pipecat.turns.user_start import MinWordsUserTurnStartStrategy

# Require at least 3 words to start a turn
strategy = MinWordsUserTurnStartStrategy(min_words=3)
```

<Note>
  When the bot is not speaking, this strategy will trigger after just 1 word.
  The `min_words` threshold only applies when the bot is actively speaking,
  preventing short affirmations from interrupting the bot.
</Note>

### ExternalUserTurnStartStrategy

Delegates turn start detection to an external processor. This strategy listens for `UserStartedSpeakingFrame` frames emitted by other components in the pipeline (such as speech-to-speech services).

```python theme={null}
from pipecat.turns.user_start import ExternalUserTurnStartStrategy

strategy = ExternalUserTurnStartStrategy()
```

<Note>
  This strategy automatically sets `enable_interruptions=False` and
  `enable_user_speaking_frames=False` since these are expected to be handled by
  the external processor.
</Note>

## Stop Strategies

Stop strategies determine when a user's turn ends and the bot should respond.

### Base Parameters

All stop strategies inherit these parameters:

<ParamField type="bool">
  If True, the aggregator will emit frames indicating when the user stops
  speaking. Disable this if another component already generates these frames.
</ParamField>

### TranscriptionUserTurnStopStrategy

The default stop strategy that signals the end of a user turn when transcription is received and VAD indicates silence.

<ParamField type="float">
  A short delay in seconds used to handle consecutive or slightly delayed
  transcriptions gracefully.
</ParamField>

```python theme={null}
from pipecat.turns.user_stop import TranscriptionUserTurnStopStrategy

strategy = TranscriptionUserTurnStopStrategy(timeout=0.5)
```

### TurnAnalyzerUserTurnStopStrategy

Uses an AI-powered turn detection model to determine when the user has finished speaking. This provides more intelligent end-of-turn detection that can understand conversational context.

<ParamField type="BaseTurnAnalyzer">
  The turn detection analyzer instance to use for end-of-turn detection.
</ParamField>

<ParamField type="float">
  A short delay in seconds used to handle consecutive or slightly delayed
  transcriptions.
</ParamField>

```python theme={null}
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy

strategy = TurnAnalyzerUserTurnStopStrategy(
    turn_analyzer=LocalSmartTurnAnalyzerV3()
)
```

<Tip>
  See the [Smart Turn
  Detection](/server/utilities/smart-turn/smart-turn-overview) documentation for
  more information on available turn analyzers.
</Tip>

### ExternalUserTurnStopStrategy

Delegates turn stop detection to an external processor. This strategy listens for `UserStoppedSpeakingFrame` frames emitted by other components in the pipeline.

<ParamField type="float">
  A short delay in seconds used to handle consecutive or slightly delayed
  transcriptions.
</ParamField>

```python theme={null}
from pipecat.turns.user_stop import ExternalUserTurnStopStrategy

strategy = ExternalUserTurnStopStrategy()
```

## UserTurnStrategies

Container for configuring user turn start and stop strategies.

<ParamField type="List[BaseUserTurnStartStrategy]">
  List of strategies used to detect when the user starts speaking. The first
  strategy to trigger will signal the start of the user's turn.
</ParamField>

<ParamField type="List[BaseUserTurnStopStrategy]">
  List of strategies used to detect when the user stops speaking and expects a
  response.
</ParamField>

## ExternalUserTurnStrategies

A convenience class that preconfigures `UserTurnStrategies` with external strategies for both start and stop detection. Use this when an external processor (such as a speech-to-speech service) controls turn management.

```python theme={null}
from pipecat.turns.user_turn_strategies import ExternalUserTurnStrategies

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_turn_strategies=ExternalUserTurnStrategies(),
    ),
)
```

## Usage Examples

### Default Behavior

The default configuration uses VAD and transcription for turn detection:

```python theme={null}
from pipecat.turns.user_turn_strategies import UserTurnStrategies

# This is equivalent to the default behavior
strategies = UserTurnStrategies(
    start=[VADUserTurnStartStrategy(), TranscriptionUserTurnStartStrategy()],
    stop=[TranscriptionUserTurnStopStrategy()],
)
```

### Minimum Words for Interruption

Require users to speak at least 3 words before they can interrupt the bot:

```python theme={null}
from pipecat.turns.user_start import MinWordsUserTurnStartStrategy
from pipecat.turns.user_stop import TranscriptionUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_turn_strategies=UserTurnStrategies(
            start=[MinWordsUserTurnStartStrategy(min_words=3)],
            stop=[TranscriptionUserTurnStopStrategy()],
        ),
    ),
)
```

### Local Smart Turn Detection

Use a local turn detection model instead of a cloud service:

```python theme={null}
from pipecat.audio.turn.smart_turn.local_smart_turn_v3 import LocalSmartTurnAnalyzerV3
from pipecat.turns.user_stop import TurnAnalyzerUserTurnStopStrategy
from pipecat.turns.user_turn_strategies import UserTurnStrategies

user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        user_turn_strategies=UserTurnStrategies(
            stop=[
                TurnAnalyzerUserTurnStopStrategy(
                    turn_analyzer=LocalSmartTurnAnalyzerV3()
                )
            ]
        ),
    ),
)
```

## Related

* [User Input Muting](/guides/fundamentals/user-input-muting) - Control when user input is ignored
* [Smart Turn Detection](/server/utilities/smart-turn/smart-turn-overview) - AI-powered turn detection


# UserIdleProcessor
Source: https://docs.pipecat.ai/server/utilities/user-idle-processor

A processor that monitors user inactivity and triggers callbacks after specified timeout periods

<Warning>
  DEPRECATED: UserIdleProcessor has been deprecated. Use `user_idle_timeout`
  parameter when creating your aggregator, see [Detecting Idle
  Users](/guides/fundamentals/detecting-user-idle) for details.
</Warning>

The `UserIdleProcessor` is a specialized frame processor that monitors user activity in a conversation and executes callbacks when the user becomes idle. It's particularly useful for maintaining engagement by detecting periods of user inactivity and providing escalating responses to inactivity.

## Constructor Parameters

<ParamField type="Union[BasicCallback, RetryCallback]">
  An async function that will be called when user inactivity is detected. Can be
  either:

  * Basic callback: `async def(processor: UserIdleProcessor) -> None`

  * Retry callback: `async def(processor: UserIdleProcessor, retry_count: int) ->
    bool` where returning `False` stops idle monitoring
</ParamField>

<ParamField type="float">
  The number of seconds to wait before considering the user idle.
</ParamField>

## Behavior

The processor starts monitoring for inactivity only after the first conversation activity (either `UserStartedSpeakingFrame` or `BotSpeakingFrame`). It manages idle state based on the following rules:

* Resets idle timer when user starts or stops speaking
* Pauses idle monitoring while user is speaking
* Resets idle timer when bot is speaking
* Stops monitoring on conversation end or cancellation
* Manages a retry count for the retry callback
* Stops monitoring when retry callback returns `False`

## Properties

<ParamField type="int">
  The current number of retry attempts made to engage the user.
</ParamField>

## Example Implementations

Here are two example showing how to use the `UserIdleProcessor`: one with the basic callback and one with the retry callback:

<Tabs>
  <Tab title="Basic Callback">
    ```python theme={null}
    from pipecat.frames.frames import LLMMessagesAppendFrame
    from pipecat.pipeline.pipeline import Pipeline
    from pipecat.processors.user_idle_processor import UserIdleProcessor

    async def handle_idle(user_idle: UserIdleProcessor) -> None:
        await user_idle.push_frame(
            LLMMessagesAppendFrame(
                [
                    {
                        "role": "system",
                        "content": "Ask the user if they are still there and try to prompt for some input.",
                    }
                ],
                run_llm=True,
            )
        )


    # Create the processor
    user_idle = UserIdleProcessor(callback=handle_idle, timeout=5.0)

    # Add to pipeline
    pipeline = Pipeline(
        [
            transport.input(),
            user_idle,  # Add the processor to monitor user activity
            context_aggregator.user(),
            # ... rest of pipeline
        ]
    )
    ```
  </Tab>

  <Tab title="Retry Callback">
    ```python theme={null}
    from pipecat.frames.frames import EndFrame, LLMMessagesAppendFrame, TTSSpeakFrame
    from pipecat.pipeline.pipeline import Pipeline
    from pipecat.processors.user_idle_processor import UserIdleProcessor

    async def handle_user_idle(user_idle: UserIdleProcessor, retry_count: int) -> bool:
        if retry_count == 1:
            # First attempt: Gentle reminder
            await user_idle.push_frame(
                LLMMessagesAppendFrame(
                    [
                        {
                            "role": "system",
                            "content": "The user has been quiet. Politely and briefly ask if they're still there.",
                        }
                    ],
                    run_llm=True,
                )
            )
            return True
        elif retry_count == 2:
            # Second attempt: Direct prompt
            await user_idle.push_frame(
                LLMMessagesAppendFrame(
                    [
                        {
                            "role": "system",
                            "content": "The user is still inactive. Ask if they'd like to continue our conversation.",
                        }
                    ],
                    run_llm=True,
                )
            )
            return True
        else:
            # Third attempt: End conversation
            await user_idle.push_frame(
                TTSSpeakFrame("It seems like you're busy right now. Have a nice day!")
            )
            await task.queue_frame(EndFrame())
            return False  # Stop monitoring


    # Create the processor
    user_idle = UserIdleProcessor(callback=handle_user_idle, timeout=5.0)

    # Add to pipeline
    pipeline = Pipeline(
        [
            transport.input(),
            user_idle,  # Add the processor to monitor user activity
            context_aggregator.user(),
            # ... rest of pipeline
        ]
    )
    ```
  </Tab>
</Tabs>

## Frame Handling

The processor handles the following frame types:

* `UserStartedSpeakingFrame`: Marks user as active, resets idle timer and retry count
* `UserStoppedSpeakingFrame`: Starts idle monitoring
* `BotSpeakingFrame`: Resets idle timer
* `EndFrame` / `CancelFrame`: Stops idle monitoring

## Notes

* The idle callback won't be triggered while the user or bot is actively speaking
* The processor automatically cleans up its resources when the pipeline ends
* Basic callbacks are supported for backward compatibility