> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# OpenTelemetry Tracing

> Monitor and analyze your Pipecat conversational pipelines using OpenTelemetry

## Overview

Pipecat includes built-in support for OpenTelemetry tracing, allowing you to gain deep visibility into your voice applications. Tracing helps you:

* Track latency and performance across your conversation pipeline
* Monitor service health and identify bottlenecks
* Visualize conversation turns and service dependencies
* Collect usage metrics and operational analytics

## Installation

To use OpenTelemetry tracing with Pipecat, install the tracing dependencies:

```bash theme={null}
uv add "pipecat-ai[tracing]"
```

<Tip>
  For local development and testing, we recommend using Jaeger as a trace collector.
  You can run it with Docker:

  ```bash theme={null}
  docker run -d --name jaeger \
    -p 16686:16686 \
    -p 4317:4317 \
    -p 4318:4318 \
    jaegertracing/all-in-one:latest
  ```

  Then access the UI at [http://localhost:16686](http://localhost:16686)
</Tip>

## Basic Setup

Enabling tracing in your Pipecat application requires two steps:

1. **Initialize the OpenTelemetry SDK** with your preferred exporter
2. **Enable tracing in your PipelineTask**

```python theme={null}
import os
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from pipecat.utils.tracing.setup import setup_tracing
from pipecat.pipeline.task import PipelineTask, PipelineParams

# Step 1: Initialize OpenTelemetry with your chosen exporter
exporter = OTLPSpanExporter(
    endpoint="http://localhost:4317",  # Jaeger or other collector endpoint
    insecure=True,
)

setup_tracing(
    service_name="my-voice-app",
    exporter=exporter,
    console_export=False,  # Set to True for debug output
)

# Step 2: Enable tracing in your PipelineTask
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        enable_metrics=True,                              # Required for some service metrics
    ),
    enable_tracing=True,                                  # Enable tracing for this task
    enable_turn_tracking=True,                            # Enable turn tracking for this task
    conversation_id="customer-123",                       # Optional - will auto-generate if not provided
    additional_span_attributes={"session.id": "abc-123"} # Optional - additional attributes to attach to the otel span
)
```

<Info>
  For complete working examples, see our sample implementations:

  * [Jaeger Tracing Example](https://github.com/pipecat-ai/pipecat-examples/tree/main/open-telemetry/jaeger) - Uses gRPC exporter with Jaeger
  * [Langfuse Tracing Example](https://github.com/pipecat-ai/pipecat-examples/tree/main/open-telemetry/langfuse) - Uses HTTP exporter with Langfuse for LLM-focused observability
</Info>

## Trace Structure

Pipecat organizes traces hierarchically, following the natural structure of conversations:

```
Conversation (conversation)
├── turn
│   ├── stt
│   ├── llm
│   └── tts
└── turn
    ├── stt
    ├── llm
    └── tts
    turn...
```

For real-time multimodal services like Gemini Live and OpenAI Realtime, the structure adapts to their specific patterns:

```
Conversation (conversation)
├── turn
│   ├── llm_setup (session configuration)
│   ├── stt (user input)
│   ├── llm_response (complete response with usage)
│   └── llm_tool_call/llm_tool_result (for function calls)
└── turn
    ├── stt (user input)
    └── llm_response (complete response)
    turn...
```

This hierarchical structure makes it easy to:

* Track the full lifecycle of a conversation
* Measure latency for individual turns
* Identify which services are contributing to delays
* Compare performance across different conversations

## Exporter Options

Pipecat supports any OpenTelemetry-compatible exporter. Common options include:

### OTLP Exporter (for Jaeger, Grafana, etc.)

```python theme={null}
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(
    endpoint="http://localhost:4317",  # Your collector endpoint
    insecure=True,  # Use False for TLS connections
)
```

### HTTP OTLP Exporter (for Langfuse, etc.)

```python theme={null}
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(
    # Configure with environment variables:
    # OTEL_EXPORTER_OTLP_ENDPOINT
    # OTEL_EXPORTER_OTLP_HEADERS
)
```

See our [Langfuse example](https://github.com/pipecat-ai/pipecat-examples/tree/main/open-telemetry/langfuse) for details on configuring this exporter.

### Console Exporter (for debugging)

The console exporter can be enabled alongside any other exporter by setting `console_export=True`:

```python theme={null}
setup_tracing(
    service_name="my-voice-app",
    exporter=otlp_exporter,
    console_export=True,  # Prints traces to stdout
)
```

### Cloud Provider Exporters

Many cloud providers offer OpenTelemetry-compatible observability services:

* **AWS X-Ray**
* **Google Cloud Trace**
* **Azure Monitor**
* **Datadog APM**

Check the OpenTelemetry documentation for specific exporter configurations: [OpenTelemetry Vendors](https://opentelemetry.io/ecosystem/vendors/)

### OpenInference

Arize-ai provides OpenInference instrumentation, compatible with OpenTelemetry.

* [openinference-instrumentation-pipecat](https://github.com/Arize-ai/openinference/tree/main/python/instrumentation/openinference-instrumentation-pipecat)

See [Observability community integrations](https://docs.pipecat.ai/server/services/community-integrations#observability) for more details.

### Future AGI

Future AGI provides OpenTelemetry attribute mapping for Pipecat traces, compatible with OpenTelemetry.

* [traceAI-pipecat](https://github.com/future-agi/traceAI/tree/main/python/frameworks/pipecat)

See [Observability community integrations](https://docs.pipecat.ai/server/services/community-integrations#observability) for more details.

## Span Attributes

Pipecat enriches spans with detailed attributes about service operations:

### TTS Service Spans

* `gen_ai.system`: Service provider (e.g., "cartesia")
* `gen_ai.request.model`: Model ID/name
* `voice_id`: Voice identifier
* `text`: The text being synthesized
* `metrics.character_count`: Number of characters in the text
* `metrics.ttfb`: Time to first byte in seconds
* `settings.*`: Service-specific configuration parameters

### STT Service Spans

* `gen_ai.system`: Service provider (e.g., "deepgram")
* `gen_ai.request.model`: Model ID/name
* `transcript`: The transcribed text
* `is_final`: Whether the transcription is final
* `language`: Detected or configured language
* `vad_enabled`: Whether voice activity detection is enabled
* `metrics.ttfb`: Time to first byte in seconds
* `settings.*`: Service-specific configuration parameters

### LLM Service Spans

* `gen_ai.system`: Service provider (e.g., "openai", "gcp.gemini")
* `gen_ai.request.model`: Model ID/name
* `gen_ai.operation.name`: Operation type (e.g., "chat")
* `stream`: Whether streaming is enabled
* `input`: JSON-serialized input messages
* `output`: Complete response text
* `tools`: JSON-serialized tools configuration
* `tools.count`: Number of tools available
* `tools.names`: Comma-separated tool names
* `system`: System message content
* `gen_ai.usage.input_tokens`: Number of prompt tokens
* `gen_ai.usage.output_tokens`: Number of completion tokens
* `metrics.ttfb`: Time to first byte in seconds
* `gen_ai.request.*`: Standard parameters (temperature, max\_tokens, etc.)

### Multimodal Service Spans (Gemini Live & OpenAI Realtime)

#### Setup Spans

* `gen_ai.system`: "gcp.gemini" or "openai"
* `gen_ai.request.model`: Model identifier
* `tools.count`: Number of available tools
* `tools.definitions`: JSON-serialized tool schemas
* `system_instruction`: System prompt (truncated)
* `session.*`: Session configuration parameters

#### Request Spans (OpenAI Realtime)

* `input`: JSON-serialized context messages being sent
* `gen_ai.operation.name`: "llm\_request"

#### Response Spans

* `output`: Complete assistant response text
* `output_modality`: "TEXT" or "AUDIO" (Gemini Live)
* `gen_ai.usage.input_tokens`: Prompt tokens used
* `gen_ai.usage.output_tokens`: Completion tokens generated
* `function_calls.count`: Number of function calls made
* `function_calls.names`: Comma-separated function names
* `metrics.ttfb`: Time to first response in seconds

#### Tool Call/Result Spans (Gemini Live)

* `tool.function_name`: Name of the function being called
* `tool.call_id`: Unique identifier for the call
* `tool.arguments`: Function arguments (truncated)
* `tool.result`: Function execution result (truncated)
* `tool.result_status`: "completed", "error", or "parse\_error"

### Turn Spans

* `turn.number`: Sequential turn number
* `turn.type`: Type of turn (e.g., "conversation")
* `turn.duration_seconds`: Duration of the turn
* `turn.was_interrupted`: Whether the turn was interrupted
* `conversation.id`: ID of the parent conversation

### Conversation Spans

* `conversation.id`: Unique identifier for the conversation
* `conversation.type`: Type of conversation (e.g., "voice")

## Usage Metrics

Pipecat's tracing implementation automatically captures usage metrics for LLM and TTS services:

### LLM Token Usage

Token usage is captured in LLM spans as:

* `gen_ai.usage.input_tokens`
* `gen_ai.usage.output_tokens`

### TTS Character Count

Character count is captured in TTS spans as:

* `metrics.character_count`

## Performance Metrics

Pipecat traces capture key performance metrics for each service:

### Time To First Byte (TTFB)

The time it takes for a service to produce its first response:

* `metrics.ttfb` (in seconds)

### Processing Duration

The total time spent processing in each service is captured in the span duration.

## Configuration Options

### PipelineTask Parameters

<ParamField path="enable_tracing" type="bool" default="True">
  Enable or disable tracing for the pipeline
</ParamField>

<ParamField path="enable_turn_tracking" type="bool" default="False">
  Whether to enable turn tracking.
</ParamField>

<ParamField path="conversation_id" type="str | None" default="None">
  Custom ID for the conversation. If not provided, a UUID will be generated
</ParamField>

<ParamField path="additional_span_attributes" type="dict | None" default="None">
  Any additional attributes to add to top-level OpenTelemetry conversation span.
</ParamField>

### setup\_tracing() Parameters

<ParamField path="service_name" type="str" default="pipecat">
  Name of the service for traces
</ParamField>

<ParamField path="exporter" type="SpanExporter | None" default="None">
  A pre-configured OpenTelemetry span exporter instance
</ParamField>

<ParamField path="console_export" type="bool" default="False">
  Whether to also export traces to console (useful for debugging)
</ParamField>

## Example

Here's a complete example showing OpenTelemetry tracing setup with Jaeger:

```python theme={null}
import os
from dotenv import load_dotenv
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.services.deepgram.stt import DeepgramSTTService
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.services.cartesia.tts import CartesiaTTSService
from pipecat.utils.tracing.setup import setup_tracing

load_dotenv()

# Initialize tracing if enabled
if os.getenv("ENABLE_TRACING"):
    # Create the exporter
    otlp_exporter = OTLPSpanExporter(
        endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317"),
        insecure=True,
    )

    # Set up tracing with the exporter
    setup_tracing(
        service_name="pipecat-demo",
        exporter=otlp_exporter,
        console_export=bool(os.getenv("OTEL_CONSOLE_EXPORT")),
    )

# Create your services
stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"))
tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    voice_id="your-voice-id"
)

# Build pipeline
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),
    context_aggregator.assistant(),
])

# Create pipeline task with tracing enabled
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        enable_metrics=True,
        enable_usage_metrics=True,
    ),
    enable_tracing=True,
    enable_turn_tracking=True,
    conversation_id="customer-123",                       # Optional - will auto-generate if not provided
    additional_span_attributes={"session.id": "abc-123"} # Optional - additional attributes to attach to the otel span
)

# Run the pipeline
runner = PipelineRunner()
await runner.run(task)
```

## Troubleshooting

If you're having issues with tracing:

* **No Traces Visible**: Ensure the OpenTelemetry packages are installed and that your collector endpoint is correct
* **Missing Service Data**: Verify that `enable_metrics=True` is set in PipelineParams
* **Debugging Tracing**: Enable console export with `console_export=True` to view traces in your logs
* **Connection Errors**: Check network connectivity to your trace collector
* **Collector Configuration**: Verify your collector is properly set up to receive traces

## References

* [OpenTelemetry Python Documentation](https://opentelemetry-python.readthedocs.io/)
* [OpenTelemetry Tracing Specification](https://opentelemetry.io/docs/reference/specification/trace/api/)
* [Jaeger Documentation](https://www.jaegertracing.io/docs/latest/)
* [Langfuse OpenTelemetry Documentation](https://langfuse.com/docs/opentelemetry/get-started)
* [Future AGI Documentation](https://docs.futureagi.com)