OpenTelemetry Tracing

​

Overview

• Track latency and performance across your conversation pipeline
• Monitor service health and identify bottlenecks
• Visualize conversation turns and service dependencies
• Collect usage metrics and operational analytics

​

Installation

pip install "pipecat-ai[tracing]"

docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  jaegertracing/all-in-one:latest

​

Basic Setup

1. Initialize the OpenTelemetry SDK with your preferred exporter
2. Enable tracing in your PipelineTask

import os
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from pipecat.utils.tracing.setup import setup_tracing
from pipecat.pipeline.task import PipelineTask, PipelineParams

# Step 1: Initialize OpenTelemetry with your chosen exporter
exporter = OTLPSpanExporter(
    endpoint="http://localhost:4317",  # Jaeger or other collector endpoint
    insecure=True,
)

setup_tracing(
    service_name="my-voice-app",
    exporter=exporter,
    console_export=False,  # Set to True for debug output
)

# Step 2: Enable tracing in your PipelineTask
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        enable_metrics=True,                              # Required for some service metrics
    ),
    enable_tracing=True,                                  # Enable tracing for this task
    enable_turn_tracking=True,                            # Enable turn tracking for this task
    conversation_id="customer-123",                       # Optional - will auto-generate if not provided
    additional_span_attributes={"session.id": "abc-123"} # Optional - additional attributes to attach to the otel span
)

​

Trace Structure

Conversation (conversation)
├── turn
│   ├── stt
│   ├── llm
│   └── tts
└── turn
    ├── stt
    ├── llm
    └── tts
    turn...

Conversation (conversation)
├── turn
│   ├── llm_setup (session configuration)
│   ├── stt (user input)
│   ├── llm_response (complete response with usage)
│   └── llm_tool_call/llm_tool_result (for function calls)
└── turn
    ├── stt (user input)
    └── llm_response (complete response)
    turn...

• Track the full lifecycle of a conversation
• Measure latency for individual turns
• Identify which services are contributing to delays
• Compare performance across different conversations

​

Exporter Options

​

OTLP Exporter (for Jaeger, Grafana, etc.)

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(
    endpoint="http://localhost:4317",  # Your collector endpoint
    insecure=True,  # Use False for TLS connections
)

​

HTTP OTLP Exporter (for Langfuse, etc.)

from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(
    # Configure with environment variables:
    # OTEL_EXPORTER_OTLP_ENDPOINT
    # OTEL_EXPORTER_OTLP_HEADERS
)

​

Console Exporter (for debugging)

setup_tracing(
    service_name="my-voice-app",
    exporter=otlp_exporter,
    console_export=True,  # Prints traces to stdout
)

​

Cloud Provider Exporters

• AWS X-Ray
• Google Cloud Trace
• Azure Monitor
• Datadog APM

​

Span Attributes

​

TTS Service Spans

• gen_ai.system: Service provider (e.g., “cartesia”)
• gen_ai.request.model: Model ID/name
• voice_id: Voice identifier
• text: The text being synthesized
• metrics.character_count: Number of characters in the text
• metrics.ttfb: Time to first byte in seconds
• settings.*: Service-specific configuration parameters

​

STT Service Spans

• gen_ai.system: Service provider (e.g., “deepgram”)
• gen_ai.request.model: Model ID/name
• transcript: The transcribed text
• is_final: Whether the transcription is final
• language: Detected or configured language
• vad_enabled: Whether voice activity detection is enabled
• metrics.ttfb: Time to first byte in seconds
• settings.*: Service-specific configuration parameters

​

LLM Service Spans

• gen_ai.system: Service provider (e.g., “openai”, “gcp.gemini”)
• gen_ai.request.model: Model ID/name
• gen_ai.operation.name: Operation type (e.g., “chat”)
• stream: Whether streaming is enabled
• input: JSON-serialized input messages
• output: Complete response text
• tools: JSON-serialized tools configuration
• tools.count: Number of tools available
• tools.names: Comma-separated tool names
• system: System message content
• gen_ai.usage.input_tokens: Number of prompt tokens
• gen_ai.usage.output_tokens: Number of completion tokens
• metrics.ttfb: Time to first byte in seconds
• gen_ai.request.*: Standard parameters (temperature, max_tokens, etc.)

​

Multimodal Service Spans (Gemini Live & OpenAI Realtime)

​

Setup Spans

• gen_ai.system: “gcp.gemini” or “openai”
• gen_ai.request.model: Model identifier
• tools.count: Number of available tools
• tools.definitions: JSON-serialized tool schemas
• system_instruction: System prompt (truncated)
• session.*: Session configuration parameters

​

Request Spans (OpenAI Realtime)

• input: JSON-serialized context messages being sent
• gen_ai.operation.name: “llm_request”

​

Response Spans

• output: Complete assistant response text
• output_modality: “TEXT” or “AUDIO” (Gemini Live)
• gen_ai.usage.input_tokens: Prompt tokens used
• gen_ai.usage.output_tokens: Completion tokens generated
• function_calls.count: Number of function calls made
• function_calls.names: Comma-separated function names
• metrics.ttfb: Time to first response in seconds

​

Tool Call/Result Spans (Gemini Live)

• tool.function_name: Name of the function being called
• tool.call_id: Unique identifier for the call
• tool.arguments: Function arguments (truncated)
• tool.result: Function execution result (truncated)
• tool.result_status: “completed”, “error”, or “parse_error”

​

Turn Spans

• turn.number: Sequential turn number
• turn.type: Type of turn (e.g., “conversation”)
• turn.duration_seconds: Duration of the turn
• turn.was_interrupted: Whether the turn was interrupted
• conversation.id: ID of the parent conversation

​

Conversation Spans

• conversation.id: Unique identifier for the conversation
• conversation.type: Type of conversation (e.g., “voice”)

​

Usage Metrics

​

LLM Token Usage

• gen_ai.usage.input_tokens
• gen_ai.usage.output_tokens

​

TTS Character Count

• metrics.character_count

​

Performance Metrics

​

Time To First Byte (TTFB)

• metrics.ttfb (in seconds)

​

Processing Duration

​

Configuration Options

​

PipelineTask Parameters

​

setup_tracing() Parameters

​

Example

import os
from dotenv import load_dotenv
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.services.deepgram.stt import DeepgramSTTService
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.services.cartesia.tts import CartesiaTTSService
from pipecat.utils.tracing.setup import setup_tracing

load_dotenv()

# Initialize tracing if enabled
if os.getenv("ENABLE_TRACING"):
    # Create the exporter
    otlp_exporter = OTLPSpanExporter(
        endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:4317"),
        insecure=True,
    )

    # Set up tracing with the exporter
    setup_tracing(
        service_name="pipecat-demo",
        exporter=otlp_exporter,
        console_export=bool(os.getenv("OTEL_CONSOLE_EXPORT")),
    )

# Create your services
stt = DeepgramSTTService(api_key=os.getenv("DEEPGRAM_API_KEY"))
llm = OpenAILLMService(api_key=os.getenv("OPENAI_API_KEY"))
tts = CartesiaTTSService(
    api_key=os.getenv("CARTESIA_API_KEY"),
    voice_id="your-voice-id"
)

# Build pipeline
pipeline = Pipeline([
    transport.input(),
    stt,
    context_aggregator.user(),
    llm,
    tts,
    transport.output(),
    context_aggregator.assistant(),
])

# Create pipeline task with tracing enabled
task = PipelineTask(
    pipeline,
    params=PipelineParams(
        allow_interruptions=True,
        enable_metrics=True,
        enable_usage_metrics=True,
    ),
    enable_tracing=True,
    enable_turn_tracking=True,
    conversation_id="customer-123",                       # Optional - will auto-generate if not provided
    additional_span_attributes={"session.id": "abc-123"} # Optional - additional attributes to attach to the otel span
)

# Run the pipeline
runner = PipelineRunner()
await runner.run(task)

​

Troubleshooting

• No Traces Visible: Ensure the OpenTelemetry packages are installed and that your collector endpoint is correct
• Missing Service Data: Verify that enable_metrics=True is set in PipelineParams
• Debugging Tracing: Enable console export with console_export=True to view traces in your logs
• Connection Errors: Check network connectivity to your trace collector
• Collector Configuration: Verify your collector is properly set up to receive traces

​

References

• OpenTelemetry Python Documentation
• OpenTelemetry Tracing Specification
• Jaeger Documentation
• Langfuse OpenTelemetry Documentation