Gemini Live

​

Overview

​

Installation

uv add "pipecat-ai[google]"

​

Prerequisites

​

Google AI Setup

1. Google Account: Set up at Google AI Studio
2. API Key: Generate a Gemini API key from AI Studio
3. Model Access: Ensure access to Gemini Live models
4. Multimodal Configuration: Set up audio, video, and text modalities

​

Required Environment Variables

• GOOGLE_API_KEY: Your Google Gemini API key for authentication

​

Key Features

• Multimodal Processing: Handle audio, video, and text inputs simultaneously
• Real-time Streaming: Low-latency audio and video processing
• Voice Activity Detection: Automatic speech detection and turn management
• Function Calling: Advanced tool integration and API calling capabilities
• Context Management: Intelligent conversation history and system instruction handling

​

Configuration

​

GeminiLiveLLMService

​

Settings

​

GeminiVADParams

​

ContextWindowCompressionParams

​

Usage

​

Basic Setup

import os
from pipecat.services.google.gemini_live.llm import GeminiLiveLLMService

llm = GeminiLiveLLMService(
    api_key=os.getenv("GOOGLE_API_KEY"),
    settings=GeminiLiveLLMService.Settings(
        voice="Charon",
        system_instruction="You are a helpful assistant.",
    ),
)

​

With Settings

from pipecat.services.google.gemini_live.llm import GeminiLiveLLMService, GeminiVADParams, ContextWindowCompressionParams

llm = GeminiLiveLLMService(
    api_key=os.getenv("GOOGLE_API_KEY"),
    settings=GeminiLiveLLMService.Settings(
        model="models/gemini-2.5-flash-native-audio-preview-12-2025",
        system_instruction="You are a helpful assistant.",
        voice="Puck",
        temperature=0.7,
        max_tokens=2048,
        language="en-US",
        vad=GeminiVADParams(
            silence_duration_ms=500,
        ),
        context_window_compression={"enabled": True},
    ),
)

​

With Local VAD

from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.services.google.gemini_live.llm import GeminiLiveLLMService, GeminiVADParams
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)

llm = GeminiLiveLLMService(
    api_key=os.getenv("GOOGLE_API_KEY"),
    settings=GeminiLiveLLMService.Settings(
        voice="Charon",
        vad=GeminiVADParams(disabled=True),  # Disable server-side VAD
    ),
)

# Configure local VAD in your aggregator. The local VAD drives turn-taking
# since server-side VAD is disabled.
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(
        vad_analyzer=SileroVADAnalyzer(),
    ),
)

​

Text-Only Mode

from pipecat.services.google.gemini_live.llm import GeminiLiveLLMService, GeminiModalities

llm = GeminiLiveLLMService(
    api_key=os.getenv("GOOGLE_API_KEY"),
    settings=GeminiLiveLLMService.Settings(
        system_instruction="You are a helpful assistant.",
        modalities=GeminiModalities.TEXT,
    ),
)

​

With Thinking Enabled

from google.genai.types import ThinkingConfig

llm = GeminiLiveLLMService(
    api_key=os.getenv("GOOGLE_API_KEY"),
    settings=GeminiLiveLLMService.Settings(
        model="models/gemini-2.5-flash-native-audio-preview-12-2025",
        system_instruction="You are a helpful assistant.",
        thinking=ThinkingConfig(include_thoughts=True),
    ),
)

​

Methods

​

set_audio_input_paused()

def set_audio_input_paused(paused: bool)

​

set_video_input_paused()

def set_video_input_paused(paused: bool)

​

file_api

llm.file_api -> GeminiFileAPI

from pipecat.processors.aggregators.llm_context import LLMContext

file_info = await llm.file_api.upload_file("menu.txt", display_name="Menu")

context = LLMContext(
    [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What's on the menu today?"},
                {
                    "type": "file_data",
                    "file_data": {
                        "mime_type": "text/plain",
                        "file_uri": file_info["file"]["uri"],
                    },
                },
            ],
        }
    ]
)

​

Notes

• Model support: The service supports Gemini 2.5, Gemini 3.x, and the 3.8 Live family (gemini-3.8-live, gemini-3.8-live-extended-thinking). The service automatically detects and handles model-specific behavior.
• Function calling behavior: Functions registered with cancel_on_interruption=False use Gemini’s NON_BLOCKING tool mechanism, allowing the conversation to continue while the tool runs in the background. The 3.8 Live family defaults to NON_BLOCKING for all function calls; synchronous tools are explicitly declared BLOCKING on gemini-3.8-live to maintain blocking semantics. The thinking model gemini-3.8-live-extended-thinking accepts only NON_BLOCKING and logs a warning for synchronous tools. Earlier Gemini 3.x models (before 3.8) do not support NON_BLOCKING, and the service pushes an error when a function registered with cancel_on_interruption=False is used with them. Note: An intermittent 1008 error can occasionally occur on Gemini 2.5 during long-running tool calls; the service auto-reconnects when this happens.
• Thinking level defaults: Live thinking models (e.g., gemini-3.8-live-extended-thinking) require a thinking_level in the thinking configuration. When none is set, the service applies LOW as the default to keep reply latency down.
• Turn completion with interaction status: On thinking models from 3.8 onward, the service holds the bot turn open while Gemini reports interaction_status: IN_PROGRESS, ensuring that a reply spanning multiple turn_complete messages is recorded as a single assistant turn. This requires google-genai>=2.19.0; older SDKs log a warning.
• Tool calls in conversation history: When seeding a session with conversation history that contains tool calls (e.g., after a multi-worker handoff or reconnect), the service automatically converts function calls and their results to text summaries. This is required because Gemini Live’s API only accepts text and media content. The conversion happens transparently and preserves the context for the model.
• System instruction precedence: The system_instruction from service settings takes precedence over an initial system message in the LLM context. A warning is logged when both are set.
• VAD modes: By default, Gemini Live uses server-side VAD for detecting when the user starts and stops speaking. To use local VAD (e.g., Silero), set vad=GeminiVADParams(disabled=True) and configure an external VAD analyzer in your LLMUserAggregatorParams. The service will automatically send activity signals to the Gemini API when local VAD detects speech. With server-side VAD, the service emits no UserStartedSpeakingFrame or UserStoppedSpeakingFrame, so processors that depend on them (RTVI user speaking events, user idle and mute handling, turn recording) do nothing unless LLMUserAggregatorParams has a vad_analyzer.
• Tools precedence: Similarly, tools provided in the context override tools provided at init time. Tools can’t change mid-session: the service drops LLMSetToolsFrame.
• Seed parameter: Gemini treats the seed parameter as best effort. Identical seeds usually but not always produce identical responses.
• Transcription aggregation: Gemini Live sends user transcriptions in small chunks. The service aggregates them into complete sentences using end-of-sentence detection with a 0.5-second timeout fallback.
• Session resumption: The service automatically handles session resumption on reconnection using session resumption handles. In the rare case where reconnection occurs before a resumption handle is received, conversation history is preserved by reseeding it into the new session.
• Connection resilience: The service reconnects after a connection error. On the third consecutive error it reports an error and stops trying.
• Video frame rate: Video frames are throttled to a maximum of one per second.
• Affective dialog and proactivity: These features require both a supporting model and API version (v1alpha).