Sarvam AI Speech-to-Text

Overview

Sarvam AI offers two real-time speech recognition services for Indian languages:

SarvamSTTService: Uses Sarvam’s transcription WebSocket API with VAD-based segmentation and multiple audio formats (saaras:v4 model by default)
SarvamRealtimeSTTService: Uses Sarvam’s realtime WebSocket endpoint (saaras:v3-realtime and saaras:v4 models) with server-side endpointing, in-band configuration updates, and lower latency

Sarvam STT API Reference

Pipecat’s API methods for Sarvam STT integration

SarvamSTTService Example

Complete example with VAD-based turn detection

SarvamRealtimeSTTService Example

Realtime STT with manual endpointing

Sarvam Documentation

Official Sarvam AI STT documentation and features

Sarvam AI Platform

Access API keys and speech models

Installation

To use Sarvam services, install the required dependency:

uv add "pipecat-ai[sarvam]"

Prerequisites

Sarvam AI Account Setup

Before using Sarvam STT services, you need:

Sarvam AI Account: Sign up at Sarvam AI
API Key: Generate an API key from your account dashboard
Model Access:
- SarvamSTTService: Access to saaras:v3 and saaras:v4 models with support for multiple modes (transcribe, translate, verbatim, translit, codemix)
- SarvamRealtimeSTTService: Access to saaras:v3-realtime and saaras:v4 models

Required Environment Variables

SARVAM_API_KEY: Your Sarvam AI API key for authentication

Configuration

SarvamSTTService

str

required

Sarvam API key for authentication.

str

default:"None"

deprecated

Sarvam model to use. Allowed values: "saaras:v3", "saaras:v4". Deprecated in v0.0.105. Use settings=SarvamSTTService.Settings(...) instead.

int

default:"None"

Audio sample rate in Hz. When None, uses the pipeline’s input sample rate.

Literal['transcribe', 'translate', 'verbatim', 'translit', 'codemix']

default:"None"

Mode of operation. Only applicable to models that support it (e.g., saaras:v3, saaras:v4). Defaults to the model’s default mode.

str

default:"wav"

Audio codec/format of the input file.

list[str]

default:"None"

Connection-time recognition hints for saaras:v4. At most 50 terms of 64 characters each; put each phrase in a separate item. Cannot be updated at runtime.

SarvamSTTService.InputParams

default:"None"

deprecated

Deprecated since 0.0.105, removed in 2.0.0. Use settings=SarvamSTTService.Settings(...) instead.

SarvamSTTService.Settings

default:"None"

Runtime-configurable settings for the STT service. See Settings below.

float

default:"None"

Seconds of no audio before sending silence to keep the connection alive. None disables keepalive.

float

default:"SARVAM_TTFS_P99"

P99 latency from speech end to final transcript in seconds. Override for your deployment. See stt-benchmark.

float

default:"5.0"

Seconds between idle checks when keepalive is enabled.

Settings

Runtime-configurable settings passed via the settings constructor argument using SarvamSTTService.Settings(...). These can be updated mid-conversation with STTUpdateSettingsFrame, except model, vad_signals, and high_vad_sensitivity, which aren’t applied at runtime. See Service Settings for details.

Parameter	Type	Default	Description
`model`	`str`	`"saaras:v4"`	STT model identifier. (Inherited from base STT settings.)
`language`	`Language \| str`	`None`	Target language for transcription. (Inherited from base STT settings.) Defaults to “unknown” (auto-detect) for `saaras:v3` and `saaras:v4`.
`vad_signals`	`bool`	`None`	Enable VAD signals in responses. When enabled, the service proposes turn starts and stops from the server signals, and the user turn strategies resolve them into `UserStartedSpeakingFrame` and `UserStoppedSpeakingFrame`.
`high_vad_sensitivity`	`bool`	`None`	Enable high VAD sensitivity for more responsive speech detection.
`positive_speech_threshold`	`float`	`None`	VAD probability threshold (0.0-1.0) above which a frame is considered speech.
`negative_speech_threshold`	`float`	`None`	VAD probability threshold (0.0-1.0) below which a frame is considered silence.
`min_speech_frames`	`int`	`None`	Minimum consecutive speech frames to start a speech segment.
`first_turn_min_speech_frames`	`int`	`None`	Minimum speech frames for the first user turn.
`negative_frames_count`	`int`	`None`	Number of silence frames within the window to end a speech segment.
`negative_frames_window`	`int`	`None`	Sliding window size (in frames) for counting negative frames.
`start_speech_volume_threshold`	`float`	`None`	Volume level (dB) below which audio is too quiet to be speech.
`interrupt_min_speech_frames`	`int`	`None`	Minimum speech frames to register a barge-in/interruption.
`pre_speech_pad_frames`	`int`	`None`	Number of audio frames to prepend before detected speech onset.
`num_initial_ignored_frames`	`int`	`None`	Number of leading audio frames to skip at connection start.

SarvamRealtimeSTTService

str

required

Sarvam API key for authentication.

str

default:"wss://api.sarvam.ai/speech-to-text-realtime/ws"

Realtime STT websocket endpoint.

Literal['vad', 'manual']

default:"vad"

Which side detects turn boundaries: vad for Sarvam’s own detection, or manual for the pipeline’s. Decides the turn strategies this service asks the user aggregator to run. Defaults to vad.

int

default:"None"

Declared input audio sample rate, 8000 or 16000. None adopts the pipeline’s input rate.

bool

default:"False"

Whether final transcripts should include segment offsets.

list[str]

default:"None"

Connection-time recognition hints for saaras:v4. At most 50 terms of 64 characters each; put each phrase in a separate item. Cannot be updated at runtime.

int

default:"None"

Optional VAD prefix padding, used only under endpointing="vad".

SarvamRealtimeSTTService.Settings

default:"None"

Runtime-updatable realtime settings. See Realtime Settings below.

bool

default:"True"

Whether the bot should be interrupted when Sarvam detects user speech. See User Turn Strategies if you pass your own user_turn_strategies.

float

default:"SARVAM_REALTIME_TTFS_P99"

P99 latency from speech end to final transcript in seconds. Override for your deployment.

bool

default:"True"

Whether to reopen the websocket with backoff when it drops. When False, or once reconnection gives up, transcription stops and the service is no longer usable.

Realtime Settings

Runtime-configurable settings passed via the settings constructor argument using SarvamRealtimeSTTService.Settings(...). These can be updated mid-conversation with STTUpdateSettingsFrame, except model, which isn’t applied at runtime.

Parameter	Type	Default	Description
`model`	`str`	`saaras:v3-realtime`	STT model identifier. Allowed values: `saaras:v3-realtime`, `saaras:v4`.
`language`	`Language \| str \| None`	`Language.EN_IN`	Target language for transcription. Use `None` to have Sarvam detect the language of each utterance. A string that isn’t a `Language` value is sent to Sarvam as-is. (Inherited from base STT settings.)
`language_code`	`str \| None`	`None`	Deprecated since 1.13.0, removed in 2.0.0. Use `language` instead, with `None` for auto-detection.
`stream_type`	`Literal['fast', 'balanced', 'simulated']`	`balanced`	Streaming cadence: `fast` for lowest latency, `balanced` for moderate latency, or `simulated` for testing.
`mode`	`Literal['transcribe', 'translate', 'verbatim', 'translit', 'codemix']`	`transcribe`	Realtime STT task mode.
`prompt`	`str \| None`	`None`	Optional decoding prompt to guide transcription style.
`threshold`	`float \| None`	`None`	Optional VAD sensitivity threshold (only applies under `endpointing="vad"`).
`silence_duration_ms`	`int \| None`	`None`	Optional silence duration for end-of-speech detection (only applies under `endpointing="vad"`).
`min_speech_duration_ms`	`int \| None`	`None`	Optional minimum speech duration (only applies under `endpointing="vad"`).

Usage

Basic Setup (SarvamSTTService)

import os

from pipecat.services.sarvam.stt import SarvamSTTService

stt = SarvamSTTService(
    api_key=os.getenv("SARVAM_API_KEY"),
)

With Language and Model Configuration

from pipecat.services.sarvam.stt import SarvamSTTService
from pipecat.transcriptions.language import Language

stt = SarvamSTTService(
    api_key=os.getenv("SARVAM_API_KEY"),
    mode="transcribe",
    settings=SarvamSTTService.Settings(
        model="saaras:v4",
        language=Language.HI_IN,
    ),
)

With Server-Side VAD

from pipecat.services.sarvam.stt import SarvamSTTService

stt = SarvamSTTService(
    api_key=os.getenv("SARVAM_API_KEY"),
    settings=SarvamSTTService.Settings(
        vad_signals=True,
        high_vad_sensitivity=True,
    ),
)

Basic Realtime Setup

from pipecat.services.sarvam.stt import SarvamRealtimeSTTService

stt = SarvamRealtimeSTTService(
    api_key=os.getenv("SARVAM_API_KEY"),
)

Realtime with Language and Stream Type

from pipecat.services.sarvam.stt import SarvamRealtimeSTTService
from pipecat.transcriptions.language import Language

stt = SarvamRealtimeSTTService(
    api_key=os.getenv("SARVAM_API_KEY"),
    settings=SarvamRealtimeSTTService.Settings(
        language=Language.HI_IN,
        stream_type="fast",
        mode="transcribe",
    ),
)

Realtime with Manual Endpointing

from pipecat.services.sarvam.stt import SarvamRealtimeSTTService
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.transcriptions.language import Language

# With manual endpointing, the pipeline drives turn boundaries
stt = SarvamRealtimeSTTService(
    api_key=os.getenv("SARVAM_API_KEY"),
    endpointing="manual",
    settings=SarvamRealtimeSTTService.Settings(
        language=Language.EN_IN,
        stream_type="balanced",
    ),
)

# A VAD analyzer is required to mark turn boundaries
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()),
)

Updating Realtime Configuration

from pipecat.frames.frames import STTUpdateSettingsFrame

# Update settings mid-conversation without reconnecting
await worker.queue_frame(
    STTUpdateSettingsFrame(
        delta=SarvamRealtimeSTTService.Settings(
            language=Language.TA_IN,
            stream_type="fast",
            threshold=0.5,
        )
    )
)

Notes

SarvamSTTService

Supported models: saaras:v3 and saaras:v4. The default is saaras:v4.
Supported languages: Bengali (bn-IN), Gujarati (gu-IN), Hindi (hi-IN), Kannada (kn-IN), Malayalam (ml-IN), Marathi (mr-IN), Tamil (ta-IN), Telugu (te-IN), Punjabi (pa-IN), Odia (od-IN), English (en-IN), Assamese (as-IN), Urdu (ur-IN), Maithili (mai-IN), Sindhi (sd-IN), and Konkani (kok-IN).
Fine-grained VAD tuning: Both models support server-side VAD with 10 tuning parameters for speech detection thresholds, frame-count controls, pre-speech padding, interruption sensitivity, and initial-frame skipping.
VAD modes: When vad_signals=False (default), the service relies on Pipecat’s local VAD and flushes the server buffer on VADUserStoppedSpeakingFrame. When vad_signals=True, the service uses Sarvam’s server-side VAD and proposes turn starts and stops from the server signals. In that mode it automatically requests ExternalUserTurnStrategies, which resolve those proposals into UserStartedSpeakingFrame and UserStoppedSpeakingFrame, so you don’t need to configure turn strategies manually. Pass your own user_turn_strategies only to override this. This service has no should_interrupt parameter, so the strategies it requests always interrupt the bot when the user starts speaking.

SarvamRealtimeSTTService

Supported sample rates: Only 8000 Hz and 16000 Hz are supported. The service validates the sample rate at initialization.
Supported languages: Auto-detection (language=None), Bengali (bn-IN), Gujarati (gu-IN), Hindi (hi-IN), Kannada (kn-IN), Malayalam (ml-IN), Marathi (mr-IN), Tamil (ta-IN), Telugu (te-IN), Punjabi (pa-IN), Odia (or-IN), English (en-IN), Assamese (as-IN), Urdu (ur-IN), Nepali (ne-IN), Konkani (kok-IN), Kashmiri (ks-IN), Sindhi (sd-IN), Sanskrit (sa-IN), Santali (sat-IN), Manipuri (mni-IN), Bodo (brx-IN), Maithili (mai-IN), and Dogri (doi-IN).
Endpointing modes:
- endpointing="vad" (default): Sarvam’s server-side VAD decides turn boundaries. The service proposes turn frames via ProposedUserStartedSpeakingFrame and ProposedUserStoppedSpeakingFrame.
- endpointing="manual": The pipeline drives turn boundaries via VADUserStartedSpeakingFrame and VADUserStoppedSpeakingFrame. Requires a VAD analyzer on the user aggregator.
VAD analyzer required: A VAD analyzer is required in either endpointing mode. Under vad it times transcription latency; under manual it also marks the turn for Sarvam.
In-band configuration updates: Settings can be updated mid-conversation with STTUpdateSettingsFrame without reconnecting. Connection-only values (sample_rate, return_timestamps, keyterms, prefix_padding_ms, endpointing) cannot be updated at runtime.
Stream types: fast provides lowest latency, balanced provides moderate latency with better quality, and simulated is for testing. The stream type controls server-side flush cadence; the client always sends audio in 50ms chunks.
Reconnection: A dropped websocket is reopened with backoff unless reconnect_on_error=False. On reconnect, the utterance in progress is closed, and under endpointing="manual" an open turn is resent to Sarvam as speech_start. Audio positions for speech_end count from the start of the new session. Once the service stops trying to reconnect it is no longer usable, so a ServiceSwitcher can move on.

Event Handlers

SarvamRealtimeSTTService supports the standard service connection events (on_connected, on_disconnected, on_connection_error). SarvamSTTService provides these events only when constructed with settings=SarvamSTTService.Settings(vad_signals=True):

Event	Description
`on_speech_started`	Sarvam’s VAD detected the start of speech
`on_speech_stopped`	Sarvam’s VAD detected the end of speech
`on_utterance_end`	A transcript message arrived from Sarvam

@stt.event_handler("on_speech_started")
async def on_speech_started(service):
    print("User started speaking")

@stt.event_handler("on_utterance_end")
async def on_utterance_end(service):
    print("Utterance ended")

Get Started

Pipecat Server

Client SDKs

Pipecat Flows

Pipecat Cloud

CLI

Pipecat Context Hub

Sarvam AI Speech-to-Text

Overview

Sarvam STT API Reference

SarvamSTTService Example

SarvamRealtimeSTTService Example

Sarvam Documentation

Sarvam AI Platform

Installation

Prerequisites

Sarvam AI Account Setup

Required Environment Variables

Configuration

SarvamSTTService

Settings

SarvamRealtimeSTTService

Realtime Settings

Usage

Basic Setup (SarvamSTTService)

With Language and Model Configuration

With Server-Side VAD

Basic Realtime Setup

Realtime with Language and Stream Type

Realtime with Manual Endpointing

Updating Realtime Configuration

Notes

SarvamSTTService

SarvamRealtimeSTTService

Event Handlers

​Overview

Sarvam STT API Reference

SarvamSTTService Example

SarvamRealtimeSTTService Example

Sarvam Documentation

Sarvam AI Platform

​Installation

​Prerequisites

​Sarvam AI Account Setup

​Required Environment Variables

​Configuration

​SarvamSTTService

​Settings

​SarvamRealtimeSTTService

​Realtime Settings

​Usage

​Basic Setup (SarvamSTTService)

​With Language and Model Configuration

​With Server-Side VAD

​Basic Realtime Setup

​Realtime with Language and Stream Type

​Realtime with Manual Endpointing

​Updating Realtime Configuration

​Notes

​SarvamSTTService

​SarvamRealtimeSTTService

​Event Handlers

Overview

Installation

Prerequisites

Sarvam AI Account Setup

Required Environment Variables

Configuration

SarvamSTTService

Settings

SarvamRealtimeSTTService

Realtime Settings

Usage

Basic Setup (SarvamSTTService)

With Language and Model Configuration

With Server-Side VAD

Basic Realtime Setup

Realtime with Language and Stream Type

Realtime with Manual Endpointing

Updating Realtime Configuration

Notes

SarvamSTTService

SarvamRealtimeSTTService

Event Handlers