NVIDIA Nemotron Speech-to-Text

Overview

NVIDIA Nemotron Speech provides three STT service implementations:

NvidiaSTTService — Real-time streaming transcription using Nemotron ASR Streaming models with interim results and continuous audio processing.
NvidiaSegmentedSTTService — Segmented transcription using Canary models with advanced language support, word boosting, and enterprise-grade accuracy.
NvidiaSageMakerSTTService — Streaming transcription using NVIDIA Nemotron ASR via an AWS SageMaker bidirectional-stream endpoint with automatic reconnection on error.

NVIDIA Nemotron Speech STT API Reference

Pipecat’s API methods for NVIDIA Nemotron Speech STT integration

Example Implementation

Complete example with NVIDIA services integration

NVIDIA ASR NIM Documentation

Official NVIDIA ASR NIM documentation

NVIDIA Developer Portal

Access API keys and Nemotron Speech services

Installation

To use NVIDIA Nemotron Speech services, install the required dependency:

uv add "pipecat-ai[nvidia]"

Prerequisites

NVIDIA Nemotron Speech Setup

Before using NVIDIA Nemotron Speech STT services, you need:

NVIDIA Developer Account (for cloud deployments): Sign up at NVIDIA Developer Portal
API Key (for cloud deployments): Generate an NVIDIA API key for Nemotron Speech services
Model Selection: Choose between Nemotron ASR Streaming (streaming) and Canary (segmented) models

For local deployments, you can run NVIDIA ASR NIM locally without an API key. See the NVIDIA ASR NIM documentation for deployment instructions.

Environment Variables

NVIDIA_API_KEY: Your NVIDIA API key for authentication (required for cloud endpoint, not needed for local deployments)

NvidiaSTTService

Real-time streaming transcription using NVIDIA Nemotron Speech’s streaming ASR models.

str | None

default:"None"

NVIDIA API key for authentication. Required when using the cloud endpoint. Not needed for local deployments.

str

default:"grpc.nvcf.nvidia.com:443"

NVIDIA Nemotron Speech server address. For local deployments, pass the local address (e.g. localhost:50051).

Mapping[str, str]

Mapping containing function_id and model_name for the ASR model.

int

default:"None"

Audio sample rate in Hz. When None, uses the pipeline’s configured sample rate.

NvidiaSTTService.InputParams

default:"None"

deprecated

Deprecated since 0.0.105, removed in 2.0.0. Use settings=NvidiaSTTService.Settings(...) instead.

NvidiaSTTService.Settings

default:"None"

Runtime-configurable settings. See Settings below.

bool

default:"True"

Whether to use SSL for the gRPC connection. Defaults to True for the NVIDIA cloud endpoint. Set to False for local deployments.

int

default:"1"

Number of audio channels.

int

default:"-1"

VAD start history in frames. Use -1 for Nemotron Speech default.

float

default:"-1.0"

VAD start threshold. Use -1.0 for Nemotron Speech default.

int

default:"320"

VAD stop history in frames. Use -1 for Nemotron Speech default.

float

default:"-1.0"

VAD stop threshold. Use -1.0 for Nemotron Speech default.

int

default:"-1"

End-of-utterance stop history in frames. Use -1 for Nemotron Speech default.

float

default:"-1.0"

End-of-utterance stop threshold. Use -1.0 for Nemotron Speech default.

str

default:"\"\""

Custom Nemotron Speech configuration string (e.g. "enable_vad_endpointing:true,neural_vad.onset:0.65").

float | None

default:"NVIDIA_TTFS_P99"

P99 latency from speech end to final transcript in seconds. Override for your deployment. See stt-benchmark.

Settings

Runtime-configurable settings passed via the settings constructor argument using NvidiaSTTService.Settings(...). These can be updated mid-conversation with STTUpdateSettingsFrame. See Service Settings for details.

Parameter	Type	Default	Description
`model`	`str`	`"nemotron-asr-streaming"`	STT model identifier, taken from `model_function_map`. (Inherited from base STT settings.)
`language`	`Language \| str`	`Language.EN_US`	Target language for transcription. (Inherited from base STT settings.)
`profanity_filter`	`bool`	`False`	Whether to filter profanity from results.
`automatic_punctuation`	`bool`	`True`	Whether to add automatic punctuation.
`verbatim_transcripts`	`bool`	`True`	Whether to return verbatim transcripts.
`boosted_lm_words`	`list[str]`	`None`	List of words to boost in the language model.
`boosted_lm_score`	`float`	`4.0`	Score boost for specified words.
`max_alternatives`	`int`	`1`	Maximum number of recognition alternatives.
`interim_results`	`bool`	`True`	Whether to return interim (partial) results.
`word_time_offsets`	`bool`	`False`	Whether to include word-level time offsets.
`speaker_diarization`	`bool`	`False`	Whether to enable speaker diarization.
`diarization_max_speakers`	`int`	`0`	Maximum number of speakers for diarization.

Usage

import os

from pipecat.services.nvidia.stt import NvidiaSTTService

stt = NvidiaSTTService(
    api_key=os.getenv("NVIDIA_API_KEY"),
)

Notes

Model cannot be changed after initialization: Use the model_function_map parameter in the constructor to specify the model and function ID.
Streaming: Provides real-time interim and final results through continuous audio streaming.
Automatic reconnection: The service automatically reconnects if the gRPC stream drops unexpectedly, maintaining continuous transcription.
Metrics support: This service supports metrics generation (can_generate_metrics() returns True).

NvidiaSegmentedSTTService

Batch/segmented transcription using NVIDIA Nemotron Speech’s Canary models. Processes complete audio segments after VAD detects speech boundaries.

str | None

default:"None"

NVIDIA API key for authentication. Required when using the cloud endpoint. Not needed for local deployments.

str

default:"grpc.nvcf.nvidia.com:443"

NVIDIA Nemotron Speech server address. For local deployments, pass the local address (e.g. localhost:50051).

Mapping[str, str]

Mapping containing function_id and model_name for the ASR model.

int

default:"None"

Audio sample rate in Hz. When None, uses the pipeline’s configured sample rate.

NvidiaSegmentedSTTService.InputParams

default:"None"

deprecated

Deprecated since 0.0.105, removed in 2.0.0. Use settings=NvidiaSegmentedSTTService.Settings(...) instead.

NvidiaSegmentedSTTService.Settings

default:"None"

Runtime-configurable settings. See Settings below.

bool

default:"True"

Whether to use SSL for the gRPC connection. Defaults to True for the NVIDIA cloud endpoint. Set to False for local deployments.

str

default:"\"\""

Custom Nemotron Speech configuration string (e.g. "enable_vad_endpointing:true,neural_vad.onset:0.65").

float | None

default:"NVIDIA_TTFS_P99"

P99 latency from speech end to final transcript in seconds. Override for your deployment. See stt-benchmark.

Settings

Runtime-configurable settings passed via the settings constructor argument using NvidiaSegmentedSTTService.Settings(...). These can be updated mid-conversation with STTUpdateSettingsFrame. See Service Settings for details.

Parameter	Type	Default	Description
`model`	`str`	`"canary-1b-asr"`	STT model identifier, taken from `model_function_map`. (Inherited from base STT settings.)
`language`	`Language \| str`	`Language.EN_US`	Target language for transcription. (Inherited from base STT settings.)
`profanity_filter`	`bool`	`False`	Whether to filter profanity from results.
`automatic_punctuation`	`bool`	`True`	Whether to add automatic punctuation.
`verbatim_transcripts`	`bool`	`False`	Whether to return verbatim transcripts.
`boosted_lm_words`	`list[str]`	`None`	List of words to boost in the language model.
`boosted_lm_score`	`float`	`4.0`	Score boost for specified words.
`max_alternatives`	`int`	`1`	Maximum number of recognition alternatives.
`word_time_offsets`	`bool`	`False`	Whether to include word-level time offsets.
`speaker_diarization`	`bool`	`False`	Whether to enable speaker diarization.
`diarization_max_speakers`	`int`	`0`	Maximum number of speakers for diarization.

Usage

from pipecat.services.nvidia.stt import NvidiaSegmentedSTTService
from pipecat.transcriptions.language import Language

stt = NvidiaSegmentedSTTService(
    api_key=os.getenv("NVIDIA_API_KEY"),
    settings=NvidiaSegmentedSTTService.Settings(
        language=Language.ES,
        automatic_punctuation=True,
        boosted_lm_words=["Pipecat", "NVIDIA"],
        boosted_lm_score=6.0,
    ),
)

Notes

Model cannot be changed after initialization: Use the model_function_map parameter in the constructor to specify the model and function ID.
Segmented processing: Processes complete audio segments for higher accuracy compared to streaming.
Language support: See the NVIDIA ASR NIM documentation for the languages each model supports.
Word boosting: Use boosted_lm_words and boosted_lm_score to improve recognition of domain-specific terms.

NvidiaSageMakerSTTService

Streaming speech recognition using NVIDIA Nemotron ASR via an AWS SageMaker bidirectional-stream endpoint. This service maintains a persistent HTTP/2 bidi-stream connection to a deployed SageMaker endpoint that proxies to NVIDIA NIM’s realtime WebSocket API.

str

required

Name of the deployed SageMaker endpoint.

str

default:"us-west-2"

AWS region where the endpoint is deployed.

int

default:"None"

Audio sample rate in Hz. When None, uses the pipeline’s configured sample rate.

NvidiaSageMakerSTTService.Settings

default:"None"

Runtime-configurable settings. See Settings below.

float | None

default:"1.5"

P99 latency from speech end to final transcript in seconds. Override for your deployment.

Settings

Runtime-configurable settings passed via the settings constructor argument using NvidiaSageMakerSTTService.Settings(...). Changes sent with STTUpdateSettingsFrame are stored but not applied to the open session. See Service Settings for details.

Parameter	Type	Default	Description
`model`	`str`	`cache-aware-parakeet-rnnt-en-US-asr-streaming-sortformer`	STT model identifier.
`language`	`Language \| str`	`en-US`	BCP-47 language code passed to NIM.

Usage

from pipecat.services.nvidia.sagemaker.stt import NvidiaSageMakerSTTService

stt = NvidiaSageMakerSTTService(
    endpoint_name=os.getenv("SAGEMAKER_ASR_ENDPOINT_NAME"),
    region=os.getenv("AWS_REGION", "us-west-2"),
    settings=NvidiaSageMakerSTTService.Settings(
        language="en-US",
    ),
)

Notes

AWS SageMaker deployment required: This service requires a deployed SageMaker endpoint running NVIDIA Nemotron ASR NIM. See the deployment example for setup instructions.
AWS credentials: Requires AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables for SageMaker authentication.
Environment variables: SAGEMAKER_ASR_ENDPOINT_NAME for the endpoint name.
Automatic reconnection: The service automatically reconnects on error to maintain continuous transcription.
Session start failures: If the SageMaker session fails to start, the service pushes an ErrorFrame categorized as RATE_LIMIT, SERVER, CONNECTIVITY, or INVALID_REQUEST and is marked unusable (is_usable is False).
NIM-driven finalization: Pushes interim transcripts as NIM returns them and a final transcript when NIM completes an utterance. VAD events don’t trigger finalization.
Metrics support: This service supports metrics generation (can_generate_metrics() returns True).

Get Started

Pipecat Server

Client SDKs

Pipecat Flows

Pipecat Cloud

CLI

Pipecat Context Hub

NVIDIA Nemotron Speech-to-Text

Overview

NVIDIA Nemotron Speech STT API Reference

Example Implementation

NVIDIA ASR NIM Documentation

NVIDIA Developer Portal

Installation

Prerequisites

NVIDIA Nemotron Speech Setup

Environment Variables

NvidiaSTTService

Settings

Usage

Notes

NvidiaSegmentedSTTService

Settings

Usage

Notes

NvidiaSageMakerSTTService

Settings

Usage

Notes

​Overview

NVIDIA Nemotron Speech STT API Reference

Example Implementation

NVIDIA ASR NIM Documentation

NVIDIA Developer Portal

​Installation

​Prerequisites

​NVIDIA Nemotron Speech Setup

​Environment Variables

​NvidiaSTTService

​Settings

​Usage

​Notes

​NvidiaSegmentedSTTService

​Settings

​Usage

​Notes

​NvidiaSageMakerSTTService

​Settings

​Usage

​Notes

Overview

Installation

Prerequisites

NVIDIA Nemotron Speech Setup

Environment Variables

NvidiaSTTService

Settings

Usage

Notes

NvidiaSegmentedSTTService

Settings

Usage

Notes

NvidiaSageMakerSTTService

Settings

Usage

Notes