NVIDIA Nemotron Omni LLM

Overview

NvidiaOmniLLMService runs NVIDIA Nemotron Omni models, which accept speech, images, and video as well as text and reply with text. It extends NvidiaLLMService, so streaming, function calling, reasoning frames, and metrics work as they do there. audio_input sets the pipeline shape:

audio_input=True (default): Omni hears the user directly and replaces the STT service. It transcribes and answers each spoken turn in one request, and pushes the transcript upstream as a TranscriptionFrame for the context. The pipeline still needs a TTS service.
audio_input=False: Omni is a text LLM in a cascaded pipeline, after an STT service.

NVIDIA Omni LLM API Reference

Pipecat’s API methods for NVIDIA Nemotron Omni integration

Example Implementation

Voice bot that hears the user with no STT service

NVIDIA NIM Documentation

Official NVIDIA NIM documentation and setup

NVIDIA Developer Portal

Access NIM services and manage API keys

Installation

uv add "pipecat-ai[nvidia]"

Prerequisites

NVIDIA NIM Setup

NVIDIA Developer Account (cloud endpoint only): Sign up at the NVIDIA Developer Portal
API Key (cloud endpoint only): Generate an NVIDIA API key
Local Deployment (optional): Serve a Nemotron Omni model and set base_url to its address

Required Environment Variables

NVIDIA_API_KEY: Your NVIDIA API key (required for the cloud endpoint)

Configuration

from pipecat.services.nvidia.omni.llm import NvidiaOmniLLMService

str | None

default:"None"

NVIDIA API key. Required for the cloud endpoint, not for local deployments.

str

default:"https://integrate.api.nvidia.com/v1"

OpenAI-compatible endpoint base URL. For local deployments, pass the local address (e.g., http://localhost:8000/v1).

bool

default:"True"

Whether Omni hears the user directly, with no STT service in the pipeline. False makes it a text LLM for a cascaded pipeline.

LLMContext | None

default:"None"

Context that audio turns are answered against until the first LLMContextFrame supplies one.

NvidiaOmniLLMService.Settings

default:"None"

Runtime-configurable settings. See Settings below.

float

default:"120.0"

HTTP request timeout, in seconds.

The constructor also accepts the other arguments of OpenAILLMService.

Settings

Runtime-configurable settings passed via the settings constructor argument using NvidiaOmniLLMService.Settings(...). These can be updated mid-conversation with LLMUpdateSettingsFrame. See Service Settings for details. The settings of OpenAILLMService apply, with the defaults model="nvidia/nemotron-3-nano-omni-30b-a3b-reasoning", temperature=0.6, and top_p=0.95. It adds:

Parameter	Type	Default	Description
`audio_response_instruction`	`str \| None`	`None`	Instruction sent with the user’s speech in audio input mode. `None` uses `DEFAULT_AUDIO_RESPONSE_INSTRUCTION`. A custom instruction should keep asking for `<transcript>` and `<response>` sections: a reply without them still reaches TTS, but the user’s turn is then missing from the context.
`min_user_audio_secs`	`float`	`0.3`	Shortest buffered utterance, in seconds, that starts a turn.
`pre_speech_buffer_secs`	`float`	`0.2`	Audio retained from before the user started speaking, in seconds, so the start of an utterance is not clipped.

Usage

Voice Bot with Audio Input

The pipeline has no STT service. Add a VAD analyzer to the user aggregator, so the service knows where each turn starts and ends:

import os
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.pipeline.pipeline import Pipeline
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.services.nvidia.omni.llm import NvidiaOmniLLMService

llm = NvidiaOmniLLMService(api_key=os.getenv("NVIDIA_API_KEY"))

context = LLMContext()
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()),
)

pipeline = Pipeline(
    [
        transport.input(),
        user_aggregator,
        llm,
        tts,
        transport.output(),
        assistant_aggregator,
    ]
)

The user’s text is only known once the transcript arrives, so read it from on_user_turn_message_added:

@user_aggregator.event_handler("on_user_turn_message_added")
async def on_user_turn_message_added(aggregator, message):
    print(f"User said: {message.content}")

Cascaded Pipeline

With audio_input=False, put an STT service before the user aggregator:

llm = NvidiaOmniLLMService(
    api_key=os.getenv("NVIDIA_API_KEY"),
    audio_input=False,
)

Audio and Video in the Context

Audio and video files added to the context, with LLMContext.create_file_message() or create_audio_message(), are sent in the form Omni reads. Images work in either mode.

Methods

run_multimodal_inference

async def run_multimodal_inference(
    context: LLMContext,
    *,
    max_tokens: int | None = None,
    reasoning_budget: int | None = None,
    temperature: float | None = None,
    stream: bool = False,
    on_text_delta: Callable[[str], Awaitable[None]] | None = None,
    on_reasoning_delta: Callable[[str], Awaitable[None]] | None = None,
) -> NvidiaOmniInferenceResult

Runs a one-off request outside the conversation, for example to describe a video clip, and returns an NvidiaOmniInferenceResult with text, reasoning, and finish_reason (such as "stop" or "length"). Pass media in context.

Parameter	Type	Description
`context`	`LLMContext`	The context to run, including any media.
`max_tokens`	`int \| None`	Overrides the configured token limit.
`reasoning_budget`	`int \| None`	NVIDIA `reasoning_budget` request field: the most tokens the model may spend reasoning.
`temperature`	`float \| None`	Overrides the configured temperature.
`stream`	`bool`	Whether to stream the completion and call the delta callbacks.
`on_text_delta`	`Callable[[str], Awaitable[None]] \| None`	Called with each response text delta while streaming.
`on_reasoning_delta`	`Callable[[str], Awaitable[None]] \| None`	Called with each reasoning delta while streaming.

Notes

Realtime mode: With audio_input=True, the service announces itself as a realtime service, which switches the user aggregator into realtime mode. It answers each spoken turn after any turn still running.
Interruptions: If the user interrupts a turn before Omni has transcribed it, the speech is answered together with the user’s next utterance.
Context frames: Context frames that need an answer, such as tool results and typed user input, start a completion in both modes, so an LLMRunFrame greeting works either way.
Reasoning content: Reasoning is emitted as LLMThought*Frame objects and is not sent to TTS, as for NvidiaLLMService.

Get Started

Pipecat Server

Client SDKs

Pipecat Flows

Pipecat Cloud

CLI

Pipecat Context Hub

NVIDIA Nemotron Omni LLM

Overview

NVIDIA Omni LLM API Reference

Example Implementation

NVIDIA NIM Documentation

NVIDIA Developer Portal

Installation

Prerequisites

NVIDIA NIM Setup

Required Environment Variables

Configuration

Settings

Usage

Voice Bot with Audio Input

Cascaded Pipeline

Audio and Video in the Context

Methods

run_multimodal_inference

Notes

​Overview

NVIDIA Omni LLM API Reference

Example Implementation

NVIDIA NIM Documentation

NVIDIA Developer Portal

​Installation

​Prerequisites

​NVIDIA NIM Setup

​Required Environment Variables

​Configuration

​Settings

​Usage

​Voice Bot with Audio Input

​Cascaded Pipeline

​Audio and Video in the Context

​Methods

​run_multimodal_inference

​Notes

Overview

Installation

Prerequisites

NVIDIA NIM Setup

Required Environment Variables

Configuration

Settings

Usage

Voice Bot with Audio Input

Cascaded Pipeline

Audio and Video in the Context

Methods

run_multimodal_inference

Notes