> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# NVIDIA Nemotron Omni LLM

> NvidiaOmniLLMService runs NVIDIA Nemotron Omni models that take speech, images, and video as input, replacing the STT service.

## Overview

`NvidiaOmniLLMService` runs NVIDIA Nemotron Omni models, which accept speech, images, and video as well as text and reply with text. It extends [`NvidiaLLMService`](https://docs.pipecat.ai/api-reference/server/services/llm/nvidia), so streaming, function calling, reasoning frames, and metrics work as they do there.

`audio_input` sets the pipeline shape:

* **`audio_input=True` (default)**: Omni hears the user directly and replaces the STT service. It transcribes and answers each spoken turn in one request, and pushes the transcript upstream as a `TranscriptionFrame` for the context. The pipeline still needs a TTS service.
* **`audio_input=False`**: Omni is a text LLM in a cascaded pipeline, after an STT service.

> **[NVIDIA Omni LLM API Reference](https://reference-server.pipecat.ai/en/latest/api/pipecat.services.nvidia.omni.llm.html)**
>
> Pipecat's API methods for NVIDIA Nemotron Omni integration

> **[Example Implementation](https://github.com/pipecat-ai/pipecat/blob/main/examples/voice/voice-nvidia-omni.py)**
>
> Voice bot that hears the user with no STT service

> **[NVIDIA NIM Documentation](https://docs.nvidia.com/nim/)**
>
> Official NVIDIA NIM documentation and setup

> **[NVIDIA Developer Portal](https://developer.nvidia.com/)**
>
> Access NIM services and manage API keys

## Installation

```bash theme={null}
uv add "pipecat-ai[nvidia]"
```

## Prerequisites

### NVIDIA NIM Setup

1. **NVIDIA Developer Account** (cloud endpoint only): Sign up at the [NVIDIA Developer Portal](https://developer.nvidia.com/)
2. **API Key** (cloud endpoint only): Generate an NVIDIA API key
3. **Local Deployment** (optional): Serve a Nemotron Omni model and set `base_url` to its address

### Required Environment Variables

* `NVIDIA_API_KEY`: Your NVIDIA API key (required for the cloud endpoint)

## Configuration

```python theme={null}
from pipecat.services.nvidia.omni.llm import NvidiaOmniLLMService
```

**api\_key** `str | None` (default: None)

NVIDIA API key. Required for the cloud endpoint, not for local deployments.

**base\_url** `str` (default: [https://integrate.api.nvidia.com/v1](https://integrate.api.nvidia.com/v1))

OpenAI-compatible endpoint base URL. For local deployments, pass the local
address (e.g., `http://localhost:8000/v1`).

**audio\_input** `bool` (default: True)

Whether Omni hears the user directly, with no STT service in the pipeline.
`False` makes it a text LLM for a cascaded pipeline.

**context** `LLMContext | None` (default: None)

Context that audio turns are answered against until the first
`LLMContextFrame` supplies one.

**settings** `NvidiaOmniLLMService.Settings` (default: None)

Runtime-configurable settings. See [Settings](#settings) below.

**request\_timeout\_secs** `float` (default: 120.0)

HTTP request timeout, in seconds.

The constructor also accepts the other arguments of
[`OpenAILLMService`](https://docs.pipecat.ai/api-reference/server/services/llm/openai#configuration).

### Settings

Runtime-configurable settings passed via the `settings` constructor argument using `NvidiaOmniLLMService.Settings(...)`. These can be updated mid-conversation with `LLMUpdateSettingsFrame`. See [Service Settings](https://docs.pipecat.ai/pipecat/fundamentals/service-settings.md) for details.

The settings of [`OpenAILLMService`](https://docs.pipecat.ai/api-reference/server/services/llm/openai#settings) apply, with the defaults `model="nvidia/nemotron-3-nano-omni-30b-a3b-reasoning"`, `temperature=0.6`, and `top_p=0.95`. It adds:

| Parameter | Type | Default | Description |
| - | - | - | - |
| `audio_response_instruction` | `str \| None` | `None` | Instruction sent with the user's speech in audio input mode. `None` uses `DEFAULT_AUDIO_RESPONSE_INSTRUCTION`. A custom instruction should keep asking for `<transcript>` and `<response>` sections: a reply without them still reaches TTS, but the user's turn is then missing from the context. |
| `min_user_audio_secs` | `float` | `0.3` | Shortest buffered utterance, in seconds, that starts a turn. |
| `pre_speech_buffer_secs` | `float` | `0.2` | Audio retained from before the user started speaking, in seconds, so the start of an utterance is not clipped. |

## Usage

### Voice Bot with Audio Input

The pipeline has no STT service. Add a VAD analyzer to the user aggregator, so the service knows where each turn starts and ends:

```python theme={null}
import os
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.pipeline.pipeline import Pipeline
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.services.nvidia.omni.llm import NvidiaOmniLLMService

llm = NvidiaOmniLLMService(api_key=os.getenv("NVIDIA_API_KEY"))

context = LLMContext()
user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
    context,
    user_params=LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer()),
)

pipeline = Pipeline(
    [
        transport.input(),
        user_aggregator,
        llm,
        tts,
        transport.output(),
        assistant_aggregator,
    ]
)
```

The user's text is only known once the transcript arrives, so read it from `on_user_turn_message_added`:

```python theme={null}
@user_aggregator.event_handler("on_user_turn_message_added")
async def on_user_turn_message_added(aggregator, message):
    print(f"User said: {message.content}")
```

### Cascaded Pipeline

With `audio_input=False`, put an STT service before the user aggregator:

```python theme={null}
llm = NvidiaOmniLLMService(
    api_key=os.getenv("NVIDIA_API_KEY"),
    audio_input=False,
)
```

### Audio and Video in the Context

Audio and video files added to the context, with `LLMContext.create_file_message()` or `create_audio_message()`, are sent in the form Omni reads. Images work in either mode.

## Methods

### run\_multimodal\_inference

```python theme={null}
async def run_multimodal_inference(
    context: LLMContext,
    *,
    max_tokens: int | None = None,
    reasoning_budget: int | None = None,
    temperature: float | None = None,
    stream: bool = False,
    on_text_delta: Callable[[str], Awaitable[None]] | None = None,
    on_reasoning_delta: Callable[[str], Awaitable[None]] | None = None,
) -> NvidiaOmniInferenceResult
```

Runs a one-off request outside the conversation, for example to describe a video clip, and returns an `NvidiaOmniInferenceResult` with `text`, `reasoning`, and `finish_reason` (such as `"stop"` or `"length"`). Pass media in `context`.

| Parameter | Type | Description |
| - | - | - |
| `context` | `LLMContext` | The context to run, including any media. |
| `max_tokens` | `int \| None` | Overrides the configured token limit. |
| `reasoning_budget` | `int \| None` | NVIDIA `reasoning_budget` request field: the most tokens the model may spend reasoning. |
| `temperature` | `float \| None` | Overrides the configured temperature. |
| `stream` | `bool` | Whether to stream the completion and call the delta callbacks. |
| `on_text_delta` | `Callable[[str], Awaitable[None]] \| None` | Called with each response text delta while streaming. |
| `on_reasoning_delta` | `Callable[[str], Awaitable[None]] \| None` | Called with each reasoning delta while streaming. |

## Notes

* **Realtime mode**: With `audio_input=True`, the service announces itself as a realtime service, which switches the user aggregator into realtime mode. It answers each spoken turn after any turn still running.
* **Interruptions**: If the user interrupts a turn before Omni has transcribed it, the speech is answered together with the user's next utterance.
* **Context frames**: Context frames that need an answer, such as tool results and typed user input, start a completion in both modes, so an `LLMRunFrame` greeting works either way.
* **Reasoning content**: Reasoning is emitted as `LLMThought*Frame` objects and is not sent to TTS, as for [`NvidiaLLMService`](https://docs.pipecat.ai/api-reference/server/services/llm/nvidia#notes).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.