> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Scenario Configuration
> The user: and judge: blocks shared by scripted and simulated scenarios: text and audio modes, TTS and STT services, the judge LLM, factory:, and !include.
Scripted and simulated scenarios share the same top-level configuration. The `user:` block sets how the user's side reaches the agent. The `judge:` block sets what the judge hears and which LLM it is. `factory:` and `!include` work the same way in both. This page covers those shared blocks. The fields that belong to one kind alone are on the [scripted](/pipecat/evals/scripted-scenarios#configuration) and [simulated](/pipecat/evals/simulated-scenarios#anatomy-of-a-simulation) pages.
Everything here is optional. A scenario with no configuration blocks runs entirely in [text mode](#text-and-audio-modes) with the default judge, which is the fastest way to start.
## Text and audio modes
Two top-level blocks control a scenario's modalities, and each has its own `modality:` field:
* `user:` sets how the user's turns are delivered to the agent: sent as text, bypassing its STT (`modality: text`), or synthesized into real speech (`modality: audio`). In a scripted scenario the turns are the ones you wrote; in a simulation they are the persona's replies.
* `judge:` sets what the judge evaluates: the agent's LLM text, with its TTS skipped (`modality: text`), or a transcription of its actual spoken audio (`modality: audio`).
When `modality:` isn't specified, or a block is omitted entirely, it defaults to `text`. The two sides are also independent: you can drive the agent with text while judging its real speech, or speak to it and judge the LLM text.
A scenario with neither block runs entirely in text mode. No audio flows on either side, so this is the fastest and cheapest way to test prompts, conversational logic, and function calling: no audio service cost, and a multi-turn scenario finishes in seconds. The only services the harness itself needs are the judge LLM (Ollama with `gemma4:12b` by default) and, for a simulation, the persona LLM.
The top-level `user:` block only configures delivery. In a scripted scenario,
each turn's own `user:` field (see [User
turns](/pipecat/evals/scripted-scenarios#user-turns)) is the utterance itself,
and is written the same way in both modes.
## User delivery with `user:`
**Text (the default).** Each user turn is sent to the agent as text, bypassing its STT. This needs no configuration; it's equivalent to:
```yaml theme={null}
user:
modality: text
```
**Audio.** Each user turn is synthesized by a TTS the harness runs and streamed into your agent's pipeline at real-time cadence, exercising its VAD, turn detection, and STT exactly as a live microphone would. Synthesized audio is cached across runs, so repeated turns don't re-synthesize. The `speech:` block (the TTS service and voice) is required, unless every spoken turn of a scripted scenario names an [`audio:` file](/pipecat/evals/scripted-scenarios#playing-audio-files-with-audio) to play instead:
```yaml theme={null}
user:
modality: audio
speech:
service: kokoro # local TTS, no API key, no per-run cost
voice: af_heart
sample_rate: 16000
```
The built-in speech service is `kokoro`, a local model. For a cloud voice,
name a [`factory:`](#custom-services-with-factory). `service: cartesia` is
deprecated since 1.9.0 in favor of a factory and will be removed in 2.0.0.
Both the `speech:` and `judge.transcription:` blocks take an optional `language`, so a non-English agent can be driven and judged in the language it speaks:
```yaml theme={null}
user:
modality: audio
speech:
service: kokoro
voice: zf_xiaobei # voices are language-specific, so match the language
language: zh
```
It accepts a code like `zh` or a `Language`, and an unrecognized code raises a `ValueError` naming it. Omitting it leaves the service's own default, which is English. Voices are not selected for you: `af_heart` speaks US English whatever `language` says.
Prefer `whisper` over `moonshine` for transcribing a non-English agent.
Moonshine's non-English models return empty or truncated transcripts, and an
empty transcript is indistinguishable from a bot that said nothing.
## Judging with `judge:`
**Text (the default).** The agent's TTS is skipped automatically, including any on-connect greeting, and the judge evaluates the LLM's text output. Fast and silent; equivalent to:
```yaml theme={null}
judge:
modality: text
```
**Audio.** The agent speaks for real. The harness captures its synthesized audio, transcribes it with the configured STT, and the `response` event becomes that transcription, so the judge evaluates what a user would actually have heard. This is the true end-to-end check: STT in, LLM in the middle, TTS out. The `transcription:` block is required:
```yaml theme={null}
judge:
modality: audio
transcription:
service: moonshine # STT for the agent's audio (or: whisper)
model: small-streaming
eval:
service: ollama # the judge LLM
model: gemma4:12b
```
The built-in transcribers are `moonshine` and `whisper`, both local models.
When `transcription.service:` is omitted, it defaults to `moonshine`.
In either modality, the `judge.eval:` block selects the judge LLM: `ollama` (the default, `gemma4:12b`, with an optional `endpoint:` for a remote Ollama), or any other OpenAI-compatible LLM through a [`factory:`](#custom-services-with-factory). `service: openai` is deprecated since 1.9.0 in favor of a factory and will be removed in 2.0.0. This is the LLM that decides a scripted scenario's [`eval:`](/pipecat/evals/scripted-scenarios#semantic-judging-with-eval) assertions, and a simulation's [`success:`](/pipecat/evals/simulated-scenarios#judging-the-outcome) and judged metrics.
An `extra:` mapping is forwarded to the judge as top-level request parameters, which is how provider-specific options reach it. The default judge is thinking-capable and only its JSON verdict is ever read, so the default `extra:` turns reasoning off. It buys nothing, costs latency, and eats into the token budget the verdict needs.
Naming a `model:` without an `extra:` drops the defaults rather than merging
with them, since they are chosen for the default model. To pin an older judge,
set only what it needs:
```yaml theme={null}
judge:
eval:
service: ollama
model: gemma2:9b # no extra:, since gemma2 isn't thinking-capable
```
A simulation's `simulator:` block, the LLM that plays the caller, takes the same `service`, `model`, `endpoint`, `extra`, and `factory` keys as `judge.eval:`, and defaults to the same local model. See [The persona LLM](/pipecat/evals/simulated-scenarios#the-persona-llm-with-simulator).
## Custom services with `factory:`
The built-in `service:` names are the local, keyless services: Ollama for the judge and the persona, Kokoro for the user's voice, Moonshine and Whisper for the agent's audio. Any other provider is a `factory:`, a dotted path to a callable that takes the block's mapping and returns the service. The scenario stays a description of the test, while the provider, its settings, and its key live in code. Any extra keys you put in the block are passed through to your factory:
```yaml theme={null}
judge:
modality: audio
eval:
factory: my_evals.judge
model: gpt-4o-mini # available to your factory as config["model"]
transcription:
factory: my_evals.transcriber
user:
modality: audio
speech:
factory: my_evals.voice
voice: luna
```
```python my_evals.py theme={null}
import os
from pipecat.services.fal.stt import FalSTTService
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.services.rime.tts import RimeHttpTTSService
def judge(config):
return OpenAILLMService(
api_key=os.environ["OPENAI_API_KEY"],
settings=OpenAILLMService.Settings(model=config["model"]),
)
def transcriber(config):
return FalSTTService(api_key=os.environ["FAL_KEY"])
def voice(config):
return RimeHttpTTSService(
api_key=os.environ["RIME_API_KEY"],
settings=RimeHttpTTSService.Settings(voice=config["voice"]),
)
```
The pipeline sets each service's sample rate, so a factory doesn't. What a factory must return depends on the block:
* `judge.eval:` and `simulator:`: an OpenAI-compatible LLM service, and for the persona one that supports function calling.
* `user.speech:`: a local or HTTP TTS. The cache around it drives synthesis directly, so a WebSocket-streaming TTS yields nothing there.
* `judge.transcription:`: any pipeline STT.
`pipecat eval` loads the nearest `.env` before it runs, so a factory can read its API key from the environment the way your agent does.
For a fully custom setup (your own caching, a pre-built service instance), pass the services to a session as `judge=`, `persona_llm=`, `user_tts=`, and `bot_stt=` through the [library](/pipecat/evals/library#customizing-the-judge-and-the-persona).
## Sharing config with `!include`
Any value can be pulled from another file with `!include`, resolved relative to the scenario file. This keeps per-scenario noise down when a whole directory of scenarios shares the same audio setup, or the same `simulator:` block:
```yaml theme={null}
name: capital_question
user: !include user_audio.yaml
judge: !include judge_audio.yaml
turns:
- user: "What is the capital of Germany?"
expect:
- event: response
eval: "the response says the capital of Germany is Berlin"
```
A scenario file has a `name:`; an included fragment has none, which is how a directory run tells the two apart.
## Running scenarios back to back
By default the bot keeps running between scenarios. When a scenario ends its eval connection closes, but the eval transport suppresses the bot's `on_client_disconnected` handler, so the pipeline stays up to serve the next scenario. This is what lets `pipecat eval run a.yaml b.yaml c.yaml` drive a whole list, scripted scenarios and simulations mixed, against one bot instance with no reboot between them, which keeps a run fast.
The trade-off is that anything the bot accumulated in one scenario is still there for the next. For results to be independent, each scenario has to start from a clean slate, and clearing that state is split between the harness and your bot:
* **Conversation context**: a scripted scenario can seed or clear it with [`context:`](/pipecat/evals/scripted-scenarios#seeding-the-context-with-context), which replaces the bot's LLM context with the messages you provide. Without it, and always for a simulation, the previous conversation carries forward, which is rarely what you want across independent scenarios.
* **Application state**: counters, flags, cached data, anything your bot holds outside the LLM context. The harness can't see this, so resetting it is your bot's job. A common place is the bot's connect handler, which runs again for each scenario's connection.
A [suite](/pipecat/evals/suites) avoids the problem by starting a fresh bot for every run. That is also why a simulation's `runs:` is honored by the suite and not by `pipecat eval run`.
## Exercising the disconnect path
Some bots do meaningful work in `on_client_disconnected`, like a goodbye message, session teardown, or resource cleanup. Because the eval transport suppresses that handler by default, set `trigger_disconnect: true` on a scenario of either kind to fire it when that scenario ends:
```yaml theme={null}
name: test_goodbye_on_disconnect
trigger_disconnect: true
turns:
- user: "Thanks for your help!"
expect:
- event: response
eval: "the agent acknowledges the thanks"
# on_client_disconnected fires after this turn, so the agent can
# send a goodbye message or clean up resources.
```
Bots often cancel their pipeline in `on_client_disconnected`, so a scenario with `trigger_disconnect: true` usually ends the bot process. Treat it as a terminal run, last in a list.
Enable it for every scenario in a run with `pipecat eval run --trigger-disconnect`; a scenario's own `trigger_disconnect` field still takes
precedence. This is independent of `--stop-bot`, which tears the bot down via
an `eval-cancel` message regardless of the disconnect handler.
## Next steps
Turns, events, and assertions, plus the scripted-only `context:` and
`stop_on_failure:` fields.
A persona and a goal, the `simulator:` block, a success criterion, and
metrics.