Scenario Configuration

Scripted and simulated scenarios share the same top-level configuration. The user: block sets how the user’s side reaches the agent. The judge: block sets what the judge hears and which LLM it is. factory: and !include work the same way in both. This page covers those shared blocks. The fields that belong to one kind alone are on the scripted and simulated pages. Everything here is optional. A scenario with no configuration blocks runs entirely in text mode with the default judge, which is the fastest way to start.

Text and audio modes

Two top-level blocks control a scenario’s modalities, and each has its own modality: field:

user: sets how the user’s turns are delivered to the agent: sent as text, bypassing its STT (modality: text), or synthesized into real speech (modality: audio). In a scripted scenario the turns are the ones you wrote; in a simulation they are the persona’s replies.
judge: sets what the judge evaluates: the agent’s LLM text, with its TTS skipped (modality: text), or a transcription of its actual spoken audio (modality: audio).

When modality: isn’t specified, or a block is omitted entirely, it defaults to text. The two sides are also independent: you can drive the agent with text while judging its real speech, or speak to it and judge the LLM text. A scenario with neither block runs entirely in text mode. No audio flows on either side, so this is the fastest and cheapest way to test prompts, conversational logic, and function calling: no audio service cost, and a multi-turn scenario finishes in seconds. The only services the harness itself needs are the judge LLM (Ollama with gemma4:12b by default) and, for a simulation, the persona LLM.

The top-level user: block only configures delivery. In a scripted scenario, each turn’s own user: field (see User turns) is the utterance itself, and is written the same way in both modes.

User delivery with `user:`

Text (the default). Each user turn is sent to the agent as text, bypassing its STT. This needs no configuration; it’s equivalent to:

user:
  modality: text

Audio. Each user turn is synthesized by a TTS the harness runs and streamed into your agent’s pipeline at real-time cadence, exercising its VAD, turn detection, and STT exactly as a live microphone would. Synthesized audio is cached across runs, so repeated turns don’t re-synthesize. The speech: block (the TTS service and voice) is required, unless every spoken turn of a scripted scenario names an audio: file to play instead:

user:
  modality: audio
  speech:
    service: kokoro # local TTS, no API key, no per-run cost
    voice: af_heart
    sample_rate: 16000

The built-in speech service is kokoro, a local model. For a cloud voice, name a factory:. service: cartesia is deprecated since 1.9.0 in favor of a factory and will be removed in 2.0.0.

Both the speech: and judge.transcription: blocks take an optional language, so a non-English agent can be driven and judged in the language it speaks:

user:
  modality: audio
  speech:
    service: kokoro
    voice: zf_xiaobei # voices are language-specific, so match the language
    language: zh

It accepts a code like zh or a Language, and an unrecognized code raises a ValueError naming it. Omitting it leaves the service’s own default, which is English. Voices are not selected for you: af_heart speaks US English whatever language says.

Prefer whisper over moonshine for transcribing a non-English agent. Moonshine’s non-English models return empty or truncated transcripts, and an empty transcript is indistinguishable from a bot that said nothing.

Judging with `judge:`

Text (the default). The agent’s TTS is skipped automatically, including any on-connect greeting, and the judge evaluates the LLM’s text output. Fast and silent; equivalent to:

judge:
  modality: text

Audio. The agent speaks for real. The harness captures its synthesized audio, transcribes it with the configured STT, and the response event becomes that transcription, so the judge evaluates what a user would actually have heard. This is the true end-to-end check: STT in, LLM in the middle, TTS out. The transcription: block is required:

judge:
  modality: audio
  transcription:
    service: moonshine # STT for the agent's audio (or: whisper)
    model: small-streaming
  eval:
    service: ollama # the judge LLM
    model: gemma4:12b

The built-in transcribers are moonshine and whisper, both local models. When transcription.service: is omitted, it defaults to moonshine.

In either modality, the judge.eval: block selects the judge LLM: ollama (the default, gemma4:12b, with an optional endpoint: for a remote Ollama), or any other OpenAI-compatible LLM through a factory:. service: openai is deprecated since 1.9.0 in favor of a factory and will be removed in 2.0.0. This is the LLM that decides a scripted scenario’s eval: assertions, and a simulation’s success: and judged metrics. An extra: mapping is forwarded to the judge as top-level request parameters, which is how provider-specific options reach it. The default judge is thinking-capable and only its JSON verdict is ever read, so the default extra: turns reasoning off. It buys nothing, costs latency, and eats into the token budget the verdict needs.

Naming a model: without an extra: drops the defaults rather than merging with them, since they are chosen for the default model. To pin an older judge, set only what it needs:

judge:
  eval:
    service: ollama
    model: gemma2:9b # no extra:, since gemma2 isn't thinking-capable

A simulation’s simulator: block, the LLM that plays the caller, takes the same service, model, endpoint, extra, and factory keys as judge.eval:, and defaults to the same local model. See The persona LLM.

Custom services with `factory:`

The built-in service: names are the local, keyless services: Ollama for the judge and the persona, Kokoro for the user’s voice, Moonshine and Whisper for the agent’s audio. Any other provider is a factory:, a dotted path to a callable that takes the block’s mapping and returns the service. The scenario stays a description of the test, while the provider, its settings, and its key live in code. Any extra keys you put in the block are passed through to your factory:

judge:
  modality: audio
  eval:
    factory: my_evals.judge
    model: gpt-4o-mini # available to your factory as config["model"]
  transcription:
    factory: my_evals.transcriber

user:
  modality: audio
  speech:
    factory: my_evals.voice
    voice: luna

my_evals.py

import os

from pipecat.services.fal.stt import FalSTTService
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.services.rime.tts import RimeHttpTTSService


def judge(config):
    return OpenAILLMService(
        api_key=os.environ["OPENAI_API_KEY"],
        settings=OpenAILLMService.Settings(model=config["model"]),
    )


def transcriber(config):
    return FalSTTService(api_key=os.environ["FAL_KEY"])


def voice(config):
    return RimeHttpTTSService(
        api_key=os.environ["RIME_API_KEY"],
        settings=RimeHttpTTSService.Settings(voice=config["voice"]),
    )

The pipeline sets each service’s sample rate, so a factory doesn’t. What a factory must return depends on the block:

judge.eval: and simulator:: an OpenAI-compatible LLM service, and for the persona one that supports function calling.
user.speech:: a local or HTTP TTS. The cache around it drives synthesis directly, so a WebSocket-streaming TTS yields nothing there.
judge.transcription:: any pipeline STT.

pipecat eval loads the nearest .env before it runs, so a factory can read its API key from the environment the way your agent does. For a fully custom setup (your own caching, a pre-built service instance), pass the services to a session as judge=, persona_llm=, user_tts=, and bot_stt= through the library. Any value can be pulled from another file with !include, resolved relative to the scenario file. This keeps per-scenario noise down when a whole directory of scenarios shares the same audio setup, or the same simulator: block:

name: capital_question

user: !include user_audio.yaml
judge: !include judge_audio.yaml

turns:
  - user: "What is the capital of Germany?"
    expect:
      - event: response
        eval: "the response says the capital of Germany is Berlin"

A scenario file has a name:; an included fragment has none, which is how a directory run tells the two apart.

Running scenarios back to back

By default the bot keeps running between scenarios. When a scenario ends its eval connection closes, but the eval transport suppresses the bot’s on_client_disconnected handler, so the pipeline stays up to serve the next scenario. This is what lets pipecat eval run a.yaml b.yaml c.yaml drive a whole list, scripted scenarios and simulations mixed, against one bot instance with no reboot between them, which keeps a run fast. The trade-off is that anything the bot accumulated in one scenario is still there for the next. For results to be independent, each scenario has to start from a clean slate, and clearing that state is split between the harness and your bot:

Conversation context: a scripted scenario can seed or clear it with context:, which replaces the bot’s LLM context with the messages you provide. Without it, and always for a simulation, the previous conversation carries forward, which is rarely what you want across independent scenarios.
Application state: counters, flags, cached data, anything your bot holds outside the LLM context. The harness can’t see this, so resetting it is your bot’s job. A common place is the bot’s connect handler, which runs again for each scenario’s connection.

A suite avoids the problem by starting a fresh bot for every run. That is also why a simulation’s runs: is honored by the suite and not by pipecat eval run.

Exercising the disconnect path

Some bots do meaningful work in on_client_disconnected, like a goodbye message, session teardown, or resource cleanup. Because the eval transport suppresses that handler by default, set trigger_disconnect: true on a scenario of either kind to fire it when that scenario ends:

name: test_goodbye_on_disconnect
trigger_disconnect: true

turns:
  - user: "Thanks for your help!"
    expect:
      - event: response
        eval: "the agent acknowledges the thanks"
  # on_client_disconnected fires after this turn, so the agent can
  # send a goodbye message or clean up resources.

Bots often cancel their pipeline in on_client_disconnected, so a scenario with trigger_disconnect: true usually ends the bot process. Treat it as a terminal run, last in a list.

Enable it for every scenario in a run with pipecat eval run --trigger-disconnect; a scenario’s own trigger_disconnect field still takes precedence. This is independent of --stop-bot, which tears the bot down via an eval-cancel message regardless of the disconnect handler.

Next steps

Scripted Scenarios

Turns, events, and assertions, plus the scripted-only context: and stop_on_failure: fields.

Simulated Scenarios

A persona and a goal, the simulator: block, a success criterion, and metrics.

Get Started

Migration

Learning Pipecat

Fundamentals

Flows

Evals

Features

Telephony

Deployment

Examples & Recipes

Scenario Configuration

Text and audio modes

User delivery with `user:`

Judging with `judge:`

Custom services with `factory:`

Running scenarios back to back

Exercising the disconnect path

Next steps

Scripted Scenarios

Simulated Scenarios

​Text and audio modes

​User delivery with user:

​Judging with judge:

​Custom services with factory:

​Sharing config with !include

​Running scenarios back to back

​Exercising the disconnect path

​Next steps

Scripted Scenarios

Simulated Scenarios

Text and audio modes

User delivery with `user:`

Judging with `judge:`

Custom services with `factory:`

Sharing config with `!include`

Running scenarios back to back

Exercising the disconnect path

Next steps