Scripted Scenarios

A scripted scenario is a YAML file that writes the user’s turns out, each with the events you expect your agent to emit in response. A file with a top-level turns: key is scripted; one with a persona: key is a simulated scenario, where an LLM plays the user instead. This page covers the full scripted format. If you haven’t run a scenario yet, start with the quickstart. Use a scripted scenario when you want exact control over the user’s side. You know what the user says on every turn, so you can assert exactly what the agent must do: call this tool with these arguments, say this, answer within this budget, recover from this interruption. The input is the same every run, so a failure is easy to reproduce and fix. Use a simulation when you want to check a goal instead, and let the caller adapt to the agent.

Anatomy of a scenario

name: multi_turn # required: the eval's name

judge: # optional: judge modality and LLM (defaults shown below)
  eval:
    service: ollama
    model: gemma4:12b
    extra: { reasoning_effort: none }

turns: # required: the conversation, in order
  - user: "My name is Alex, and I'm planning a trip to Italy."
    expect:
      - event: response
        eval: "acknowledges the user's message (the name Alex and/or the trip to Italy)"

  - user: "Remind me, what's my name and where am I going?"
    expect:
      - event: response
        eval: "recalls that the user's name is Alex and the destination is Italy"

Each turn optionally sends a user utterance (user:) and lists the events expected in response (expect:). Expected events must arrive in the order listed, but the agent may emit other events in between, so you don’t have to enumerate everything it does. The rest of this page is in four parts:

Configuration

The shared user: and judge: blocks, plus the scripted-only context: and stop_on_failure:.

User turns

Drive each turn with an utterance, keypresses, an image, or timing.

Events

The semantic events the agent emits, and what each one means.

Assertions

Check an event’s content or timing with eval:, text_contains:, and more.

Configuration

The user: and judge: blocks, factory:, !include, running scenarios back to back, and the disconnect path are shared with simulated scenarios and documented once, on Scenario Configuration. A scenario with none of those blocks runs entirely in text mode with the default judge, which is the fastest way to start. Two fields are scripted-only, since a simulation has no scripted turns to seed or to score:

Seeding the context with `context:`

By default the harness leaves the bot’s LLM context alone: whatever the bot sets up for itself (for example, a system prompt added in its connect handler) is what the scenario runs against. Provide context: to replace that with messages of your own, which lets a scenario start mid-conversation:

context:
  - role: developer
    content: "The user has already introduced themselves as Alex."
  - role: assistant
    content: "Nice to meet you, Alex! How can I help?"

The harness sends these right after the bot-ready handshake as an LLMMessagesUpdateFrame that replaces the bot’s context wholesale. Omit context: and the harness sends nothing, leaving the bot’s own context in place.

Scoring every turn

By default the first turn with a failed assertion ends the scenario, since a conversation that has gone wrong rarely tells you much about the turns after it. Set stop_on_failure: false when the turns are independent and you want a score across all of them, for example when benchmarking intent classification over a list of utterances:

name: intent_benchmark
stop_on_failure: false

turns:
  - user: "Book me a flight to Tokyo."
    expect:
      - event: function_call
        within_ms: 15000
        calls: [{ name: book_flight }]

Give those turns an explicit within_ms. With the 60-second default, a bot that has stopped answering costs a full budget on every remaining turn.

This governs progression from one turn to the next. Within a turn, an expectation that times out still ends that turn’s matching, because a turn’s expectations share one deadline anchored at the moment its input was sent.

User turns

Each turn drives the agent by speaking (a user: utterance) or pressing keys (a dtmf: sequence); the two are mutually exclusive. A turn can also register an image:, or be observation-only with no input. send_after: controls when the input is sent.

Utterances with `user:`

Each turn’s user: field is the user’s utterance for that turn, a plain string. You write it the same way in both modes; whether it’s delivered as text or synthesized into real speech is set once by the user: block, not per turn. A turn without a user: field is observation-only: the harness just waits for the expected events. This is how you test agent-first behavior like an on-connect greeting:

turns:
  # No user input: just wait for the agent to speak first.
  - expect:
      - event: response
        eval: "the bot opens the conversation with a greeting or an offer to help"

Playing audio files with `audio:`

In audio mode, a turn can play a recording instead of synthesizing its user: text. The audio: field (a path relative to the scenario file) names the audio file to stream to the agent:

turns:
  - user: "What is the capital of Germany?"
    audio: ../assets/capital_question.wav
    expect:
      - event: user_transcription
        text_contains: "capital of Germany"
      - event: response
        eval: "the response says the capital of Germany is Berlin"

The file is sent at its own sample rate (it need not match the agent’s input rate), and any format soundfile reads works: WAV, MP3, FLAC, OGG. Multi-channel audio is downmixed to mono. user: is required alongside audio: and gives what the recording says, since the judge and text_contains see it as the turn’s input. A scenario whose spoken turns all name an audio: file needs no user.speech: block, since nothing is synthesized.

DTMF keypresses with `dtmf:`

Instead of a user: utterance, a turn can press phone keypad keys with dtmf:. The two are mutually exclusive: a turn either speaks or presses keys. This drives keypad menus (IVR) and any agent that reacts to telephony tones:

turns:
  - dtmf: "123#"
    expect:
      - event: user_transcription
        text_contains: "DTMF: 123#"
      - event: response
        eval: "confirms the entered digits"

Each character is sent as one InputDTMFFrame, the same path a telephony transport’s keypress takes, regardless of the scenario’s user:/judge: modality. Valid characters are the keypad entries 0-9, *, and #; any other character is a parse error.

Quote the value in YAML (dtmf: "123#"). An unquoted # starts a YAML comment, so dtmf: 123# would silently drop the #. An unquoted all-digit sequence (dtmf: 123) is coerced to a string for you, but quoting is the safe habit.

A bot running a DTMFAggregator accumulates the keys and flushes them into a DTMF: ... transcription, which (with the default transcription-based turn-start strategy) drives a full user turn: user_started_speaking, user_transcription, user_stopped_speaking, and the agent’s response. So a dtmf: turn can assert on user_transcription and response just like a spoken turn. The aggregator flushes either on the # terminator or on its idle timeout. To exercise the idle-timeout path, omit the # and pace the keys with a time-based send_after::

turns:
  # Flushes immediately on the '#' terminator.
  - dtmf: "1#"
    expect:
      - event: response
        eval: "states the business hours"

  # No terminator: the aggregator flushes on its idle timeout instead.
  - dtmf: "2"
    expect:
      - event: response
        eval: "gives the office location"

Like any input turn, expect: is optional on a dtmf: turn: omit it for a turn that only presses keys, with the assertion living on a later turn.

Vision with `image:`

A turn may register an image with image: (a path relative to the scenario file). When a vision agent requests a user image during the turn, the eval transport serves it:

turns:
  - user: "What do you see in this image?"
    image: assets/cat.jpg
    expect:
      - event: response
        eval: "the response describes a cat"

Scheduling with `send_after:`

send_after: controls when a turn’s input (its user: utterance or dtmf: keypresses) is sent, either relative to a prior event or after a plain delay. Anchoring it to an event is how you script barge-in tests:

turns:
  - user: "Tell me a long, detailed story about the history of Paris."
    expect:
      - event: llm_started

  # Interrupt 2 seconds after the agent starts its long answer.
  - user: "Actually, never mind that. What's the capital of Japan?"
    send_after:
      event: llm_started
      delay_ms: 2000
    expect:
      - event: response
        eval: "the response says the capital of Japan is Tokyo, instead of continuing the Paris story"

The event: anchor is optional. A bare send_after: { delay_ms: 500 } is a pure time delay measured from the previous turn’s send, with no event to wait on. This is handy for pacing turns by time rather than off a bot event (for example, spacing out DTMF keypresses to exercise an aggregator’s idle-timeout flush):

turns:
  - dtmf: "1"
  # Wait 1.5s after the first keypress before sending the next.
  - dtmf: "2"
    send_after:
      delay_ms: 1500

A send_after: with no event: and a zero delay_ms is rejected as a no-op: give it an event:, a positive delay_ms, or both.

Events

Scenarios assert on a small set of semantic events, mapped from the RTVI messages the agent emits:

Event	Meaning
`response`	The agent’s reply. In audio mode this is a transcription of the agent’s actual synthesized speech; in text mode it resolves to `llm_response`. Prefer this for content checks.
`llm_response`	The LLM’s text output for the turn. Available in both modes.
`tts_response`	The text the TTS reports speaking, one segment at a time. Audio mode only.
`llm_started`	The LLM began generating a response.
`function_call`	The LLM called a function.
`function_call_stopped`	A function call ended. Its `args` carry `tool_call_id` and `cancelled`, so a scenario can tell work that was stopped from work that finished on its own.
`user_transcription`	The agent’s STT finalized a transcription of the user. Audio mode only, except on DTMF turns, where the aggregated keys become a transcription in either mode.
`user_started_speaking`	The agent’s VAD detected the start of user speech. Audio mode, or a DTMF turn in either mode.
`user_stopped_speaking`	The agent’s VAD detected the end of user speech. Audio mode, or a DTMF turn in either mode.

Use response for the agent’s reply unless you have a reason not to. It’s modality-agnostic: the same scenario judges LLM text in text mode and the transcription of real spoken audio in audio mode, so one file covers both.

Assertions

Each entry in expect: names an event and, optionally, asserts on its content or timing.

Semantic judging with `eval:`

The eval: field is a natural-language criterion that the event’s text must satisfy, decided by the judge LLM:

- user: "What's 2 plus 2?"
  expect:
    - event: response
      eval: "the response says the answer is four"

The judge sees the whole conversation so far, so it can resolve terse or context-dependent replies (like “That’s four”). It also understands that audio-mode responses come from a speech-to-text pass and judges intended meaning rather than exact spelling, so “for” transcribed instead of “four” still passes. The judge handles interim replies gracefully: if the agent says “Let me check on that.” before the real answer, the harness keeps accumulating response text and re-judges until the criterion is met or the time budget runs out. eval: only makes sense on the agent’s text output (response, llm_response, tts_response).

Substring checks with `text_contains:`

For exact content, text_contains: does a substring check, ignoring whitespace differences:

- user: "What is the capital of France?"
  expect:
    - event: response
      text_contains: "Paris"

On response, llm_response, and tts_response, the harness accumulates successive segments and re-checks on each new segment until the check passes. On user_transcription, it accumulates the STT’s final transcription segments within the turn, so an STT that finalizes an utterance in pieces can still satisfy a phrase that spans them.

Latency budgets with `within_ms:`

within_ms: bounds how long after the turn’s user send the event may arrive. All of a turn’s expectations share that one anchor:

- user: "What is the capital of France?"
  expect:
    - event: llm_started
      within_ms: 2000 # the LLM must start responding within 2s
    - event: response
      text_contains: "Paris"

When omitted, an expectation defaults to a generous 60 second budget (configurable with --timeout), so timing is only asserted when you ask for it. Because every deadline is measured from the send, time spent matching earlier expectations counts against later ones. In the example above, if llm_started arrives at 1.5 seconds, the response (with the default 60 second budget) has 58.5 seconds left, and a turn that stalls completely fails within a single budget rather than one per expectation.

Function calls

A function_call expectation asserts that the turn invoked one or more tools. List the expected calls under calls:; they’re matched by name in any order, and the expectation passes once all are found:

- user: "What's the weather in San Francisco? And recommend a restaurant."
  expect:
    - event: function_call
      calls:
        - name: get_current_weather
          args: { location: "San Francisco" }
        - name: get_restaurant_recommendation
    - event: response
      eval: "describes the weather and recommends a restaurant"

args is a subset check: every listed key/value must be present in the call’s arguments, and extra arguments are ignored. A single expected call can use the name:/args: shorthand directly on the expectation, and a bare function_call with neither just asserts that some call happened. Arguments take part in the matching, so the turn is satisfied by any call matching both the name and the arguments. A call the model gets wrong and immediately repeats correctly still passes. When nothing matches, the failure names the arguments that did arrive. function_call_stopped takes the same calls: shape and reports a call ending, which is how a scenario asserts that a cancellable tool was actually stopped:

- user: "Actually, cancel that report."
  expect:
    - event: function_call_stopped
      calls:
        - name: write_report
          args: { cancelled: true }

Next steps

Simulated Scenarios

Hand the user’s side to an LLM with a persona and a goal, and judge the whole conversation.

The Eval Loop

Let a coding assistant write agent code, run evals, and iterate automatically until the agent is better.

Get Started

Migration

Learning Pipecat

Fundamentals

Flows

Evals

Features

Telephony

Deployment

Examples & Recipes

Scripted Scenarios

Anatomy of a scenario

Configuration

User turns

Events

Assertions

Configuration

Seeding the context with `context:`

Scoring every turn

User turns

Utterances with `user:`

Playing audio files with `audio:`

DTMF keypresses with `dtmf:`

Vision with `image:`

Scheduling with `send_after:`

Events

Assertions

Semantic judging with `eval:`

Substring checks with `text_contains:`

Latency budgets with `within_ms:`

Function calls

Next steps

Simulated Scenarios

The Eval Loop

​Anatomy of a scenario

Configuration

User turns

Events

Assertions

​Configuration

​Seeding the context with context:

​Scoring every turn

​User turns

​Utterances with user:

​Playing audio files with audio:

​DTMF keypresses with dtmf:

​Vision with image:

​Scheduling with send_after:

​Events

​Assertions

​Semantic judging with eval:

​Substring checks with text_contains:

​Latency budgets with within_ms:

​Function calls

​Next steps

Simulated Scenarios

The Eval Loop

Anatomy of a scenario

Configuration

Seeding the context with `context:`

Scoring every turn

User turns

Utterances with `user:`

Playing audio files with `audio:`

DTMF keypresses with `dtmf:`

Vision with `image:`

Scheduling with `send_after:`

Events

Assertions

Semantic judging with `eval:`

Substring checks with `text_contains:`

Latency budgets with `within_ms:`

Function calls

Next steps