book_table.yaml
persona: is a simulation; one with turns: is a scripted scenario. In a scripted scenario you control the user’s side exactly and assert exactly what the agent does. In a simulation you give the caller a goal and let them adapt to the agent. One file covers the many ways a conversation can go, and you check that the goal was reached and that each reply met your quality bar. Both run with the same commands, sit side by side in a suite manifest, and share the user: and judge: blocks.
Anatomy of a simulation
The caller
The persona, its goal, and the LLM that plays it.
Text and audio
Drive the agent with text or synthesized speech, and judge text or real
audio.
Judging the outcome
The
success: criterion, and what the judge sees.Metrics
Judged criteria scored per reply, and the built-in measures.
The caller
Persona and goal
persona: says who the caller is and goal: what they want from the call. Both go word for word into the persona LLM’s instructions. The instructions tell it to stay in character, reply with one short spoken turn at a time, ask for or give one thing at a time, and end the call once the goal is reached or clearly out of reach:
end_call tool, reporting whether it thinks it succeeded and why. That claim is advisory: it’s shown in verbose output and recorded in the result, but the judge decides the outcome.
The persona listens first. It answers each finished response of the agent, so
the agent has to open the conversation, as most do with a greeting on connect.
An agent that waits for the user to speak leaves the persona waiting too, and
the run ends as
silence after max_silence_s.The persona LLM with simulator:
simulator: configures the LLM that plays the caller. It is optional: without one, the persona runs on the same local Ollama model as the default judge, gemma4:12b, so a simulation needs no API key. The block has the same shape as judge.eval:: a service (ollama, the only built-in), a model, an optional endpoint:, and an extra: mapping forwarded as request parameters. This is the default, written out:
factory:, a dotted path to a callable that takes the block and returns an OpenAI-compatible LLM service, exactly as for the judge:
end_call tool. A model that writes the call out as text never hangs up. pipecat eval loads the nearest .env file, walking up from the working directory, before it runs, so a factory finds its API key in the environment the same way your agent does. Variables already set in the shell win.
Text and audio modes
A simulation takes the same top-leveluser: and judge: blocks as a scripted scenario, with the same defaults, so the Scenario Configuration page applies here: user.modality and user.speech decide whether the persona’s turns reach the agent as text or as synthesized speech, and judge.modality, judge.transcription, and judge.eval decide whether the agent speaks and which LLM judges the outcome. factory: and !include work the same way.
With neither block, a simulation runs entirely in text mode: the persona’s replies are sent to the agent as text, the agent’s TTS is skipped, and the persona hears the LLM’s text output. This is the fast way to check a flow end to end and to iterate on prompts and tools.
In audio mode every persona turn is synthesized, so user.modality: audio requires a user.speech: block naming the TTS service and voice. The agent’s real VAD, STT, and turn taking run against an autonomous caller, and the persona hears the agent’s transcribed speech:
latency measure means what a caller experiences; see Measured metrics.
Interruptions aren’t scripted in a simulation. The persona is a caller: when the agent keeps talking over it, its speech is cut off the way a real caller’s would be, and the conversation carries on from there. A speech-to-speech agent has no separate text LLM step, so for it both audio blocks are required.
Judging the outcome
The success: criterion
success: says what counts as the agent having done its job, judged once over the whole conversation. Write it as prose, as long as it needs to be. It is the agent’s side of the goal:. Usually that means the caller got what they asked for. When the right outcome is to say no, to add a condition, or to hand off, success: says so:
complete_order, belongs in success:, not in a metric. Metrics are scored on every reply.
What the judge sees
The judge reads the whole transcript in one call, with each of the agent’s function calls placed before the reply it preceded, by name and arguments. A completed call is stronger evidence of an action than the agent saying it did it, and the judge is told so. The judge does not see a call’s result. So:- Whether the agent made a call at all is a
function_callsmeasure, no judge needed. - If a reply must match backend data, write the expected value into the criterion (“the reply says the appointment is on Tuesday September fifteenth”) and keep your mocks deterministic so it stays true across runs.
judge.eval: block, Ollama with gemma4:12b by default.
Metrics
metrics: is a list of quality checks. Each is either judged, a criterion: the judge decides on every agent reply, or measured, a measure: the harness computes from the run with no judge involved. A metric fails the run only when it has a threshold: min_score: for a judged metric, a range or a calls: list for a measured one. Without one, it is reported and never fails anything.
Judged metrics with criterion:
A criterion says what every reply of the agent should be. The judge answers yes or no for each agent turn, never a partial score, taking into account the conversation before that turn and the tool calls the agent had made by then. The metric’s score is the share of turns that got a yes: 0.80 is four replies in five.
min_score: is the share of turns that must pass, in 0..1: 1 means every reply, 0.8 allows one slip in five. The brevity metric above has none, so its score is reported but never fails the run.
Phrase a rule as a condition plus what a reply outside it does. The judge is told that a rule which forbids something, or only applies in some situation, is met by a reply that doesn’t do that thing or isn’t in that situation. Spelling it out anyway (“a reply that turns down no time passes”) keeps a bare “never” from being read as “always”.
A turn the judge gives no answer for counts as a no, but is recorded as none rather than no, so you can tell judge trouble from agent trouble. A metric that failed only on unanswered turns has the failure_kind judge_no_verdict; one the judge said no to has judge_no. A run in which the agent said nothing has no turns to score, so a judged metric is unscored and passes.
Measured metrics with measure:
A measured metric names one of the built-in measures and bounds it with min_value: and/or max_value:, both inclusive. The harness computes the value from the run, and the metric scores 1 inside the range and 0 outside, which fails the run. name: defaults to the measure:
The per-reply measures,
words and latency, bound the worst reply, so max_value: 5 on latency means no reply took longer than five seconds. In text mode there is no speech, so latency times the agent’s own work before it starts answering: any tool call, then the LLM’s first token. It is a budget on the LLM, not on what a caller would hear. Only in audio mode does it measure what a caller experiences. The two are not comparable, so a failed latency metric says what it timed, “to the first token” or “from the persona stopping to the first spoken sentence”. A measure outside its range fails with the failure_kind out_of_range; a function_calls list that did not match, with function_calls.
There is deliberately no interruptions measure. The persona causes interruptions; the agent cannot.
Checking tool calls with function_calls
measure: function_calls takes a calls: list instead of a range: the calls the agent should make over the whole conversation, each a name or a name: with args:, in any order. args: is a subset check, so every listed key and value must appear in the call’s arguments and extra arguments are ignored:
calls: [] the check for a caller who should be turned down: the agent must call nothing. A call the agent cancelled did not happen.
Limits and runs
max_turns: caps the persona’s turns (default 20), max_duration_s: caps the run’s wall clock (default 300 seconds), and max_silence_s: caps how long neither side does anything (default 30 seconds). An agent that never greets, or stops answering, ends the run as silence instead of running out the clock. A reply in progress is never silence: a token or a spoken sentence from either side resets the timer. These limits stop a conversation that never ends. A run they cut short has not succeeded, whatever the transcript says. If the harness’s own pipeline fails, for example the persona LLM, the run ends at once as an error.
runs: is how many times a suite runs the simulation (default 1), and every run must pass. A persona doesn’t say the same thing twice, so one run proves little and three are a real check:
pipecat eval run plays a simulation once whatever runs: says. A success rate over several runs is the suite’s job, since each run needs a fresh agent. The suite’s --repeat flag overrides runs: and turns the sweep into a measurement instead: rates are reported and the exit code stays 0.
trigger_disconnect: true fires the agent’s on_client_disconnected handler when the run ends. See Exercising the disconnect path.
How a run ends and what passing means
A run ends in one of these ways:
A run passes when it completed, the judge said
success: was met, and no metric with a threshold fell short. A run that ended in error is neither a success nor a failure. It is reported as an error and left out of any pass rate, because a run that never connected says nothing about whether the agent did its job.
Running a simulation
Start your agent with its eval transport, then run the file the same way as a scripted scenario. Pass-v to watch the conversation as it happens and read the judge’s reasons at the end:
<scenario>.eval.log, and in results.jsonl when a suite ran it. The judge and the persona LLM’s own logs are in <scenario>.debug.log under -d.
Writing good simulations
- Give the persona the facts the agent will ask for. A caller who doesn’t know their own phone number stalls, and the run fails for a reason that isn’t the agent’s.
success:is the agent’s side of the goal, and may name tool calls. Anything the agent must do once belongs here, not in a metric.- A judged metric is scored per reply. Phrase it as a condition plus what a reply outside it does, so a rule about one situation doesn’t fault every other reply.
- A measured metric bounds the worst reply. Set
latencyandwordsfor the slowest and longest reply you can accept, not the typical one. - Use
function_callsfor what the judge can’t see. The judge reads calls, not results, andcalls: []is the check for an agent that should call nothing. - Run it more than once. Set
runs: 3and let the suite prove the flow holds, not just that it held once. - Use a script when you need exact control. If the test is “when the user says this, the agent must call this tool with these arguments”, write a scripted scenario. A simulation is for checking a goal, not a specific exchange.
Next steps
Eval Suites
List scripted and simulated scenarios in one manifest, run each simulation
its
runs: times, and read results.jsonl.Using the Library
Run simulations from Python, inject your own persona LLM or judge, and read
per-metric scores.