Skip to main content
The EvalConversationObserver records a bot’s conversation in the same <stem>.conversation.json format that pipecat eval simulation runs save. This lets you judge a call the eval harness didn’t drive, such as a production call, against a simulation’s success criterion and metrics with pipecat eval judge.

​
What Gets Recorded

Each side’s turn is what the context aggregators write to the LLM context:
  • User turns: What the bot heard. Text a client sends with RTVI’s send-text is also recorded as a user turn.
  • Bot turns: What the bot said, cut where the user interrupted it.
  • Function calls: Each call the bot makes, recorded in place in the conversation. Calls that are cancelled are removed.
Each turn is recorded as it is written, so summarizing the context later doesn’t lose it. The context’s other messages, such as tool changes made by Pipecat Flows, are left out. The observer records what passes through the worker it is attached to. In a bot with several workers, attach it to the worker holding the conversation’s context; calls made in other workers are not recorded.

​
Events

The observer emits one event:
  • on_conversation_ended: Emitted once the pipeline is done
    • Parameters: observer (EvalConversationObserver), message (EvalConversationEndedMessage)

​
Usage

Then judge the saved file against a simulation scenario:

​
EvalConversationEndedMessage

​
list[dict]
required
The conversation’s messages, in the format of a conversation file.