> ## Documentation Index > Fetch the complete documentation index at: https://docs.pipecat.ai/llms.txt > Use this file to discover all available pages before exploring further. # Real-time Context Standard > The open standard for context passed between real-time models. The Real-time Context Standard (RCS) defines how the models in a voice agent pass context to each other beyond plain text. A voice agent is a chain of models: speech-to-text transcribes the user, an LLM writes a reply, and text-to-speech speaks it. Between them they pass text, so what isn't words is lost along the way, such as how the user sounded, who said what, whether they meant to take a turn or were only saying "mm-hmm", and how the reply should be said. RCS carries that context in the same text stream, in formats any model in the chain can produce and read. RCS 0.1 carries how the user sounded and how the agent should speak. Who is speaking and when someone means to speak are on the [roadmap](#roadmap). > [!WARNING] > > This page documents RCS 0.1, an alpha. Until 1.0, any version may change its > formats, vocabularies and context types in ways that aren't backward > compatible. See [Versions](#versions). ## Design Principles RCS context is written by an LLM as it streams a reply, and read by models that act on it a sentence at a time. Every token of markup is a token the LLM generates before the user hears it, so the formats trade a little expressiveness for as few tokens as they can: * **Few tokens.** A tag is a short name in brackets, such as `<>` or `[[laugh]]`. It costs a few tokens where SSML or other XML markup costs several times as many. * **No closing tags across sentences.** A delivery tag lasts until the next one, and a span closes within its sentence. Each sentence can be spoken as soon as it's written, without waiting for the markup to close. * **Safe to cut off.** A reply can stop anywhere, at its end or when the user interrupts. What's left of an unfinished tag is removed, so nothing malformed is spoken. * **Readable inline.** Context travels in the text the models already pass each other. The markup stays legible to an LLM that reads it back in later turns. ## Conformance MUST, SHOULD and MAY are used as defined in [RFC 2119](https://www.rfc-editor.org/rfc/rfc2119). A model, with any processor beside it, conforms to RCS as a producer or a consumer of a context type when it meets that type's requirements. It doesn't need to support every context type, or both roles. The grammars are [ABNF](https://www.rfc-editor.org/rfc/rfc5234), with prose in `< >`. ## Terms * **Context model**: a model in a voice agent that produces or consumes context, such as a speech-to-text (STT) model, an LLM, a text-to-speech (TTS) model, or a classifier. * **Context type**: a kind of context RCS carries. Each context type defines its direction, its format, its vocabulary, and how a model declares support for it. * **Producer**: the model, or a processor beside it, that writes context of a type. * **Consumer**: the model, or a processor before it, that reads context of a type, or acts on it. * **Capabilities**: what a producer reports or a consumer performs, declared before the conversation starts, so the models upstream only write what will be used. ## How Context Flows ```mermaid theme={null} flowchart LR U([User audio]) --> STT[STT and perception producers] STT -- "words + perception" --> LLM[LLM] LLM -- "words + expression tags" --> TTS[TTS] TTS --> A([Agent audio]) ``` A context type travels inline, with the words it describes. Capabilities are declared out of band. RCS defines the fields of each declaration, listed under each context type; how a declaration reaches the models that read it, and in what encoding, is up to the implementation. ## Perception How the user said what they said: their tone, how fast they spoke, and other signals read from the audio or the words. * **Direction**: from an STT model, or a classifier model after it, to the LLM. * **Producers**: any model that reads how a user turn was spoken. * **Consumers**: the LLM, which reads it at the head of the user's message. ### Format The reading comes first in the user's message, followed by a space and the words: ```text wrap theme={null} ([tone frustrated=0.86, neutral=0.10],[speech_rate value=4.1]) Can you tell me where my order is? ``` ```text theme={null} reading = "(" signal *( "," signal ) ")" signal = "[" name SP ( labels / value ) "]" labels = label "=" confidence *( ", " label "=" confidence ) value = "value=" number name = 1*name-char label = 1*name-char name-char = confidence = DIGIT "." 2DIGIT ; 0.00 to 1.00 number = [ "-" ] 1*DIGIT [ "." 1*DIGIT ] [ "e" ( "-" / "+" ) 1*DIGIT ] ``` * A signal read as labels gives its likeliest labels, likeliest first, each with a confidence from 0 to 1. A confidence below 0.5 is a guess. * A signal read as a value gives a measure, such as a rate or a level. * `value`, and the `e` of an exponent, are lowercase. `value` is reserved: it isn't a label. ### Producer Requirements * A producer MUST write a reading only at the head of the user's message, followed by one space. * A producer MUST write only the final reading of a turn. A turn with no reading MUST have no prefix. * A producer MUST list a signal's labels likeliest first. * A producer MUST declare each signal it reports before the conversation starts, and MUST NOT report a signal it didn't declare. ### Consumer Requirements * A consumer MUST read the reading as context about the user's words, not as words the user said. * A consumer MUST NOT write a reading in its own output. * A consumer SHOULD NOT act on a label with a confidence below 0.5. * A consumer MAY ignore a signal it doesn't recognize. ### Declaring Perception A producer declares each signal it reports before the conversation starts, so the LLM can be told how to read them: | Field | Meaning | | - | - | | `name` | The signal's name, as written in a reading, such as `tone` | | `description` | A short line on how to read it, such as its unit or range, or empty | | `labels` | The labels it can report, for a signal read as labels; empty for values | RCS 0.1 has no shared vocabulary of signals: a producer names its own. ## Expression How the agent should say its reply: the delivery of its words, and the sounds and pauses between them. * **Direction**: from the LLM, or a classifier model after it, to the TTS model. * **Producers**: an LLM taught the tags, or a classifier model that adds tags to an LLM's plain reply. * **Consumers**: the TTS model, and anything that follows the agent's delivery, such as a client application or a real-time avatar. ### Tags | Tag | Example | Description | | - | - | - | | `<>` | `<>` | Sets the delivery of the words after it, until the next delivery tag or the end of the response. A delivery can be several words, such as an accent | | `<>` | `<>` | Returns to the voice's baseline, until the next delivery tag or the end of the response | | `<>` | `<>` | A span: speaks just those words in that delivery, added to the delivery around them. `<>` speaks them in the baseline. Spans don't nest | | `^^words^^` | `^^did^^ ship` | Stresses the words | | `[[sound]]` | `[[laugh]]` | Makes a sound at that point | | `[[pause N]]` | `[[pause 600]]` | Pauses for N milliseconds | ```text wrap theme={null} <> I'm so sorry about the wait. <> Good news, though: it ^^did^^ ship yesterday. [[breath]] Your tracking number is <>. ``` ### Syntax ```text theme={null} delivery-tag = "<<" name ">>" span = "<<" span-name ":" *WSP span-words ">>" emphasis = "^^" stressed "^^" sound-tag = "[[" name "]]" pause-tag = "[[pause" SP 1*DIGIT "]]" name = *( span-char / ":" ) span-name = 1*span-char span-char = letter / DIGIT / "_" / " " / "'" / "-" / "/" letter = span-words = 1* stressed = 1* ``` * A tag doesn't cross a line break. * Where text matches more than one rule, a span takes precedence over a delivery tag and a pause over a sound tag. Stressed words end at the first `^^` after them. * Names are case-sensitive, and universal names and `pause` are lowercase. Spaces at either end of a name are ignored. * An empty `<<>>` returns to the baseline, as `<>` does. An empty `[[]]` is left out. * A span can hold stressed words, but not another span or tag. Stressed words can't hold a tag. * RCS 0.1 has no escape. Text that matches a tag is read as one. An unpaired `^^`, or a `<<` or `[[` left open at the end of a response, is removed. Any other `<<` or `[[` that doesn't start a tag is text. ### Producer Requirements * A producer SHOULD write only the deliveries and sounds the consumer declared, by their universal names where one exists. * A producer SHOULD write a pause only within the consumer's declared range, and stress only when the consumer declared emphasis. * A producer MUST NOT write a tag inside a span or inside stressed words. * A producer MUST write stress as `^^words^^`, not as a delivery. ### Consumer Requirements * A consumer MUST NOT speak a tag. It MUST leave a tag it can't perform out of the speech. * A consumer MUST read a tag's kind from its brackets, except that a universal name, or a name the consumer declared, keeps its kind in the other brackets: `[[warm]]` still sets a delivery, and `<>` is still a sound. * A consumer MUST remove an unfinished tag at the end of a response, or one cut off by an interruption, without speaking it. * A consumer MUST perform each delivery and sound it declared when it's written by its universal name, as well as by the consumer's own name. The agent SHOULD keep a reply the LLM wrote with tags in the conversation's context as written, so later turns see how it was said. ### Universal Names Deliveries and sounds are named by their universal names, the same whatever the TTS model. A consumer translates each into its own name for it, such as a TTS model that performs `warm` as `warmly`. * **One spelling per word**: an adjective for a delivery, such as `nervous`, and a base word for a sound, such as `laugh`. * **Different words stay distinct**, so `content` and `happy` are both names. * **Only sounds a voice makes** have names. Sound effects, such as applause, have none. * **Any other name** passes through as written, for a consumer that takes free text, such as an accent. Two names are reserved: `neutral`, the voice's baseline, and `emphasis`, the delivery reported for stressed words. > **Deliveries (118)** > > `affectionate`, `agitated`, `alarmed`, `amazed`, `amused`, `angry`, > `annoyed`, `anticipating`, `anxious`, `apologetic`, `appalled`, `ashamed`, > `bored`, `calm`, `cautious`, `compassionate`, `confident`, `confused`, > `contemplative`, `contemptuous`, `content`, `crying`, `curious`, `cute`, > `deadpan`, `dejected`, `delighted`, `depressed`, `desperate`, `determined`, > `disappointed`, `disdainful`, `disgusted`, `dismissive`, `distant`, > `doubtful`, `dramatic`, `elated`, `embarrassed`, `empathetic`, > `encouraging`, `enthusiastic`, `envious`, `euphoric`, `excited`, `fast`, > `fearful`, `flirtatious`, `frustrated`, `grateful`, `guilty`, `happy`, > `hesitant`, `hopeful`, `hurt`, `hysterical`, `impressed`, `indifferent`, > `insecure`, `ironic`, `jealous`, `lonely`, `loud`, `mad`, `melancholic`, > `mischievous`, `mocking`, `moved`, `muttering`, `mysterious`, `nervous`, > `nostalgic`, `optimistic`, `outraged`, `panicked`, `peaceful`, > `pessimistic`, `playful`, `professional`, `proud`, `questioning`, > `reassuring`, `regretful`, `rejected`, `relaxed`, `relieved`, `resigned`, > `sad`, `sarcastic`, `satisfied`, `scared`, `screaming`, `serene`, `serious`, > `sheepish`, `shouting`, `sincere`, `singing`, `skeptical`, `slow`, `soft`, > `somber`, `stern`, `surprised`, `sympathetic`, `thoughtful`, `threatened`, > `tired`, `triumphant`, `trusting`, `uncertain`, `unhappy`, `upset`, > `urgent`, `warm`, `whispering`, `wistful`, `worried` > **Sounds (46)** > > `breath`, `burp`, `cackle`, `chuckle`, `clear throat`, `cough`, `cry`, `deep > breath`, `exhale`, `gasp`, `giggle`, `groan`, `growl`, `grunt`, `gulp`, > `hiccup`, `hiss`, `huff`, `hum`, `inhale`, `laugh`, `laugh harder`, `light > breath`, `lip smack`, `moan`, `pant`, `scoff`, `sharp exhale`, `shriek`, > `shush`, `sigh`, `sneeze`, `sniff`, `sniffle`, `snore`, `snort`, `sob`, > `start laughing`, `swallow`, `tongue click`, `um`, `wheeze`, `whimper`, > `whistle`, `woo`, `yawn` ### Reporting Delivery A consumer that reports what it performed, such as to a client app, reports the delivery of each word as a list of universal names: the delivery in effect, a span's delivery added to it, and `emphasis` for a stressed word. `<> it ^^did^^ ship` reports `did` as `["warm", "emphasis"]`. An empty list is the voice's baseline. Each sound and pause is reported on its own, where it is performed. ### Declaring Expression A TTS model declares what it performs before the conversation starts, so the producer is offered only those tags: | Field | Meaning | | - | - | | Deliveries | The deliveries it performs, by its own names | | Sounds | The sounds it performs, by its own names | | Free text | Whether it also takes deliveries, or sounds, described in a few words, declared for each kind on its own | | Universal names | Its own name for each universal name, where they differ | | Pauses | The shortest and longest pause it makes, or none | | Emphasis | Whether it stresses words | | Where delivery changes | Anywhere in a response, or only between sentences. A model that changes it only between sentences speaks a span's whole sentence in the span's delivery | ## Versions | Version | Changes | | - | - | | 0.1 | First version: perception and expression. | In 0.1, declarations don't carry a version. ## Implementations * **Pipecat** 1.15.0 and later implement RCS 0.1 in [expressive conversation](https://docs.pipecat.ai/pipecat/features/expressive-conversation.md). * **Pipecat clients** receive reported delivery, sounds, pauses and perception readings over [RTVI](https://docs.pipecat.ai/api-reference/standards/rtvi-standard.md) 2.3.0 and later. RCS doesn't define how context reaches client apps. ## Roadmap Context types under consideration for later versions: * **Diarization**: who is speaking, for conversations with more than one person. * **Intent**: when someone means to speak, and whether what they said needs a response. ## Contributing RCS is an open standard, for any framework or model to produce and consume. To propose a change or a new context type, open an issue on [GitHub](https://github.com/pipecat-ai/pipecat/issues). This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.