Learn how to write your own custom FrameProcessor
OpenAILLMService
or CartesiaTTSService
, utilities, like UserIdleProcessor
, and other things. Largely, you can build most of your application with these built-in FrameProcessors, but commonly, your application code may require custom frame processing logic. For example, you may want to perform an action as a result of a frame that’s pushed in the pipeline.
BotStartedSpeakingFrame
and BotStoppedSpeakingFrame
. When it sees a BotStartedSpeakingFrame
, it will show an image that says the bot is speaking. When it sees a BotStoppedSpeakingFrame
, it will show an image that says the bot is not speaking.
ImageSyncAggregator
FrameProcessorImageSyncAggregator
FrameProcessor will receive the BotStartedSpeakingFrame
and BotStoppedSpeakingFrame
outputted by the TTS processor and then push its own frame—OutputImageRawFrame
—to the output transport.
FrameProcessor
class. This ensures that your custom FrameProcessor will correctly handle frames like StartFrame
, EndFrame
, StartInterruptionFrame
without having to write custom logic for those frames. This inheritance also provides it with the ability to process_frame()
and push_frame()
:
process_frame()
is what allows the FrameProcessor to receive frames and add custom conditional logic based on the frames that are received.push_frame()
allows the FrameProcessor to push frames to the pipeline. Normally, frames are pushed DOWNSTREAM, but based on which processors need the output, you can also push UPSTREAM or in both directions.super().__init__()
in your __init__
methodawait super().process_frame(frame, direction)
in your process_frame()
methodImageSyncAggregator
’s process_frame()
method. It handles both bot speaking frames and also has an await self.push_frame(frame)
which pushes the frame through to the next processor in the pipeline.
super().__init__()
and await super().process_frame()
await self.push_frame(frame)
isinstance()
checks to handle specific frame types