Utilities
Pipeline Heartbeats
Monitor pipeline health with heartbeat frames
Overview
Pipeline heartbeats provide a way to monitor the health of your pipeline by sending periodic heartbeat frames through the system. When enabled, the pipeline will send heartbeat frames every second and monitor their progress through the pipeline.
Enabling Heartbeats
Heartbeats can be enabled by setting enable_heartbeats
to True
in the PipelineParams
:
How It Works
When heartbeats are enabled:
- The pipeline sends a
HeartbeatFrame
every second - The frame traverses through all processors in the pipeline, from source to sink
- The pipeline monitors how long it takes for heartbeat frames to complete their journey
- If a heartbeat frame isn’t received within 5 seconds, a warning is logged
Monitoring Output
The system will log:
- Trace-level logs showing heartbeat processing time
- Warning messages if heartbeats aren’t received within the monitoring window
Example warning message:
Use Cases
Heartbeat monitoring is useful for:
- Detecting pipeline stalls or blockages
- Monitoring processing latency through the pipeline
- Identifying performance issues in specific processors
- Ensuring the pipeline remains responsive
Configuration
The heartbeat system uses two key timing constants:
HEARTBEAT_SECONDS = 1.0
- Interval between heartbeat framesHEARTBEAT_MONITOR_SECONDS = 5.0
- Time before warning if no heartbeat received
These values are currently fixed but may be configurable in future versions.