Smart Turn Detection

​

Overview

Smart Turn Detection is an advanced feature in Pipecat that determines when a user has finished speaking and the bot should respond. Unlike basic Voice Activity Detection (VAD) which only detects speech vs. non-speech, Smart Turn Detection uses a machine learning model to recognize natural conversational cues like intonation patterns and linguistic signals.

Pipecat provides two implementations of Smart Turn Detection:

1. SmartTurnAnalyzer - Uses a remote service for inference
2. LocalCoreMLSmartTurnAnalyzer - Runs inference locally on Apple Silicon using CoreML

Both implementations share the same underlying API and parameters, making it easy to switch between them based on your deployment requirements.

​

Installation

The Smart Turn Detection feature requires additional dependencies depending on which implementation you choose.

For remote inference:

pip install "pipecat-ai[remote-smart-turn]"

For local inference (CoreML-based):

pip install "pipecat-ai[local-smart-turn]"

​

Integration with Transport

Smart Turn Detection is integrated into your application by setting either SmartTurnAnalyzer or LocalCoreMLSmartTurnAnalyzer as the turn_analyzer parameter in your transport configuration:

from pipecat.transports.base_transport import TransportParams

transport = SmallWebRTCTransport(
    webrtc_connection=webrtc_connection,
    params=TransportParams(
        # Other transport parameters...
        turn_analyzer=SmartTurnAnalyzer(url=remote_smart_turn_url),
    ),
)

vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2))

​

Configuration

Both implementations use the same SmartTurnParams class to configure behavior:

​

Remote Smart Turn

The SmartTurnAnalyzer class uses a remote service for turn detection inference.

​

Constructor Parameters

​

Example

import os
from pipecat.audio.turn.smart_turn import SmartTurnAnalyzer
from pipecat.audio.turn.base_smart_turn import SmartTurnParams
from pipecat.transports.base_transport import TransportParams

# Get the URL for the remote Smart Turn service
remote_smart_turn_url = os.getenv("REMOTE_SMART_TURN_URL")

# Create the transport with Smart Turn detection
transport = SmallWebRTCTransport(
    webrtc_connection=webrtc_connection,
    params=TransportParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
        vad_enabled=True,
        vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
        vad_audio_passthrough=True,
        turn_analyzer=SmartTurnAnalyzer(
            url=remote_smart_turn_url,
            params=SmartTurnParams(
                stop_secs=3.0,
                pre_speech_ms=0.0,
                max_duration_secs=8.0
            )
        ),
    ),
)

​

Local Smart Turn (CoreML)

The LocalCoreMLSmartTurnAnalyzer runs inference locally using CoreML, providing lower latency and no network dependencies.

​

Constructor Parameters

​

Example

import os
from pipecat.audio.turn.local_smart_turn import LocalCoreMLSmartTurnAnalyzer
from pipecat.audio.turn.base_smart_turn import SmartTurnParams
from pipecat.transports.base_transport import TransportParams

# Path to the Smart Turn model directory
smart_turn_model_path = os.getenv("LOCAL_SMART_TURN_MODEL_PATH")

# Create the transport with local Smart Turn detection
transport = SmallWebRTCTransport(
    webrtc_connection=webrtc_connection,
    params=TransportParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
        vad_enabled=True,
        vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.2)),
        vad_audio_passthrough=True,
        turn_analyzer=LocalCoreMLSmartTurnAnalyzer(
            smart_turn_model_path=smart_turn_model_path,
            params=SmartTurnParams(
                stop_secs=2.0,  # Shorter stop time when using Smart Turn
                pre_speech_ms=0.0,
                max_duration_secs=8.0
            )
        ),
    ),
)

​

Local Model Setup

To use the LocalCoreMLSmartTurnAnalyzer, you need to set up the CoreML model locally. Follow these steps from the official repository instructions:

1. Install Git LFS (Large File Storage):

   macOS

   Ubuntu/Debian

   brew install git-lfs
   

2. Initialize Git LFS

   git lfs install
   

3. Clone the Smart Turn model repository:

   git clone https://huggingface.co/pipecat-ai/smart-turn
   

4. Set the environment variable to the cloned repository path:

   # Add to your .env file or environment
   export LOCAL_SMART_TURN_MODEL_PATH=/path/to/smart-turn
   

Note that the CoreML model is optimized for Apple Silicon devices. If you’re using a different platform, the remote Smart Turn service might be a better option.

​

How It Works

Smart Turn Detection continuously analyzes audio streams to identify natural turn completion points:

1. Audio Buffering: The system continuously buffers audio with timestamps, maintaining a small buffer of pre-speech audio.

2. VAD Processing: Voice Activity Detection segments the audio into speech and non-speech portions.

3. Turn Analysis: When VAD detects a pause in speech:

   • The ML model analyzes the speech segment for natural completion cues
   • It identifies acoustic and linguistic patterns that indicate turn completion
   • A decision is made whether the turn is complete or incomplete

The system includes a fallback mechanism: if a turn is classified as incomplete but silence continues for longer than stop_secs, the turn is automatically marked as complete.

​

Notes

• The model is designed for English speech; performance may vary with other languages
• You can adjust the stop_secs parameter based on your application’s needs for responsiveness
• Smart Turn generally provides a more natural conversational experience but is computationally more intensive than simple VAD