​
Overview

MoondreamService provides local image analysis and question-answering capabilities using the Moondream model. It runs entirely on your local machine, supporting various hardware acceleration options including CUDA, Intel XPU, and Apple MPS.

​
Installation

To use MoondreamService, install the required dependencies:

pip install "pipecat-ai[moondream]"

You can obtain a Moondream API key by signing up at Moondream.

​
Configuration

​
Constructor Parameters

​
model
str
default:"vikhyatk/moondream2"

Hugging Face model identifier

​
revision
str
default:"2024-08-26"

Model revision/version

​
use_cpu
bool
default:"False"

Force CPU usage instead of available accelerators

​
Hardware Acceleration

The service automatically detects and uses the best available hardware:

  1. Intel XPU (if intel_extension_for_pytorch is installed)
  2. NVIDIA CUDA
  3. Apple Metal (MPS)
  4. CPU (fallback)

​
Input

​
VisionImageRawFrame

​
format
str

Image format (e.g., ‘RGB’, ‘RGBA’)

​
size
tuple

Image dimensions (width, height)

​
image
bytes

Raw image data

​
text
str

Question about the image

​
Output Frames

​
TextFrame

​
text
str

Generated description or answer about the image

​
ErrorFrame

​
error
str

Error information if processing fails

​
Methods

See the Vision base class methods for additional functionality.

​
Usage Example

from pipecat.services.moondream.vision import MoondreamService
from pipecat.frames.frames import VisionImageRawFrame
from PIL import Image

# Configure service
service = MoondreamService(
    model="vikhyatk/moondream2",
    revision="2024-08-26"
)

# Create pipeline
pipeline = Pipeline([
    image_input,      # Produces VisionImageRawFrame
    service,          # Analyzes images
    text_handler      # Handles text responses
])

# Example frame processing
image = Image.open("example.jpg")
frame = VisionImageRawFrame(
    format=image.mode,
    size=image.size,
    image=image.tobytes(),
    text="What objects are in this image?"
)

​
Hardware Configuration Examples

​
CUDA (NVIDIA GPU)

# Automatically uses CUDA if available
service = MoondreamService()

​
Intel XPU

# Requires intel_extension_for_pytorch
import intel_extension_for_pytorch
service = MoondreamService()

​
Force CPU Usage

service = MoondreamService(use_cpu=True)

​
Frame Flow

​
Metrics Support

The service collects processing metrics:

  • Processing duration
  • Model loading time
  • Inference time

​
Performance Considerations

​
Memory Usage

  • Model size varies by version
  • GPU memory requirements depend on image size
  • CPU mode uses more system memory

​
Processing Speed

Relative performance by hardware:

  1. NVIDIA GPU (fastest)
  2. Intel XPU
  3. Apple MPS
  4. CPU (slowest)

​
Best Practices

​
1. Image Preparation

# Optimize image before processing
def prepare_image(image_path):
    image = Image.open(image_path)
    # Resize if needed
    if max(image.size) > 1024:
        image.thumbnail((1024, 1024))
    return image

​
2. Error Handling

try:
    async for frame in service.run_vision(vision_frame):
        if isinstance(frame, ErrorFrame):
            logger.error(f"Vision processing error: {frame.error}")
        elif isinstance(frame, TextFrame):
            process_result(frame.text)
except Exception as e:
    logger.error(f"Unexpected error: {e}")

​
3. Resource Management

# Initialize once, reuse for multiple images
service = MoondreamService()
try:
    # Process multiple images
    for image in images:
        await process_image(service, image)
finally:
    # Cleanup if needed
    await service.cleanup()

​
Notes

  • Runs completely offline after model download
  • First run requires model download
  • Supports multiple hardware acceleration options
  • Thread-safe processing
  • Automatic error handling
  • Manages model lifecycle
  • Supports various image formats