Moondream
Vision service implementation using Moondream for local image analysis and question answering
Overview
MoondreamService
provides local image analysis and question-answering capabilities using the Moondream model. It runs entirely on your local machine, supporting various hardware acceleration options including CUDA, Intel XPU, and Apple MPS.
Installation
To use MoondreamService
, install the required dependencies:
You can obtain a Moondream API key by signing up at Moondream.
Configuration
Constructor Parameters
Hugging Face model identifier
Model revision/version
Force CPU usage instead of available accelerators
Hardware Acceleration
The service automatically detects and uses the best available hardware:
- Intel XPU (if intel_extension_for_pytorch is installed)
- NVIDIA CUDA
- Apple Metal (MPS)
- CPU (fallback)
Input
VisionImageRawFrame
Image format (e.g., ‘RGB’, ‘RGBA’)
Image dimensions (width, height)
Raw image data
Question about the image
Output Frames
TextFrame
Generated description or answer about the image
ErrorFrame
Error information if processing fails
Methods
See the Vision base class methods for additional functionality.
Usage Example
Hardware Configuration Examples
CUDA (NVIDIA GPU)
Intel XPU
Force CPU Usage
Frame Flow
Metrics Support
The service collects processing metrics:
- Processing duration
- Model loading time
- Inference time
Performance Considerations
Memory Usage
- Model size varies by version
- GPU memory requirements depend on image size
- CPU mode uses more system memory
Processing Speed
Relative performance by hardware:
- NVIDIA GPU (fastest)
- Intel XPU
- Apple MPS
- CPU (slowest)
Best Practices
1. Image Preparation
2. Error Handling
3. Resource Management
Notes
- Runs completely offline after model download
- First run requires model download
- Supports multiple hardware acceleration options
- Thread-safe processing
- Automatic error handling
- Manages model lifecycle
- Supports various image formats