py-xiaozhi
huangjunsen0406 · huangjunsen0406/py-xiaozhi
This open-source ecosystem provides an MCP server for building AI assistants with multimodal workflows, IoT support, and cross-platform voice interaction.
Open-source AI assistant ecosystem with MCP integrations, multimodal workflows, IoT support, and cross-platform voice interaction.
Preview
What you can do with it
- Build a custom AI assistant with voice control and IoT device integration.
- Create multimodal AI agents capable of processing various data types.
- Develop a cross-platform AI solution for diverse application scenarios.
Safety check
No risky patterns found in the files we scanned.
Automated scan of the repo files. Always review code you run.
From the README
py-xiaozhi
English | 简体中文
❤️ Sponsors
Click to collapse
Thanks to GitDo.net for sponsoring this project! GitDo.net is an AI API aggregation platform — one API Key for Claude, Gemini, GPT and other major models. Direct connection, no proxy needed, stable and efficient. An ideal choice for enterprise-grade AI programming. Visit GitDo.net
| Token能量站 | Thanks to Token能量站 (Factory.pub) for sponsoring this project! Providing API relay for GPT, Grok, Claude and other major models — stable, reliable, and affordable. Register here |
| 良心AI | Thanks to 良心AI for sponsoring this project! Providing API relay for GPT, Claude, Gemini and other major models — direct connection, stable and high-speed. Register here |
About
py-xiaozhi is a lightweight, cross-platform multi-modal AI interaction framework built on Python's async architecture. It supports real-time voice streaming, vision-language tasks, and IoT device control. Deployable across Windows, macOS, Linux desktops, and ARM embedded platforms (Raspberry Pi, Horizon Robotics RDK, Jetson Nano), it bridges the gap between Large Language Models and physical hardware — out of the box.
Evolved from the xiaozhi-esp32 firmware project. Officially adopted by D-Robotics (xiaozhi-in-rdk) as an upstream dependency.
Related Projects
- xiaozhi-desktop — Electron desktop client with AEC echo cancellation, Live2D, floating window modes, and Windows / macOS installers
Demo
Key Features
- Real-time Voice AI — Opus codec with auto frame detection (RFC 6716 TOC parsing), async streaming, sub-20ms latency
- Multi-modal Vision — Camera capture + vision-language model integration for image understanding and scene perception
- MCP Tool Ecosystem — Modular JSON-RPC 2.0 tool server: music player, camera, screenshot, app management, weather, volume control
- Cross-platform Deployment — Windows 10+ / macOS 10.15+ / Linux (x86_64 & ARM), optimized for Raspberry Pi and edge boards
- Multiple UI Modes — PySide6 + QML GUI / CLI / GPIO, adapting to desktop, headless server, and embedded environments
- Offline Wake Word — Sherpa-ONNX based on-device keyword spotting with custom wake word support
- IoT & Embodied AI Ready — GPIO interface for robotics control, hardware actuation, and sensor integration
- WebSocket / MQTT — Dual protocol communication with WSS/TLS encrypted transmission and auto-reconnection
- Plugin Architecture — Event-driven async design, clean dependency injection, extensible plugin system
System Requirements
Basic Requirements
- Python Version: 3.10 - 3.12
- Operating System: Windows 10+, macOS 10.15+, Linux
- Audio Devices: Microphone and speaker devices
- Network Connection: Stable internet connection (for AI services and online features)
Recommended Configuration
- Memory: At least 4GB RAM (8GB+ recommended)
- Processor: Modern CPU with AVX instruction set support
- Storage: At least 2GB available disk space (for model files and cache)
- Audio: Audio devices supporting 16kHz sampling rate
Optional Feature Requirements
- Voice Wake-up: Requires downloading Sherpa-ONNX speech recognition models
- Camera Features: Requires camera device and OpenCV support
Read This First
- Carefully read 项目文档 for startup tutorials and file descriptions
- The main branch has the latest code; manually reinstall pip dependencies after each update to ensure you have new dependencies
Zero to Xiaozhi Client (Video Tutorial)
Technical Architecture
Core Architecture Design
- Event-Driven Architecture: Based on asyncio asynchronous event loop, supporting high-concurrency processing
- Layered Design: Clear separation of application layer, protocol layer, and UI layer
- Dependency Injection: Component lifecycle managed via bootstrap container
- Plugin System: Audio, UI, MCP tools and other components loaded via plugin system
Key Technical Components
- Audio Processing: Opus codec, real-time resampling
- Speech Recognition: Sherpa-ONNX offline models, wake word recognition
- Protocol Communication: WebSocket/MQTT dual protocol support, encrypted transmission, auto-reconnection
- Configuration System: Hierarchical configuration, dot notation access, dynamic updates
Performance Optimization
- Async First: Full system asynchronous architecture, avoiding blocking operations
- Memory Management: Smart caching, garbage collection
- Audio Optimization: 5ms low-latency processing, queue manag