vmlx

MLX Inference Server for Apple Silicon

Self-hosted inference server for LLMs, VLMs, and image generation on Apple Silicon. OpenAI + Anthropic + Ollama compatible HTTP API. Self-hosted; no third-party API keys required. Native MTP artifact detection and family-specific cache policy gates keep speculative/cache settings explicit and model-safe.

Looking for a native Swift macOS app or Swift inference engine? See osaurus.ai.

Quickstart • Models • Features • Image Gen • API • Desktop App • JANG • CLI • Config • Contributing • 한국어

---

JANG 2-bit destroys MLX 4-bit on MiniMax M2.5:

Quantization	MMLU (200q)	Size
JANG_2L (2-bit)	74%	89 GB
MLX 4-bit	26.5%	120 GB
MLX 3-bit	24.5%	93 GB
MLX 2-bit	25%	68 GB

Adaptive mixed-precision keeps critical layers at higher precision. Scores at jangq.ai. Models at JANGQ-AI.

Chat with any MLX model -- thinking mode, streaming, and syntax highlighting Agentic chat with full coding capabilities -- tool use and structured output

---

Quickstart

Install from PyPI

Published on PyPI as vmlx -- install and run in one command:

# Recommended: uv (fast, no venv hassle)
brew install uv
uv tool install vmlx
vmlx serve mlx-community/Qwen3-8B-4bit

# Or: pipx (isolates from system Python)
brew install pipx
pipx install vmlx
vmlx serve mlx-community/Qwen3-8B-4bit

# Or: pip in a virtual environment
python3 -m venv ~/.vmlx-env && source ~/.vmlx-env/bin/activate
pip install vmlx
vmlx serve mlx-community/Qwen3-8B-4bit

Note: On macOS 14+, bare pip install fails with "externally-managed-environment". Use uv, pipx, or a venv.

The vMLX inference server is now running at http://0.0.0.0:8000 with an OpenAI + Anthropic compatible API. Works with any model from mlx-community -- thousands of models ready to go.

Or download the desktop app

Get MLX Studio -- a native macOS app with chat UI, model management, image generation, and developer tools. No terminal required. Just download the DMG and drag to Applications.

Use with OpenAI SDK

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
response = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
)
for chunk in response:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Use with Anthropic SDK

import anthropic

client = anthropic.Anthropic(base_url="http://localhost:8000/v1", api_key="not-needed")
message = client.messages.create(
    model="local",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello!"}],
)
print(message.content[0].text)

Use with curl

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": true
  }'

---

Model Support

vMLX runs any MLX model. Point it at a HuggingFace repo or local path and go.

Type	Models
Text LLMs	Qwen 2/2.5/3/3.5/3.6, Llama 3/3.1/3.2/3.3/4, Mistral/Mixtral, Mistral-Medium-3.5 (ministral3), Mistral-Small-4, Gemma 3/4, Phi-4, DeepSeek V2/V3/V4, GLM-4/5, MiniMax M2.5/M2.7, Nemotron, Laguna (poolside), ZAYA (CCA + MoE), Kimi K2.5/K2.6, StepFun, and any mlx-lm model
Vision LLMs	Qwen-VL, Qwen3.5-VL / Qwen3.6-VL, Pixtral, InternVL, LLaVA, Gemma 3n / 4-VL, Mistral-Medium-3.5 (PIXTRAL)
Multimodal Omni	Nemotron-3-Nano-Omni (text + image + audio + video) — Parakeet audio encoder + RADIO ViT vision tower; routed via OmniMultimodalDispatcher across /v1/chat/completions, /v1/messages, /v1/responses, /api/chat
MoE Models	Qwen 3.5/3.6 MoE (A3B/A10B), Mixtral, DeepSeek V2/V3/V4, MiniMax M2.5/M2.7, Llama 4, Laguna (256 routed experts top-8)
Hybrid SSM	Nemotron-H, Jamba, GatedDeltaNet (Mamba + Attention), Qwen3.5-A3B hybrid, Granite MoE Hybrid, LFM2
Image Gen	Flux Schnell/Dev, Z-Image Turbo (via mflux)
Image Edit	Qwen Image Edit (via mflux)
Embeddings	Any mlx-lm compatible embedding model
Reranking	Cross-encoder reranking models
Audio	Kokoro TTS, Whisper STT (via mlx-audio)

---

Features

Inference Engine

Feature	Description
Continuous Batching	Handle multiple concurrent requests efficiently
Prefix Cache	Reuse KV states for repeated prompts -- makes follow-up messages instant
Paged KV Cache	Block-based caching with content-addressable deduplication
KV Cache Quantization	Compress cached states to q4/q8 for 2-4x memory savings
Disk Cache (L2)	Persist prompt caches to SSD -- survives server restarts
Block Disk Cache	Per-block persistent cache paired with paged KV cache
Speculative Decoding	Small draft model proposes tokens for 20-90% speedup
Prompt Lookup Decoding	No draft model needed — reuses n-gram matches from the prompt/context. Best for structured or repetitive output (code, JSON, schemas). Enable with --enable-pld.
JIT Compilation	mx.compile Metal kernel fusion (experimental)
Hybrid SSM Support	Mamba/GatedDeltaNet layers handled correctly alongside attention
Distributed Compute	Pipeline parallelism across multiple Macs via Thunderbolt 5 / Ethernet / WiFi

Distributed Inference (Multi-Mac)

Run models too large for a single Mac across 2+ machines. Each Mac loads a sub