jundot/omlx

Project overview

The project combines an SSD-backed KV cache that persists across server restarts with a native macOS menu bar interface, allowing users to manage concurrent requests, swap models, and retain context without terminal operations.

Project type

MCP · Model Runtime

Use cases

Chat Assistants · Coding & Development

Deployment

Refer to project documentation

License

Apache-2.0

Best for

• Developers, AI engineers, and general users on Apple Silicon hardware (M1/M2/M3/M4) running macOS 15.0+ who need a local model API service or local ChatGPT alternative.
• Users who want to manage server operations, model swapping, and per-model settings without terminal commands, using a native macOS menu bar app and a web dashboard.

Key capabilities

• Block-based KV cache management with prefix sharing and Copy-on-Write. Offloads cold blocks to SSD in safetensors format and restores them on matching prefix, even after a server restart.
• Handles concurrent requests through mlx-lm's BatchGenerator with configurable max concurrent limits.
• Serves LLMs, VLMs, embedding models, and rerankers simultaneously with LRU eviction, manual load/unload, model pinning, per-model TTL, and total process memory enforcement.
• Native Swift / SwiftUI menubar app to start, stop, and monitor the server without opening a terminal, featuring persistent stats and auto-restart on crash.
• Web UI at `/admin` for real-time monitoring, model management, chat, benchmark, and per-model settings with multi-language support and offline CDN dependencies.
• Provides API compatibility for OpenAI and Anthropic APIs supporting chat completions, text completions, embeddings, reranking, messages API, streaming, and vision inputs.
• Runs VLMs with multi-image chat, base64/URL/file inputs, and tool calling. Auto-detects and optimizes OCR models like DeepSeek-OCR, DOTS-OCR, GLM-OCR.
• Search and download MLX models from HuggingFace directly via the admin dashboard.

Limitations and risks

• Requires Apple Silicon (M1/M2/M3/M4).
• Native custom kernels require full Xcode to build from source.
• Plain pip install without custom kernels leads to 30x slower fallback paths for GLM-5.2.

Getting started

• Install via graphical DMG installer or Homebrew formula. The documented quick start path uses 'brew install omlx', 'omlx start', and navigating to 'http://localhost:8000/admin/chat'. Auto-update functionality is available.

Alternatives and comparisons

• An MLX-native Anthropic-API server designed to run Claude Code on-device without proxies.
• An easy-to-use desktop application to download and run local and cloud LLMs.
• A simple way to get up and running with open models locally via CLI or REST API.

Project comparisons

• jundot/omlx vs Run Claude Code locally on Apple Silicon
• jundot/omlx vs Desktop application for local LLMs
• jundot/omlx vs Run and manage large language models locally

Evidence and sources

• GitHub project description: LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
• README: making local LLMs practical for real coding work with tools like Claude Code.
• README: Continuous batching and tiered KV caching, managed directly from your menu bar.
• README: Cold tier (SSD): When the hot cache fills up, blocks are offloaded to SSD in safetensors format. On the next request with a matching prefix, they're restored from disk instead of…
• README: Handles concurrent requests through mlx-lm's BatchGenerator. Max concurrent requests is configurable via CLI or admin panel.

Reviewed at: 2026-07-17T13:36:26Z

Open original GitHub project