Verified project record

jundot/omlx

A macOS-native LLM inference server for Apple Silicon that keeps models in memory and manages KV cache across RAM and SSD through a menu bar app and web dashboard. Designed for developers and general users who need local, multi-model serving with OpenAI and Anthropic API compatibility.

Project overview

Project type: MCP · Model Runtime
Use cases: Chat Assistants · Coding & Development
Deployment: Refer to project documentation
License: Apache-2.0

Best for

Developers and AI engineers on Apple Silicon who need to manage multiple local models simultaneously with persistent KV caching.
General users on macOS who prefer starting, stopping, and monitoring local models through a native menubar application and web interface.

Key capabilities

Manages KV cache at the block level with prefix sharing and Copy-on-Write. Cold blocks are offloaded to SSD in safetensors format and restored on matching prefix, including after a server restart.
Processes concurrent requests via mlx-lm's BatchGenerator with configurable limits on maximum concurrency.
Serves LLMs, VLMs, embedding models, and rerankers at the same time. Includes LRU eviction, manual load and unload, model pinning, per-model TTL, and total process memory enforcement.
A native Swift and SwiftUI menubar application to start, stop, and monitor the server without terminal access. It maintains persistent stats and restarts the server automatically on crash.
A web interface at `/admin` for real-time monitoring, model management, chat, benchmarking, and per-model settings. It supports multiple languages and uses offline CDN dependencies.
Functions as a drop-in replacement for OpenAI and Anthropic APIs. Supports chat completions, text completions, embeddings, reranking, the messages API, streaming, and vision inputs.
Runs VLMs supporting multi-image chat, base64, URL, and file inputs, alongside tool calling. Automatically detects and optimizes OCR models such as DeepSeek-OCR, DOTS-OCR, and GLM-OCR.
Allows users to search and download MLX models from HuggingFace directly within the admin dashboard.

Limitations and risks

Requires an Apple Silicon Mac (M1/M2/M3/M4) running macOS 15.0 or later.
Building and utilizing native custom kernels requires the full version of Xcode to compile from source.
Performing a plain pip install without the custom kernels results in 30x slower fallback paths for GLM-5.2 models.

Getting started

Installation is categorized as easy due to a DMG graphical installer, Homebrew formula, and auto-update functionality. The first success path is installing via Homebrew, starting the service via command line, and opening the local admin dashboard URL.

Evidence and sources

GitHub project description: LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
README: making local LLMs practical for real coding work with tools like Claude Code.
README: Continuous batching and tiered KV caching, managed directly from your menu bar.
README: Cold tier (SSD): When the hot cache fills up, blocks are offloaded to SSD in safetensors format. On the next request with a matching prefix, they're restored from disk instead of…
README: Handles concurrent requests through mlx-lm's BatchGenerator. Max concurrent requests is configurable via CLI or admin panel.

Reviewed at: 2026-07-17T13:36:26Z