Verified project record

EricLBuehler/mistral.rs

mistral.rs is a fast and flexible LLM inference engine that supports various models, quantization formats, and hardware with zero configuration. It provides true multimodality, OpenAI and Anthropic compatible serving, and built-in agentic support.

Project overview

It stands out by combining smart in-situ quantization of any Hugging Face model with a unified server that handles text, image, audio, and video inputs, all runnable locally through straightforward deployment options.

Project type: AI Agent · Model Runtime
Use cases: Chat Assistants · Automation
Deployment: Refer to project documentation
License: MIT

Best for

Developers who need to run various models and quantization formats locally or in self-hosted environments with minimal setup.
Researchers exploring local multimodal AI agent automation and benchmarking.

Key capabilities

Exposes OpenAI-compatible /v1 endpoints and Anthropic-compatible /v1/messages endpoints for API compatibility.
Connect to external tools via MCP over Process, HTTP, or WebSocket for agentic workflows.
Includes a 'bench' command to measure performance locally.
Enables in-process inference from Python to load a model and send OpenAI-shaped requests without a server.

Limitations and risks

Supported models are explicitly dictated by the engine's loader registry, restricting use to recognized formats and architectures.

Getting started

Run 'mistralrs run -m user/model' using prebuilt binaries provided by an installer script, or install the Python SDK via pip.

Evidence and sources

GitHub project description: Fast, flexible LLM inference
Release: v0.9.0
README: Anthropic Messages API: `mistralrs serve` now exposes Anthropic-compatible `/v1/messages` and `/v1/messages/count_tokens` endpoints alongside the OpenAI-compatible `/v1` API.
README: v0.8.2 CUDA performance: CUDA graphs, FlashInfer paged kernels, and MoE optimizations deliver strong results on GB10, B200, and H100 SXM.
README: Agentic runtime: web search, local Python code execution, shell execution, OpenAI-compatible Skills, session management, and custom tool hooks.

Reviewed at: 2026-07-18T06:14:48Z