nano-vllm is a lightweight Python library that reimplements vLLM's offline LLM inference in roughly 1,200 lines of code. It provides a code-based interface for running locally downloaded Hugging Face models with optimizations including prefix caching and tensor parallelism.
Project overview
Offers a minimal, readable codebase for developers who want to understand or run optimized offline LLM inference comparable to vLLM.
Project type
Model Runtime
Use cases
Chat Assistants
Deployment
Refer to project documentation
License
MIT
Best for
Developers and AI engineers who interact via Python code, want to study a readable vLLM implementation, and have locally downloaded Hugging Face weights ready.
Key capabilities
Provides offline LLM inference speeds comparable to the original vLLM engine.
Includes optimizations such as prefix caching, tensor parallelism, torch compilation, and CUDA graph support.
Mirrors vLLM's interface with minor differences in the LLM.generate method to facilitate adoption.
Limitations and risks
Requires writing Python code to operate and lacks a standalone CLI, web interface, or server executable.
Designed specifically for offline inference and lacks built-in network serving capabilities.
Getting started
Installation is considered medium difficulty because it requires setting up a Python environment, manually downloading multi-gigabyte model weights, and writing Python code to execute inference.
Install the package via pip from GitHub, download model weights such as Qwen3-0.6B using huggingface-cli to a local directory, and call LLM.generate() from the example script with the model path.
Alternatives and comparisons
Provides a CLI and REST API for running and managing local large language models, contrasting with nano-vllm's Python library approach.
A C/C++ implementation offering a CLI tool and model quantization for LLM inference across diverse hardware, contrasting with nano-vllm's Python-based library.
A self-hosted web interface for interacting with local LLM runners and OpenAI-compatible APIs, suited for users needing a GUI rather than a Python library.