Verified project record

InternLM/lmdeploy

lmdeploy is a library for compressing, deploying, and serving large language models (LLMs) and vision-language models (VLMs). It provides high-throughput inference through persistent batching, tensor parallelism, and quantization.

Project overview

lmdeploy achieves up to 1.8x higher request throughput compared to vLLM while offering both a TurboMind engine for ultimate performance and a pure-Python PyTorch engine to lower the barrier for developers.

Project type: Model Runtime · Infrastructure
Use cases: Chat Assistants
Deployment: Refer to project documentation
License: Apache-2.0

Best for

AI engineers, developers, and operations teams who need efficient, high-throughput local deployment and API serving for large language models and vision-language models.

Key capabilities

Delivers high-throughput LLM inference via TurboMind for ultimate performance and PyTorch for developer accessibility, utilizing persistent batching and tensor parallelism.
Supports weight-only and k/v quantization, including 4-bit AWQ to speed up inference.
Provides a Python pipeline API to easily load and run offline batch inference for LLMs and VLMs.
Supports inference and serving for Vision-Language Models (VLMs) such as InternVL, LLaVA, and Qwen-VL.

Getting started

Installable directly via pip in a conda environment. To start, run: conda create -n lmdeploy python=3.12 -y, then conda activate lmdeploy, and finally pip install lmdeploy.

Evidence and sources

GitHub project description: LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
README: - \[2024/01\] Support [PyTorch inference engine](./docs/en/inference/pytorch.md), developed entirely in Python, helping to lower the barriers for developers and enable rapid exper…
README: LMDeploy delivers up to 1.8x higher request throughput than vLLM, by introducing key features like persistent batch(a.k.a. continuous batching), blocked KV cache, dynamic split&fu…
README: LMDeploy has developed two inference engines - [TurboMind](./docs/en/inference/turbomind.md) and [PyTorch](./docs/en/inference/pytorch.md), each with a different focus.
README: By default, LMDeploy downloads model from HuggingFace. If you would like to use models from ModelScope, please install ModelScope by `pip install modelscope` and set the environme…

Reviewed at: 2026-07-16T12:40:36Z