tensorrt-llm is a library for optimizing large language model and visual generation model inference to run efficiently on NVIDIA GPUs. It provides a Python API with specialized kernels, runtime optimizations, and a modular PyTorch-native architecture for defining and executing LLMs.
Project overview
The project combines custom kernels for common inference operations such as attention, GEMMs, and MoE with algorithmic runtime optimizations including prefill-decode disaggregation and speculative decoding, targeting efficient LLM and visual generation model inference on NVIDIA GPUs.
Project type
Model Runtime
Deployment
Refer to project documentation
License
License pending
Best for
AI engineers who need to optimize LLM or visual generation model inference on NVIDIA GPUs using a PyTorch-native API with specialized kernels and runtime optimizations.
Developers who require Python and C++ runtime components to orchestrate inference execution for large language models.
Key capabilities
A high-level Python API to define large language models and customize or extend the system using native PyTorch code.
Components to create Python and C++ runtimes that orchestrate inference execution in a performant way.
Custom kernels for common inference operations including attention, GEMMs, and MoE.
Includes prefill-decode disaggregation, wide expert parallelism, and speculative decoding.
Support for various parallelism strategies spanning single-GPU to multi-node deployments.
Supports diffusion models for visual generation.
Limitations and risks
Efficient inference execution requires NVIDIA GPUs; non-NVIDIA hardware platforms are not supported.
APIs and tools may be deprecated and removed after a 3-month migration period, which should be factored into adoption planning.
Getting started
Setup difficulty is medium because installation requires building the Python package from source or using containers. The first success path is to install from source or use pre-compiled wheels or containers.