kvcache-ai/ktransformers

Project overview

ktransformers enables high-performance CPU-GPU heterogeneous computing for large language models, including MoE architectures, reducing hardware requirements for inference and fine-tuning. It provides kt-kernel serving with AMX/AVX acceleration and NUMA-aware memory management, and supports fine-tuning via LLaMA-Factory integration.

Project type

Model Development · Model Runtime · Infrastructure

Use cases

Chat Assistants

Deployment

Refer to project documentation

License

Apache-2.0

Best for

• Developers and researchers needing to run inference or fine-tuning on large MoE models using limited or heterogeneous GPU and CPU hardware configurations.

Key capabilities

• Provides high-performance kt-kernel serving that optimizes CPU and GPU usage for MoE models using AMX/AVX acceleration and NUMA-aware memory management.
• Enables fine-tuning of ultra-large MoE models with CPU/GPU hybrid support via LLaMA-Factory integration, achieving faster speeds and lower memory usage.

Limitations and risks

• The original integrated KTransformers framework has been archived to the archive/ directory.

Getting started

• Setup requires a Python environment and familiarity with command-line execution and model weights management. The first success path involves navigating to the kt-kernel directory, installing via pip, executing an SGLang or Python API serving script, and sending an inference query via the API.

Evidence and sources

• GitHub project description: A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations
• README: KTransformers is a research project focused on efficient inference and fine-tuning of large language models through CPU-GPU heterogeneous computing.
• README: Support Deepseek-R1 and V3 on single (24GB VRAM)/multi gpu and 382G DRAM, up to 3~28x speedup.
• README: CPU-optimized kernel operations for heterogeneous LLM inference. <img width="1049" height="593" alt="image" src="https://github.com/user-attachments/assets/68f423da-3f55-4025-bdc9…
• README: KTransformers × LLaMA-Factory integration for ultra-large MoE model fine-tuning.

Reviewed at: 2026-07-19T18:47:08Z