kvcache-ai/ktransformers is a framework for CPU-GPU heterogeneous computing designed to run large language models, with a focus on Mixture-of-Experts (MoE) models. It provides inference and fine-tuning capabilities to help mitigate GPU memory requirements.
Project overview
The framework uses kt-kernel serving to optimize CPU and GPU usage for MoE models via AMX/AVX acceleration and NUMA-aware memory management, reporting faster speeds and lower memory usage for fine-tuning.
Project type
Model Development · Model Runtime · Infrastructure
Use cases
Chat Assistants
Deployment
Refer to project documentation
License
Apache-2.0
Best for
Developers, AI engineers, and researchers who need to run inference or fine-tuning on large MoE models and want to utilize CPU/GPU hybrid computing to mitigate GPU memory limitations.
Key capabilities
Provides kt-kernel serving that optimizes CPU and GPU usage for MoE models using AMX/AVX acceleration and NUMA-aware memory management.
Enables fine-tuning of large MoE models with CPU/GPU hybrid support via LLaMA-Factory integration to achieve faster speeds and lower memory usage.
Limitations and risks
The original integrated KTransformers framework has been archived to the archive/ directory.
Operation requires specific GPU memory or large system memory, such as 24GB VRAM or 382G DRAM.
Data boundary, cost dependencies, and telemetry are not documented. A local installation does not prove a local-only data boundary or the absence of external network interactions.
Getting started
Setup requires a Python environment and familiarity with command-line execution and model weights management. Users navigate to the kt-kernel directory, install the package via pip, execute an SGLang or Python API serving script, and send inference queries via the API.
Alternatives and comparisons
Provides LLM inference via a plain C/C++ implementation without dependencies, supporting extensive hardware acceleration and integer quantization.
An open-source community ecosystem aggregating Chinese Llama models, fine-tuning and inference code, datasets, and tutorials.
Enables text generation, chat, quantization, and fine-tuning of Hugging Face LLMs natively on Apple silicon with MLX.
GitHub project description: A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations
README: KTransformers is a research project focused on efficient inference and fine-tuning of large language models through CPU-GPU heterogeneous computing.
README: Support Deepseek-R1 and V3 on single (24GB VRAM)/multi gpu and 382G DRAM, up to 3~28x speedup.