arcee-ai/mergekit

Project overview

By combining model weights directly, mergekit avoids the compute overhead of ensembling or retraining while preserving the inference cost of a single model and operating within severe hardware constraints.

Project type

Model Development · Model Runtime

Deployment

Refer to project documentation

License

LGPL-3.0

Best for

• AI engineers and researchers who need to combine multiple pretrained language models without the compute cost of ensembling or additional training.
• Practitioners working with limited hardware, including CPU-only setups or GPUs with as little as 8 GB of VRAM.

Key capabilities

• Provides a suite of merging algorithms including Linear, SLERP, TIES, DARE, and more to combine pretrained language models directly in weight space.
• Uses an out-of-core approach with lazy loading of tensors to perform elaborate merges in resource-constrained situations, running on CPU or accelerated with as little as 8 GB of VRAM.
• Allows extracting PEFT-compatible low-rank approximations of finetuned models.
• Supports merging multiple dense models into a mixture of experts, either for direct use or for further training.
• Enables the execution of complex, multi-stage model merging workflows where later merges can use the outputs of earlier ones as inputs.
• Applies merging algorithms directly to raw .pt or .safetensors PyTorch checkpoints, supporting arbitrary PyTorch models outside of Hugging Face Transformers.
• Transplants tokenizers between models, aligning the vocabulary of one model with another, useful for cheaply producing draft models for speculative decoding or cross-tokenizer knowledge distillation.
• Provides fine-grained control over vocabulary, embeddings, and chat templates for merged models.

Limitations and risks

• When applying merging algorithms directly to raw .pt or .safetensors PyTorch checkpoints, layer slicing and tokenizer configuration are not supported.

Getting started

• Clone the repository, install the package locally with pip, create a YAML configuration file defining the merge operations, and execute the merge using the provided CLI. Steps: git clone https://github.com/arcee-ai/mergekit.git && cd mergekit && pip install -e . && mergekit-yaml path/to/your/config.yml ./output-model-directory.

Evidence and sources

• GitHub project description: Tools for merging pretrained large language models.
• README: `mergekit` uses an out-of-core approach to perform unreasonably elaborate merges in resource-constrained situations. Merges can be run entirely on CPU or accelerated with as littl…
• README: Model merging is a powerful technique that allows combining the strengths of different models without the computational overhead of ensembling or the need for additional training.
• README: merged models maintain the same inference cost as a single model while often achieving comparable or superior performance.
• README: If you find `mergekit` useful in your research, please consider citing the [paper](https://aclanthology.org/2024.emnlp-industry.36/)

Reviewed at: 2026-07-18T06:31:36Z