cloneofsimo/lora is a Python library and CLI toolkit that enables Low-Rank Adaptation fine-tuning for Stable Diffusion models. It produces small, shareable weights (1MB-6MB) and trains twice as fast as standard Dreambooth methods.
Project overview
By applying Low-Rank Adaptation to diffusion models, this project addresses the computational and storage bottlenecks of customization, delivering tiny, easily mergeable weight files without sacrificing training speed.
Project type
Model Development · Image & Vision · Infrastructure
Deployment
Refer to project documentation
License
Apache-2.0
Best for
AI engineers and researchers who need to programmatically inject LoRA parameters into PyTorch models or fine-tune Stable Diffusion models via CLI.
Key capabilities
Fine-tune Stable Diffusion models using Low-Rank Adaptation (LoRA) with Dreambooth, Textual Inversion, or Pivotal Tuning, achieving faster speeds than the dreambooth method.
Merge and interpolate trained LoRA models together or with base checkpoints, including conversion to CKPT format, using the lora_add CLI.
Provides an out-of-the-box multi-vector pivotal tuning inversion CLI (lora_pti) for training with face segmentation and textual inversion.
Distill a fully trained model into a LoRA model using SVD distillation via CLI.
A Python library API (inject_trainable_lora) to programmatically inject and extract trainable LoRA parameters into PyTorch models like UNets.
Support for applying Kronecker product adaptations, similar to LoRA, as a future enhancement.
Support for time-aware fine-tuning as a future enhancement.
Limitations and risks
The file format used by this repository is different from the one currently integrated into the Huggingface diffusers library.
Setting the LoRA alpha value too high may cause overfitting, requiring manual tuning to avoid degraded outputs.
Getting started
Install the package via pip, prepare your dataset, and execute the lora_pti CLI with arguments. Setup requires understanding of fine-tuning configurations and downloading base models.
README: However, the fine-tuning process is very slow, and it is not easy to find a good balance between the number of steps and the quality of the results. Also, the final results (fully…
README: Their model is nicely ported through Huggingface API, so this repo has built various fine-tuning methods around them.
README: - Fine-tune Stable diffusion models twice as fast than dreambooth method, by Low-rank Adaptation - Get insanely small end result (1MB ~ 6MB), easy to share and download.
GitHub project description: Using Low-rank adaptation to quickly fine-tune diffusion models.