dreambooth-stable-diffusion enables fine-tuning the entire Stable Diffusion model on a specific subject using the Dreambooth method with just a few images. It addresses the need to personalize text-to-image generation when the original Dreambooth model weights are unavailable.
Project overview
By adapting the Dreambooth method to the Stable Diffusion architecture, this project enables subject-level personalization of text-to-image models without requiring the original Imagen weights.
Project type
Model Development · Image & Vision
Deployment
Refer to project documentation
License
MIT
Best for
AI engineers and researchers who need to personalize Stable Diffusion for a specific subject using a small image set.
Key capabilities
Trains and fine-tunes the whole Stable Diffusion Unet model using user-provided images and regularization images to learn a specific subject.
Generates personalized images using the fine-tuned checkpoint by running a text-to-image script with a subject identifier prompt.
Reduces GPU memory usage during training by turning on the gradient checkpointing trick for the Unet.
Limitations and risks
All training and generation workflows are executed through CLI scripts.
Getting started
Setup is hard: it requires manually setting up the ldm environment, sourcing pre-trained Stable Diffusion model weights, and executing training scripts via the command line. The first success path is to set up the ldm environment following the Textual Inversion or Stable Diffusion repository.
A GPU is required; the documented reference hardware is two A6000 GPUs.
Evidence and sources
README: To enable people to fine-tune a text-to-image model with a few examples, I implemented the idea of Dreambooth on Stable diffusion. This code repository is based on that of Textual…
README: I just found a way to reduce the GPU memory a bit.
README: First set-up the ```ldm``` enviroment following the instruction from textual inversion repo, or the original Stable Diffusion repo.
README: I train the model use two A6000 GPUs and it takes ~15 mins.
README: After training, personalized samples can be obtained by running the command python scripts/stable_txt2img.py --ddim_eta 0.0 --n_samples 8 --n_iter 1 --scale 10.0 --ddim_steps 100…