controlnet is a neural network structure that adds spatial conditional control to text-to-image diffusion models without destroying the production-ready base model. It copies weights into a locked copy and a trainable copy connected by zero convolution layers, enabling conditional control training on small datasets.
Project overview
Its zero convolution approach prevents destructive interference with the original production-ready model during training, directly addressing the challenge of adding conditional controls to diffusion models while preserving base model quality.
Project type
Model Development · Image & Vision · Infrastructure
Use cases
Image Processing
Deployment
Refer to project documentation
License
Apache-2.0
Best for
AI engineers and researchers who need to add spatial conditional controls to Stable Diffusion 1.5 while preserving the base model.
Creators who want to guide image generation using structural inputs such as edge maps, depth maps, poses, or scribbles.
Key capabilities
A neural network structure to control diffusion models by adding extra conditions via a locked copy and a trainable copy with 1×1 zero convolutions, preventing destruction of production-ready models during small dataset training.
Provides Stable Diffusion 1.5 + ControlNet using soft HED Boundary, suitable for recoloring and stylizing.
Provides Stable Diffusion 1.5 + ControlNet using a 512×512 depth map, preserving more details than 64×64 alternatives.
Provides Stable Diffusion 1.5 + ControlNet using normal maps, better at preserving geometry than depth maps.
Provides Stable Diffusion 1.5 + ControlNet using human pose via Openpose detection.
Provides Stable Diffusion 1.5 + ControlNet using semantic segmentation via the ADE20K protocol and Uniformer model.
Provides Stable Diffusion 1.5 + ControlNet using user scribbles via Gradio UI.
Limitations and risks
The anime line drawing model is not available right now pending risk evaluation.
Potential risks exist for the anime line drawing model requiring evaluation before release. Users should verify availability and risk status before relying on this model.
Getting started
Create a conda environment using the provided environment.yaml file, activate it, manually download the required models from the Hugging Face page into specific directories, then run a Gradio script such as gradio_canny2image.py.
Evidence and sources
README: Official implementation of [Adding Conditional Control to Text-to-Image Diffusion Models](https://arxiv.org/abs/2302.05543). ControlNet is a neural network structure to control di…
README: title={Adding Conditional Control to Text-to-Image Diffusion Models}, author={Lvmin Zhang and Anyi Rao and Maneesh Agrawala}, booktitle={IEEE International Conference on Computer…
README: We provide 9 Gradio apps with these models.
README: The soft HED Boundary will preserve many details in input images, making this app suitable for recoloring and stylizing. Just try it for more details.