Verified project record

lllyasviel/ControlNet

lllyasviel/controlnet is a neural network structure for controlling text-to-image diffusion models by adding extra spatial conditions. It learns conditional control on small datasets using zero convolution layers, without destroying the original production-ready base model.

Project overview

By introducing a locked copy and a trainable copy connected through zero-initialized convolutions, this project makes it possible to add spatial conditioning to Stable Diffusion without degrading the base model. The variety of supported control types, combined with the ability to train on custom data, makes it a practical tool for controllable image generation.

Project type: Model Development · Image & Vision · Infrastructure
Use cases: Image Processing
Deployment: Refer to project documentation
License: Apache-2.0

Best for

AI engineers and researchers who need precise spatial control over Stable Diffusion text-to-image generation while preserving base model quality.
Creators looking to guide image outputs using conditions such as edges, depth, poses, or scribbles.

Key capabilities

A neural network structure to control diffusion models by adding extra conditions via a locked copy and a trainable copy with 1×1 zero convolutions, preventing the destruction of production-ready models during small dataset training.
Provides Stable Diffusion 1.5 + ControlNet using simple Canny edge detection.
Provides Stable Diffusion 1.5 + ControlNet using soft HED Boundary, suitable for recoloring and stylizing.
Provides Stable Diffusion 1.5 + ControlNet using a 512×512 depth map, preserving more details than 64×64 alternatives.
Provides Stable Diffusion 1.5 + ControlNet using normal maps, better at preserving geometry than depth maps.
Provides Stable Diffusion 1.5 + ControlNet using human pose via Openpose detection.
Provides Stable Diffusion 1.5 + ControlNet using semantic segmentation via the ADE20K protocol and Uniformer model.
Provides Stable Diffusion 1.5 + ControlNet using user scribbles via Gradio UI.

Limitations and risks

The anime line drawing model is not available right now pending risk evaluation.
Potential risks exist for the anime line drawing model requiring evaluation before release.

Getting started

Create and activate a conda environment using environment.yaml, then download the required models from the Hugging Face page and place them in the correct directories. Launch the Gradio interface by running python gradio_canny2image.py.

Evidence and sources

README: Official implementation of [Adding Conditional Control to Text-to-Image Diffusion Models](https://arxiv.org/abs/2302.05543). ControlNet is a neural network structure to control di…
README: title={Adding Conditional Control to Text-to-Image Diffusion Models}, author={Lvmin Zhang and Anyi Rao and Maneesh Agrawala}, booktitle={IEEE International Conference on Computer…
README: We provide 9 Gradio apps with these models.
README: Stable Diffusion 1.5 + ControlNet (using simple Canny edge detection) python gradio_canny2image.py
README: The soft HED Boundary will preserve many details in input images, making this app suitable for recoloring and stylizing. Just try it for more details.

Reviewed at: 2026-07-17T03:32:47Z