Open-source diffusion models for text-to-video and image-to-video generation. Requires GPU hardware for inference, with support for running on lower-memory devices via quantized methods.
Project overview
Open-source diffusion models for text-to-video and image-to-video generation. Requires GPU hardware for inference, with support for running on lower-memory devices via quantized methods.
Project type
Model Runtime · Image & Vision · Video
Use cases
Image Processing
Deployment
Refer to project documentation
License
Apache-2.0
Best for
Developers and researchers who need open-source text-to-video and image-to-video models and have access to GPU hardware.
Key capabilities
Generates videos from text prompts using diffusion models.
Takes an image as background input and generates a video combined with prompt words.
Continues an existing video to extend its length.
Provides fine-tuning schemes for the diffusers version of the CogVideoX model.
A simple Gradio web application demonstrating video generation.
Limitations and risks
Requires a GPU with significant memory for inference.
Getting started
Install Python 3.10 to 3.12, pip install dependencies from requirements.txt, and run cli_demo.py. The setup requires a GPU.
Evidence and sources
GitHub project description: text and image to video generation: CogVideoX (2024) and CogVideo (ICLR 2023)
README: Experience the CogVideoX-5B model online at
README: You can run **CogVideoX-2B** on older GPUs like `GTX 1080TI`, and **CogVideoX-5B** on desktop GPUs like `RTX 3060`.
README: diffusers INT8 (torchao): 3.6GB minimum*
README: Contains the inference code and fine-tuning code of SAT weights.