DeepSpeed is a PyTorch-integrated library that provides system innovations such as ZeRO, ZeRO-Infinity, and 3D-Parallelism to make large-scale deep learning training and inference efficient. It is designed for developers, researchers, and AI engineers who need to scale models across distributed compute resources.
Project overview
It delivers a combined set of system optimizations specifically aimed at making large-scale deep learning compute effective and efficient.
Project type
Model Development · Data Processing · Infrastructure
Deployment
Refer to project documentation
License
Apache-2.0
Best for
Developers, researchers, and AI engineers working with PyTorch who need to scale large deep learning models.
Teams requiring specific distributed training optimizations such as ZeRO, ZeRO-Infinity, 3D-Parallelism, and Ulysses Sequence Parallelism.
Key capabilities
Provides system innovations like ZeRO, ZeRO-Infinity, 3D-Parallelism, and Ulysses Sequence Parallelism to make large scale DL training effective and efficient.
Includes C++/CUDA extensions (ops) that are built just-in-time (JIT) at runtime using torch's JIT C++ extension loader.
Supports many features for training and inference on Windows.
Provides a tool to validate installation and check machine compatibility with extensions/ops.
Limitations and risks
Some features like async io (AIO) and GDS are not supported on Windows.
Building C++/CUDA extensions just-in-time (JIT) requires a compatible compiler and might fail if the environment is not correctly set up.
Getting started
Setup involves installing PyTorch first, followed by installing DeepSpeed via pip. The process has a medium difficulty because it requires PyTorch and a CUDA or ROCm compiler to be installed beforehand, and involves JIT compilation of extensions.
Evidence and sources
GitHub project description: DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
README: DeepSpeed offers a confluence of system innovations, that has made large scale DL training effective, and efficient, greatly improved ease of use, and redefined the DL training la…
README: DeepSpeed has been used to train many different large-scale models, below is a list of several examples that we are aware of
README: DeepSpeed includes several C++/CUDA extensions that we commonly refer to as our 'ops'. By default, all of these extensions/ops will be built just-in-time (JIT) using torch's JIT C…
README: Many DeepSpeed features are supported on Windows for both training and inference. You can read more about this in the original blog post here. Among features that are currently no…