horovod is a distributed deep learning training framework that scales single-GPU training scripts across multiple GPUs or hosts with minimal code changes using MPI and NCCL.
Project overview
horovod leverages the MPI model to simplify distributed training code while achieving high scaling efficiency, reaching 90% scaling efficiency for Inception V3 and ResNet-101 on large clusters.
Project type
Model Development · Infrastructure
Deployment
Refer to project documentation
License
License pending
Best for
Developers and AI engineers who need to scale existing single-GPU deep learning training scripts across multiple GPUs or hosts.
Key capabilities
Scales single-GPU training scripts across multiple GPUs or hosts with minimal code changes using MPI and NCCL.
Batches small allreduce operations to interleave communication and computation for improved performance.
Records a timeline of Horovod activity to analyze performance.
Automates the optimization of parameters like Tensor Fusion via a single command line argument.
Allows concurrently running distinct collective operations in different groups of processes.
Limitations and risks
Installing horovod requires CMake and specific versions of g++ or Conda packages.
Users must modify existing single-GPU training scripts to add distributed optimizers and initializations.
Installing MPI and NCCL can be an extra hassle requiring infrastructure team involvement, presenting an adoption check for teams without dedicated infrastructure support.
Getting started
Setup difficulty is medium. Install CMake and g++, then run pip install horovod. Modify your training script to initialize Horovod with hvd.init(), add a Distributed Optimizer, and execute the script via horovodrun.
Evidence and sources
GitHub project description: Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.
README: Horovod is a distributed deep learning training framework for TensorFlow, Keras, PyTorch, and Apache MXNet. The goal of Horovod is to make distributed deep learning fast and easy…
README: The primary motivation for this project is to make it easy to take a single-GPU training script and successfully scale it to train across many GPUs in parallel.
README: opt = hvd.DistributedOptimizer(opt)
README: To run on CPUs: .. code-block:: bash $ pip install horovod To run on GPUs with NCCL: .. code-block:: bash $ HOROVOD_GPU_OPERATIONS=NCCL pip install horovod