petals enables distributed inference and fine-tuning of large language models over the internet by loading model layers cooperatively across a decentralized swarm of participants. It addresses the problem that running very large language models locally is too slow or impossible due to memory constraints.
Project overview
Its BitTorrent-style collaborative approach to model inference and fine-tuning lets users run models such as Llama 3.1 405B by pooling GPU resources across a decentralized network, which directly confronts the local memory barrier for very large models.
Project type
Model Development · Model Runtime · Infrastructure
Deployment
Refer to project documentation
License
MIT
Best for
Developers and AI engineers who need to run very large language models that exceed local memory and are comfortable with Python library usage and coding.
Researchers who want to fine-tune large language models collaboratively across a decentralized network.
Key capabilities
Generates text using large models cooperatively loaded across a decentralized network.
Allows users to connect their GPUs and serve model layers to the public swarm to increase system capacity.
Allows users to set up a private, trusted swarm for processing sensitive data.
Limitations and risks
Data processed in the public swarm relies on other unknown participants. Users with sensitive data should set up a private, trusted swarm.
Getting started
Setup requires a Python environment and coding knowledge to use the library. The path to first success is to install the library, connect to the swarm, and generate text.
Evidence and sources
GitHub project description: 🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
README: from petals import AutoDistributedModelForCausalLM
README: Run large language models at home, BitTorrent-style.
README: Petals: Collaborative Inference and Fine-tuning of Large Models.
Release: v2.2.0: Falcon, macOS support, and more
AI Search
Find projects, verify facts, compare options, or turn a complex need into an actionable plan
Try a searchA click only fills the search box; you stay in control