Petals lets you run large language models at home using a BitTorrent-style distributed approach. Load a portion of a model, then join a network of users serving the remaining parts — enabling inference on consumer GPUs or Google Colab.
Supported models include Llama 3.1 (405B), Mixtral (8x22B), Falcon (180B), and BLOOM (176B). Single-batch inference reaches up to 6 tokens/sec for Llama 2 (70B) and 4 tokens/sec for Falcon (180B), sufficient for chatbots and interactive applications.
Beyond standard LLM APIs, Petals offers full PyTorch and Transformers flexibility: apply any fine-tuning or sampling method, execute custom model paths, and inspect hidden states. You get API convenience with low-level control.
The project is part of the Big Science research workshop. Follow development via Discord or occasional email updates. Top contributors are listed on the project page.
Source: Hacker News · Summarized by HeadlinesBriefing