Petals: Collaborative Inference and Fine-tuning of Large Models

Abstract

Large language models (LLMs) are useful in many NLP tasks and become more capable with size, with the best open-source models having over 50 billion parameters. However, using these 50B+ models requires high-end hardware, making them inaccessible to most researchers. In this work, we investigate methods for cost-efficient inference and fine-tuning of LLMs, comparing local and distributed strategies. We observe that a large enough model (50B+) can run efficiently even on geodistributed devices in a consumer-grade network. This could allow running LLM efficiently by pooling together idle compute resources of multiple research groups and volunteers. We address two open problems: (1) how to perform inference and fine-tuning reliably if any device can disconnect abruptly and (2) how to partition LLMs between devices with uneven hardware, joining and leaving at will. In order to do that, we develop special fault-tolerant inference algorithms and load-balancing protocols that automatically assign devices to maximize the total system throughput. We showcase these algorithms in PETALS 1 -a decentralized system that runs Llama 2 (70B) and BLOOM (176B) over the Internet up to 10× faster than offloading for interactive generation. We evaluate the performance of our system in simulated conditions and a real-world setup spanning two continents.

Latency-aware service placement for GenAI at the edge

“…Borzunov et al 12 proposed an innovative approach to distribute the workload of LLMs across multiple servers. This method involves splitting the LLM model on consumer hardware and executing it in a distributed manner, incorporating dynamic quantization and server load balancing along with fault tolerance.…”

“…The selection of edge devices for services is based on the pheromone trail strength, which reflects the accumulated experience of previous iterations, and a heuristic factor to include domain-specific knowledge or performance metrics. This decision-making process is followed with an immediate update to the local pheromone value, following each service placement (lines [11][12][13][14][15]. This local update serves as a rapid feedback technique, guiding subsequent selections within the same cycle.…”

Google and DeepMind: Deep Learning Systems in Ophthalmology

“…The concept of local-SGD (or FedAvg) has previously been applied in the realm of language modeling. Crossdevice federated learning, for instance, has been utilized to pretrain and fine-tune language models (Borzunov et al, 2022;Diskin et al, 2021a;Hilmkil et al, 2021;Presser, 2020;Ro et al, 2022;Ryabinin et al, 2021). More recently, DiLoCo has extended the local-SGD methodology to encompass larger language models, specifically proposing the use of AdamW + Nesterov momentum as the InnerOpt + OuterOpt pairing.…”