Understanding data storage and ingestion for large-scale deep recommendation model training

Abstract

Datacenter-scale AI training clusters consisting of thousands of domain-specific accelerators (DSA) are used to train increasinglycomplex deep learning models. These clusters rely on a data storage and ingestion (DSI) pipeline, responsible for storing exabytes of training data and serving it at tens of terabytes per second. As DSAs continue to push training efficiency and throughput, the DSI pipeline is becoming the dominating factor that constrains the overall training performance and capacity. Innovations that improve the efficiency and performance of DSI systems and hardware are urgent, demanding a deep understanding of DSI characteristics and infrastructure at scale. This paper presents Meta's end-to-end DSI pipeline, composed of a central data warehouse built on distributed storage and a Data PreProcessing Service that scales to eliminate data stalls. We characterize how hundreds of models are collaboratively trained across geo-distributed datacenters via diverse and continuous training jobs. These training jobs read and heavily filter massive and evolving datasets, resulting in popular features and samples used across training jobs. We measure the intense network, memory, and compute resources required by each training job to preprocess samples during training. Finally, we synthesize key takeaways based on our production infrastructure characterization. These include identifying hardware bottlenecks, discussing opportunities for heterogeneous DSI hardware, motivating research in datacenter scheduling and benchmark datasets, and assimilating lessons learned in optimizing DSI infrastructure. CCS CONCEPTS• Software and its engineering → Distributed systems organizing principles; • Information systems → Database management system engines; • Computing methodologies → Machine learning.

Amalur: Data Integration Meets Machine Learning

“…Given a model, its architecture and algebraic computations are fixed; we also know which LA operators are affected by factorization [37]. To examine the relative speedup of factorization, we mainly need to inspect data redundancy [35], [37], and the interactions between physical data transfers (e.g., network and memory bandwidth) [73]. Existing solutions.…”

The Difficult Balance Between Modern Hardware and Conventional CPUs

“…The current erbium implementation uses dictionary encoding to reduce both the storage requirement and the online data movement. Therefore, queries must be encoded before being sent to the accelerators, just like data quantisation and normalisation in machine learning pipelines [33]. This process is carried out individually at the worker level in a pipeline manner, while the previous query batch is being executed by the FPGA kernel.…”

“…This imbalance is pervasive and often quite large. For instance, a recent study by Meta of their ML pipelines [33] shows that GPUs used for training ML models are stalled up to 56 % of the time waiting for input data. They also show the increasing amount of compute power, network, and memory bandwidth needed on the CPU side to be able to match the throughput of the accelerator.…”

How Do Consumer Innovation Characteristics and Consumption Value Shape Users’ Willingness to Buy Innovative Car Safety Seats?

“…Thus, defined in the field of innovation performance more precisely forecasts the adoption of new products, leading to the development of innovation (domain-specific innovativeness) in specific areas, using innovative specific areas to forecast consumers' specific interest in the field of new products early adoption behavior and attitude [45]. Researchers worldwide have been adapting this model's structure to new fields and products in recent years [46][47][48][49][50][51]. However, many studies look at the product side of things, concentrating on the early degree of customers' adoption of new items while disregarding that some people only pay attention to the information about new products but do not necessarily buy them.…”