Understanding Capacity-Driven Scale-Out Neural Recommendation Inference

Abstract

Deep learning recommendation models have grown to the terabyte scale. Traditional serving schemes-that load entire models to a single server-are unable to support this scale. One approach to support this scale is with distributed serving, or distributed inference, which divides the memory requirements of a single large model across multiple servers.This work is a first-step for the systems research community to develop novel model-serving solutions, given the huge system design space. Large-scale deep recommender systems are a novel workload and vital to study, as they consume up to 79% of all inference cycles in the data center. To that end, this work describes and characterizes scale-out deep learning recommendation inference using data-center serving infrastructure. This work specifically explores latency-bounded inference systems, compared to the throughput-oriented training systems of other recent works. We find that the latency and compute overheads of distributed inference are largely a result of a model's static embedding table distribution and sparsity of input inference requests. We further evaluate three embedding table mapping strategies of three DLRM-like models and specify challenging design trade-offs in terms of end-to-end latency, compute overhead, and resource efficiency. Overall, we observe only a marginal latency overhead when the data-center scale recommendation models are served with the distributed inference manner-P99 latency is increased by only 1% in the best case configuration. The latency overheads are largely a result of the commodity infrastructure used and the sparsity of embedding tables. Even more encouragingly, we also show how distributed inference can account for efficiency improvements in data-center scale recommendation serving.

RecShard: statistical feature-based memory optimization for industry-scale neural recommendation

“…Baselines: To evaluate the efficacy of RecShard, we compare the performance of EMB operators under RecShard's throughput optimized sharding strategy with sharding schemes from prior work on production DLRM training systems [1,26,31]. State-of-the art sharding schemes typically follow a two-step approach.…”

“…• Size [1,26]: An EMB's cost is the product of its hash size and its embedding dimension (latent vector length). • Lookup [1,26]: An EMB's cost is the product of its average pooling factor and its embedding dimension.…”

“…• Size [1,26]: An EMB's cost is the product of its hash size and its embedding dimension (latent vector length). • Lookup [1,26]: An EMB's cost is the product of its average pooling factor and its embedding dimension. • Size-and-Lookup: An EMB's cost is the product of its lookup based cost (above) and the log of its hash size -𝑙𝑜𝑔 10 (ℎ𝑎𝑠ℎ_𝑠𝑖𝑧𝑒 𝐸𝑀𝐵 ) -adding a non-linear function that attemps to capture potential caching effects of smaller EMBs.…”

“…Figure 1a shows that between 2017-2021, the memory capacity requirements of DLRMs have grown by 16 times. EMB memory footprints are on the order of terabytes (TB) [26,46] and account for over 99% of the total model capacity [11]. The growth in the number and sizes of EMBs stems from the increase in the number of features and feature categories represented, in order to improve the overall DLRM prediction quality.…”

“…DLRMs DLRMs exhibit distinct systems implications compared to more traditional neural network architectures [10,16,23,24,38]. This is due to their use of embedding layers which demand ordersof-magnitude higher memory capacity and exhibit significantly lower compute-intensity [11,26,33]. Embedding layers, comprised of embedding tables (EMBs), support the transformation of categorical (i.e., sparse) features into dense representations.…”