Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters

Abstract

Deep learning based models have dominated the current landscape of production recommender systems. Furthermore, recent years have witnessed an exponential growth of the model scale-from Google's 2016 model with 1 billion parameters to the latest Facebook's model with 12 trillion parameters. Significant quality boost has come with each jump of the model capacity, which makes us believe the era of 100 trillion parameters is around the corner. However, the training of such models is challenging even within industrial scale data centers. This difficulty is inherited from the staggering heterogeneity of the training computation-the model's embedding layer could include more than 99.99% of the total model size, which is extremely memory-intensive; while the rest neural network is increasingly computation-intensive. To support the training of such huge models, an efficient distributed training system is in urgent need. In this paper, we resolve this challenge by careful co-design of both the optimization algorithm and the distributed system architecture. Specifically, in order to ensure both the training efficiency and the training accuracy, we design a novel hybrid training algorithm, where the embedding layer and the dense neural network are handled by different synchronization mechanisms; then we build a system called Persia (short for parallel recommendation training system with hybrid acceleration) to support this hybrid training algorithm. Both theoretical demonstrations and empirical studies up to 100 trillion parameters have been conducted to justified the system design and implementation of Persia. We make Persia publicly available (at https://github.com/PersiaML/Persia) so that anyone would be able to easily train a recommender model at the scale of 100 trillion parameters.

AutoShard: Automated Embedding Table Sharding for Recommender Systems

“…Embedding tables are commonly used to deal with sparse features in recommendation models [1,2,3,4,5,29,30,31]. However, the extremely large embedding tables are often the storage and efficiency bottlenecks [6,7,8,3,6,9,10,11,32]. To our knowledge, the only two studies that target the embedding table placement problem are RecShard [27] and our previous work AutoShard [33].…”

BagPipe: Accelerating Deep Recommendation Model Training

“…We on the other hand implement only synchronous training. A very recent work by [27] introduces a new hybrid sync-async algorithm to train recommender models, unlike this work we only focus on synchronous training. [47] proposes methods for improving data processing while training recommender systems.…”

Compact lensless optoelectronic convolutional neural network for image classification

“…Among DNNs, convolutional neural networks (CNNs), one of the representative algorithms of deep learning, are a specialized kind of feedforward neural network with deep structure and convolution computation, and have been tremendously successful in computer vision applications such image recognition and image segmentation because of its smart use of strategies including sparse interactions, parameter sharing and equivariant representations [5,6]. Currently, deep CNNs are driven by high-performance processors such as graphics processing unit (GPU) and tensor processing unit (TPU) for performing a large number of computations such as addition and multiplication [7], which need huge computation time and energy resources. However, as the Moore's law approaches the limits of physics, electronic chips will be hard to keep up with performance growth of the artificial intelligence.…”