Scaling Distributed Machine Learning with the Parameter Server

Abstract

We propose a parameter server framework for distributed machine learning problems. Both data and workloads are distributed over worker nodes, while the server nodes maintain globally shared parameters, represented as dense or sparse vectors and matrices. The framework manages asynchronous data communication between nodes, and supports flexible consistency models, elastic scalability, and continuous fault tolerance.To demonstrate the scalability of the proposed framework, we show experimental results on petabytes of real data with billions of examples and parameters on problems ranging from Sparse Logistic Regression to Latent Dirichlet Allocation and Distributed Sketching.

Distributed Training Large-Scale Deep Architectures

“…To effectively orchestrate multiple machines for a training task, the system must provide a way to manage the globally shared model parameters. The parameter server architecture, i.e., a cluster of machines to manage parameters, is widely-used to reduce I/O latency for handling parameter updates [35,36]. As shown in Figure 1, parameter servers maintain latest parameter values and serve all workers.…”

A Novel Approach for Processing Big Data

“…However, Hadoop provides superior performance than its in-memory counterparts discussed in [8] and [9]. There are other systems that are powerful and versatile with low level programming interfaces [10], [11]. The problem with them is that they are specific and cannot provide general high level programming interface, scheduling and other needed mechanisms.…”

Application of Artificial Neural Networks for Catalysis: A Review

“…As its name implies, machine learning is able to "learn" the highly complicated relationships between the independent and dependent variables via non-linear "black box" data processing. During the past decades, it has been widely used in many scientific and industrial areas, such as biology [7][8][9], medicine [10][11][12], energy [13][14][15][16][17][18][19], environment [20][21][22], engineering [23][24][25], and information technology (IT) [26,27]. These application studies indicate that machine learning techniques have dramatically boosted the development of many different areas.…”