NuPS: A Parameter Server for Machine Learning with Non-Uniform Parameter Access

Good Intentions: Adaptive Parameter Servers via Intent Signaling

“…This approach is popular for models with dense parameter access [28,31,45]. For sparse workloads, however, it is inefficient because it overcommunicates: it constantly synchronizes updates for all parameters to all nodes, although at each point in time, each node accesses only a small subset of these parameters [41]. A classic PS partitions the parameters among the nodes, and provides global reads and writes to these parameters by transparently communicating with the node that holds the accessed parameter.…”

“…In contrast to static full replication, a classic PS uses network bandwidth only when parameters are actually accessed. However, a classic PS is often inefficient due to access latency [41,42]. Figure 1 depicts the performance of both approaches for a task of training large-scale knowledge graph embeddings.…”

“…The Lapse PS [42] dynamically relocates parameters among nodes to hide access latency. Multi-technique PSs [41,57] combine different parameter management techniques (e.g., replication and relocation) and pick a suitable one for each parameter.…”

“…In multi-technique PSs, applications need to specify upfront which technique to use for which parameter, and-for optimal performance-need to tune these choices. Figure 1 depicts the performance of multiple configurations of NuPS [41], a multi-technique PS: NuPS can be more efficient than static approaches, but requires tuning.…”

“…Parameter access is fast, as every worker can access every parameter locally, without synchronous network communication. However, full replication is communication-inefficient for sparse workloads [41], as it maintains the replicas of all parameters on all nodes throughout the training task, even though each node accesses only a small subset of these replicas at each point in time. Also, full replication limits model size to the memory capacity of a single node.…”