The Role of Network Topology for Distributed Machine Learning

Abstract

Many learning problems are formulated as minimization of some loss function on a training set of examples. Distributed gradient methods on a cluster are often used for this purpose. In this paper, we study how the variability of task execution times at cluster nodes affects the system throughput. In particular, a simple but accurate model allows us to quantify how the time to solve the minimization problem depends on the network of information exchanges among the nodes. Interestingly, we show that, even when communication overhead may be neglected, the clique is not necessarily the most effective topology, as commonly assumed in previous works.

Federated Learning at the Network Edge: When Not All Nodes are Created Equal

“…Narrowing its focus to node capabilities, [5] aims at choosing the set of learning nodes that results in the shortest learning time, solving a double-edged conundrum. On the one hand, more nodes mean that convergence can be reached in fewer iterations; on the other hand, the duration of each iteration is determined by the slowest node [7]. In a similar spirit, [2] addresses the problem of jointly selecting the learning nodes to use for the learning process and assigning them the wireless resources they need to communicate effectively.…”

“…Following an orthogonal, more theoretical approach, several works [7], [10] aim at characterizing the learning performance, deriving closed-form expressions for their (expected) training time. Such a characterization is then exploited to make optimal or near-optimal decisions on the cooperation among nodes [7] and the equilibrium between local learning and global updates [10].…”

“…Following an orthogonal, more theoretical approach, several works [7], [10] aim at characterizing the learning performance, deriving closed-form expressions for their (expected) training time. Such a characterization is then exploited to make optimal or near-optimal decisions on the cooperation among nodes [7] and the equilibrium between local learning and global updates [10]. In order to obtain manageable closed-form expressions, some of these works make simplifying assumptions (e.g., that local datasets be homogeneous), or target specific parameter optimization algorithms (e.g., stochastic gradient descent).…”

“…1. It follows that the pace of the learning process as a whole is determined by the slowest learning node, which becomes an issue when different learning nodes take very different times to perform their iterations [7]. Owing to the limited amount of control that can be exerted on FL nodes, the most viable solution is often to exclude overly-slow nodes from the learning process [2], [5], [7].…”

Towards Node Liability in Federated Learning: Computational Cost and Network Overhead

“…One approach, followed in [4], [5], is to assign more resources (e.g., radio resource blocks) to the nodes that need them the most (e.g., experience connectivity issues), so as to avoid performance bottlenecks. Another option is simply to drop overly-slow nodes ("stragglers") from the learning process, thus making individual iterations faster [3], [4], [10].…”