Why random reshuffling beats stochastic gradient descent

Abstract

We analyze the convergence rate of the random reshuffling (RR) method, which is a randomized first-order incremental algorithm for minimizing a finite sum of convex component functions. RR proceeds in cycles, picking a uniformly random order (permutation) and processing the component functions one at a time according to this order, i.e., at each cycle, each component function is sampled without replacement from the collection. Though RR has been numerically observed to outperform its with-replacement counterpart stochastic gradient descent (SGD), characterization of its convergence rate has been a long standing open question. In this paper, we answer this question by providing various convergence rate results for RR and variants when the sum function is strongly convex. We first focus on quadratic component functions and show that the expected distance of the iterates generated by RR with stepsize α k = Θ(1/k s ) for s ∈ (0, 1] converges to zero at rate O(1/k s ) (with s = 1 requiring adjusting the stepsize to the strong convexity constant). Our main result shows that when the component functions are quadratics or smooth (with a Lipschitz assumption on the Hessian matrices), RR with iterate averaging and a diminishing stepsize α k = Θ(1/k s ) for s ∈ (1/2, 1) converges at rate Θ(1/k 2s ) with probability one in the suboptimality of the objective value, thus improving upon the Ω(1/k) rate of SGD. Our analysis draws on the theory of Polyak-Ruppert averaging and relies on decoupling the dependent cycle gradient error into an independent term over cycles and another term dominated by α 2 k . This allows us to apply law of large numbers to an appropriately weighted version of the cycle gradient errors, where the weights depend on the stepsize. We also provide high probability convergence rate estimates that shows decay rate of different terms and allows us to propose a modification of RR with convergence rate O( 1 k 2 ).

Differentially Private Model Publishing for Deep Learning

“…Therefore, we argue that the privacy accounting method has to be chosen according to which data batching method is used. In our implementation, we focus on random reshuffling, because it is a common practice [5], [24] followed by existing deep learning implementations and it is also numerically observed that random reshuffling outperforms random sampling with replacement in the convergence rate [25]. Note: In our previous conference version we set P (q, σ) = q 2 /σ 2 directly based on the derivation from the log moment bound stated in the proof of Theorem 1 of [8] which misses an asymptotic term.…”

Fundamental Limits of Decentralized Data Shuffling

“…The scheme proposed in [3] uses a random uncoded storage (to fill users' extra memories independently when M > q) and a coded multicast transmission from the master to the workers, and yields a gain of a factor of O(K) in terms of communication load with respect to the naive 2 Notice that putting all nodes on the same bus (typical terminology in Compute Science) is very common and practically relevant since this is what happens for example with Ethernet, or with the Peripheral Component Interconnect Express (PCI Express) bus inside a multi-core computer, where all cores share a common bus for intercommunication. The access of such bus is regulated by some collision avoidance protocol such as Carrier Sense Multiple Access (CSMA) [4] or Token ring [5], such that once one node talks at a time, and all other listen.…”

“…To cope with such a large size/dimension of data and the complexity of machine learning algorithms, it is increasingly popular to use distributed computing platforms such as Amazon Web Services Cloud, Google Cloud, and Microsoft Azure services, where large scale distributed machine learning algorithms can be implemented. The approach of data shuffling has been identified as one of the core elements to improve the statistical performance of modern large scale machine learning algorithms [1], [2].…”

“…In this work, we consider such a decentralized data shuffling framework, where workers, 1 In practice, workers communicate with each other as described in [1]. connected by the same communication bus (common shared link), are allowed to communicate 2 .…”

Curvature-aided incremental aggregated gradient method

“…We also define L H := m i=1 L H,i as the Lipschitz constant for the Hessian of the sum function F (θ). In the above, the first assumption can be satisfied when Line 4 of Algorithm 3.1 is implemented with either cyclic function selection, i.e., i k = (k mod m) + 1, or a random shuffling at the beginning of every epoch [15]. The second and the last assumptions are standard and they can be satisfied by a number of functions relevant to machine learning applications, e.g., the logistic loss function.…”