Reproducibility in Optimization: Theoretical Framework and Limits

Abstract

We initiate a formal study of reproducibility in optimization. We define a quantitative measure of reproducibility of optimization procedures in the face of noisy or error-prone operations such as inexact or stochastic gradient computations or inexact initialization. We then analyze several convex optimization settings of interest such as smooth, non-smooth, and strongly-convex objective functions and establish tight bounds on the limits of reproducibility in each setting. Our analysis reveals a fundamental trade-off between computation and reproducibility: more computation is necessary (and sufficient) for better reproducibility.

Almost Tight Approximation Algorithms for Explainable Clustering

“…Later, (Esfandiari et al, 2022) considered a natural adaption of this definition to the setting of bandits and designed replicable algorithms that have small regret. A slightly different notion of replicability in optimization was studied in (Ahn et al, 2022), where it is required that an optimization algorithm that uses noisy operations during its execution, e.g., noisy gradient evaluations, outputs solutions that are close when executed twice. Since the problem we are studying has a statistical nature, we adopt the definition of (Impagliazzo et al, 2022).…”

“…These concerns recently led to the study of replicability as a theoretical property of algorithms themselves. In particular, the works of (Impagliazzo et al, 2022;Ahn et al, 2022;Esfandiari et al, 2022) explore replicability and reproducibility in offline learning, convex optimization, and interactive learning, respectively. In this work, we initiate the study of replicability in clustering, which is one of the canonical problems of unsupervised learning.…”

“…Originally this definition was called reproducibility(Impagliazzo et al, 2022) but it was later pointed out that the correct term is replicability(Ahn et al, 2022).…”

Real World Large Scale Recommendation Systems Reproducibility and Smooth Activations

“…Only recently, a series of empirical works [11,13,17,[46][47][48][49]53] demonstrated it. An initial theoretical framework for reproducibility in optimization only appears in very recent work [2] and demonstrates the problem for the much simpler case of convex optimization. Generalizing such results to deep learning, with highly non-convex loss landscape, is even more challenging.…”