On Equivalent Optimization of Machine Learning Methods

Abstract

At the core of many machine learning methods resides an iterative optimization algorithm for their training. Such optimization algorithms often come with a plethora of choices regarding their implementation. In the case of deep neural networks, choices of optimizer, learning rate, batch size, etc. must be made. Despite the fundamental way in which these choices impact the training of deep neural networks, there exists no general method for identifying when they lead to equivalent, or non-equivalent, optimization trajectories. By viewing iterative optimization as a discrete-time dynamical system, we are able to leverage Koopman operator theory, where it is known that conjugate dynamics can have identical spectral objects. We find highly overlapping Koopman spectra associated with the application of online mirror and gradient descent to specific problems, illustrating that such a data-driven approach can corroborate the recently discovered analytical equivalence between the two optimizers. We extend our analysis to feedforward, fully connected neural networks, providing the first general characterization of when choices of learning rate, batch size, layer width, data set, and activation function lead to equivalent, and non-equivalent, evolution of network parameters during training. Among our main results, we find that learning rate to batch size ratio, layer width, nature of data set (handwritten vs. synthetic), and activation function affect the nature of conjugacy. Our data-driven approach is general and can be utilized broadly to compare the optimization of machine learning methods.

Not so griddy: Internal representations of RNNs path integrating more than one agent

“…Recent work [50] has used data-driven tools [85–87] from Koopman operator theory [88, 89] to extract properties of RNN dynamics that can be compared using a precise notion of equivalence, called “topological conjugacy” [90]. Similar approaches have been used to study the dynamics associated with training deep neural networks [91], as well as iterative optimization algorithms, more generally [92]. Therefore, we complimented the geometric and topological analysis performed on the structure of the population level representations (Fig.…”