The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks

Abstract

In this paper, we conjecture that if the permutation invariance of neural networks is taken into account, SGD solutions will likely have no barrier in the linear interpolation between them. Although it is a bold conjecture, we show how extensive empirical attempts fall short of refuting it. We further provide a preliminary theoretical result to support our conjecture. Our conjecture has implications for lottery ticket hypothesis, distributed training and ensemble methods.

Linear Connectivity Reveals Generalization Strategies

“…This result suggests that generalization, or at least high performance, is closely tied to the linear mode connectivity of the models in question. This implied result is further supported by Entezari et al [7], which found a larger barrier between models when they exhibited higher test error. Neyshabur et al [35] even described linear mode connectivity as a crucial component of transfer learning, finding that finetuned models initialized from the same pretrained model will be in the same linearly connected basin, in contrast to models trained from scratch, which exhibit barriers even when initialized from the same random weights.…”

“…This behavior characterizes the two basins: the larger basin is syntax-aware (tending to acquire heuristics that require awareness of constituent structure), while the smaller basin is syntax-unaware (acquiring heuristics that rely only on unordered sets of words). 7 Distributions of the clusters: We find (Fig. 5) that CG-based cluster membership accounts for some of the heavy tail of performance on HANS-LO, supporting the claim that the convex basins on the loss surface differentiate generalization strategies.…”

“…Inspired by Entezari et al [7], we define our notion of a basin in order to formalize types of behavior during linear interpolation. In contrast to their work, however, our goal is not to identify a largely stable region (the bottom of a basin).…”

“…Entezari et al [7] define a barrier's height on a linear path from θ 1 to θ 2 as: We define the convexity gap on a linear path from θ 1 to θ 2 as the maximum possible barrier height on any sub-segment of the linear path joining θ 1 and θ 2 . Mathematically,…”

“…Garipov et al [10] and Fort and Jastrzebski [8] conceptualized these paths as high dimensional volumes connecting entire sets of solutions. Goodfellow et al [12] explored linear connections between model pairs by interpolating between them, while [7] suggested that in general, models that seem linearly disconnected may be connected after permuting their neurons. Our work explores models that are linearly connected without permutation, and is most closely linked with Neyshabur et al [35], which found that models initialized from a shared pretrained checkpoint are linearly connected.…”