Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit

Barak, Boaz; Edelman, Benjamin; Goel, Surbhi; Kakade, Sham M.; Malach, Eran; Zhang, Cyril

doi:10.48550/arxiv.2207.08799

Cited by 4 publications

(6 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Millidge (2022) suggests that this may be due to SGD being a random walk on the optimal manifold. Our results echo Barak et al (2022) in showing that the network instead makes continuous progress toward the generalizing algorithm. Liu et al (2022) construct small examples of grokking, which they use to compute phase diagrams with four separate "phases" of learning.…”

Section: Related Worksupporting

confidence: 63%

“…Progress measures. Barak et al (2022) introduce the notion of progress measures-metrics that improve smoothly and that precede emergent behavior. They prove theoretically that training would amplify a certain mechanism and heuristically define a progress measure.…”

Section: Related Workmentioning

confidence: 99%

“…Emergence is most surprising when it is abrupt, as in the case of reward hacking, chain-of-thought reasoning, or other phase transitions Wei et al, 2022a). We could better understand and predict these phase transitions by finding hidden progress measures (Barak et al, 2022): metrics that precede and are causally linked to the phase transition, and which vary more smoothly. For example, Wei et al (2022a) show that while large language models show abrupt jumps in their performance on many benchmarks, their cross-entropy loss decreases smoothly with model scale.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Progress measures for grokking via mechanistic interpretability

Nanda¹,

Chan²,

Tom³

et al. 2023

Preprint

View full text Add to dashboard Cite

Neural networks often exhibit emergent behavior, where qualitatively new capabilities arise from scaling up the amount of parameters, training data, or training steps. One approach to understanding emergence is to find continuous progress measures that underlie the seemingly discontinuous qualitative changes. We argue that progress measures can be found via mechanistic interpretability: reverseengineering learned behaviors into their individual components. As a case study, we investigate the recently-discovered phenomenon of "grokking" exhibited by small transformers trained on modular addition tasks. We fully reverse engineer the algorithm learned by these networks, which uses discrete Fourier transforms and trigonometric identities to convert addition to rotation about a circle. We confirm the algorithm by analyzing the activations and weights and by performing ablations in Fourier space. Based on this understanding, we define progress measures that allow us to study the dynamics of training and split training into three continuous phases: memorization, circuit formation, and cleanup. Our results show that grokking, rather than being a sudden shift, arises from the gradual amplification of structured mechanisms encoded in the weights, followed by the later removal of memorizing components.

show abstract

Section: Related Worksupporting

confidence: 63%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Progress measures for grokking via mechanistic interpretability

Nanda¹,

Chan²,

Tom³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Work in mechanistic interpretability aims to discover, understand, and verify the algorithms that model weights implement by reverse engineering model computation into human-understandable components (Olah, 2022;Meng et al, 2022;Geiger et al, 2021;Geva et al, 2020). By understanding underlying mechanisms, we can better predict out-of-distribution behavior (Mu & Andreas, 2020), identify and fix model errors Vig et al, 2020), and understand emergent behavior (Nanda & Lieberum, 2022;Barak et al, 2022;Wei et al, 2022).…”

Section: Introductionmentioning

confidence: 99%

Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small

Wang¹,

Variengien²,

Conmy³

et al. 2022

Preprint

View full text Add to dashboard Cite

Research in mechanistic interpretability seeks to explain behaviors of machine learning (ML) models in terms of their internal components. However, most previous work either focuses on simple behaviors in small models or describes complicated behaviors in larger models with broad strokes. In this work, we bridge this gap by presenting an explanation for how GPT-2 small performs a natural language task called indirect object identification (IOI). Our explanation encompasses 26 attention heads grouped into 7 main classes, which we discovered using a combination of interpretability approaches relying on causal interventions. To our knowledge, this investigation is the largest end-to-end attempt at reverse-engineering a natural behavior "in the wild" in a language model. We evaluate the reliability of our explanation using three quantitative criteria-faithfulness, completeness and minimality. Though these criteria support our explanation, they also point to remaining gaps in our understanding. Our work provides evidence that a mechanistic understanding of large ML models is feasible, pointing toward opportunities to scale our understanding to both larger models and more complex tasks. Code for all experiments is available at https://github.com/redwoodresearch/Easy-Transformer.

show abstract

“…An early work with high relevance to the present work is (Wei et al, 2018), which in addition to establishing that the NTK requires Ω(d 2 / ) samples whereas O(d/ ) suffice for the global maximum margin solution, also provided a noisy Wasserstein Flow (WF) analysis which achieved the maximum margin solution, albeit using noise, infinite width, and continuous time to aid in local search, The global maximum margin work of Chizat and Bach (2020) was mentioned before, and will be discussed in Section 3. The work of Barak et al (2022) uses a two phase algorithm: the first step has a large minibatch and effectively learns the support of the parity in an unsupervised manner, and thereafter only the second layer is trained, a convex problem which is able to identify the signs within the parity; as in Table 1, this work stands alone in terms of the narrow width it can handle. The work of (Abbe et al, 2022) uses a similar two-phase approach, and while it can not learn precisely the parity, it can learn an interesting class of "staircase" functions, and presents many valuable proof techniques.…”

Section: Further Related Workmentioning

confidence: 99%

Feature selection with gradient descent on two-layer networks in low-rotation regimes

Telgarsky¹

2022

Preprint

View full text Add to dashboard Cite

This work establishes low test error of gradient flow (GF) and stochastic gradient descent (SGD) on two-layer ReLU networks with standard initialization, in three regimes where key sets of weights rotate little (either naturally due to GF and SGD, or due to an artificial constraint), and making use of margins as the core analytic technique. The first regime is near initialization, specifically until the weights have moved by O( √ m), where m denotes the network width, which is in sharp contrast to the O(1) weight motion allowed by the Neural Tangent Kernel (NTK); here it is shown that GF and SGD only need a network width and number of samples inversely proportional to the NTK margin, and moreover that GF attains at least the NTK margin itself, which suffices to establish escape from bad KKT points of the margin objective, whereas prior work could only establish nondecreasing but arbitrarily small margins. The second regime is the Neural Collapse (NC) setting, where data lies in extremely-well-separated groups, and the sample complexity scales with the number of groups; here the contribution over prior work is an analysis of the entire GF trajectory from initialization. Lastly, if the inner layer weights are constrained to change in norm only and can not rotate, then GF with large widths achieves globally maximal margins, and its sample complexity scales with their inverse; this is in contrast to prior work, which required infinite width and a tricky dual convergence assumption. As purely technical contributions, this work develops a variety of potential functions and other tools which will hopefully aid future work.

show abstract

Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit

Cited by 4 publications

References 0 publications

Progress measures for grokking via mechanistic interpretability

Progress measures for grokking via mechanistic interpretability

Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small

Feature selection with gradient descent on two-layer networks in low-rotation regimes

Contact Info

Product

Resources

About