Emergent Structures and Training Dynamics in Large Language Models

Teehan, Ryan; Clinciu, Miruna; Serikov, Oleg; Szczechla, Eliza; Seelam, Natasha; Mirkin, Shachar; Gokaslan, Aaron

doi:10.18653/v1/2022.bigscience-1.11

Cited by 6 publications

(15 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…After fitting equation (9.1), Hoffmann et al [273] find scriptLtrueprefixminfalse(N,Dfalse)=406.4N0.34+410.7D0.28+1.69.If we then plug in N and D for a selection of real foundation models we arrive at figure 26. We can see in figure 26 that the model size term for real foundation models is far lower than the dataset size term.…”

Section: Foundation Models: a Fourth Astroconnectionist Wave?mentioning

confidence: 99%

“…The table above shows the number of parameters in a model ( N ), the number of tokens within that model’s training set ( D ), and their corresponding calculated emergent terms from equation (9.1). Here we use Hoffmann et al [273] to source values for A , α , B and β . The minimum loss for each model accordingto Hoffmann et al [273] is shown as scriptLtrueprefixmin.…”

Section: Foundation Models: a Fourth Astroconnectionist Wave?mentioning

confidence: 99%

“…Here we use Hoffmann et al [273] to source values for A , α , B and β . The minimum loss for each model accordingto Hoffmann et al [273] is shown as scriptLtrueprefixmin. The contour plot shows the emergent parameters B/Dβ and A/Nα plotted against each other for our models.…”

Section: Foundation Models: a Fourth Astroconnectionist Wave?mentioning

confidence: 99%

See 2 more Smart Citations

Astronomia ex machina: a history, primer and outlook on neural networks in astronomy

Smith

2023

R. Soc. open sci.

View full text Add to dashboard Cite

In this review, we explore the historical development and future prospects of artificial intelligence (AI) and deep learning in astronomy. We trace the evolution of connectionism in astronomy through its three waves, from the early use of multilayer perceptrons, to the rise of convolutional and recurrent neural networks, and finally to the current era of unsupervised and generative deep learning methods. With the exponential growth of astronomical data, deep learning techniques offer an unprecedented opportunity to uncover valuable insights and tackle previously intractable problems. As we enter the anticipated fourth wave of astronomical connectionism, we argue for the adoption of GPT-like foundation models fine-tuned for astronomical applications. Such models could harness the wealth of high-quality, multimodal astronomical data to serve state-of-the-art downstream tasks. To keep pace with advancements driven by Big Tech, we propose a collaborative, open-source approach within the astronomy community to develop and maintain these foundation models, fostering a symbiotic relationship between AI and astronomy that capitalizes on the unique strengths of both fields.

show abstract

Section: Foundation Models: a Fourth Astroconnectionist Wave?mentioning

confidence: 99%

Section: Foundation Models: a Fourth Astroconnectionist Wave?mentioning

confidence: 99%

See 1 more Smart Citation

Astronomia ex machina: a history, primer and outlook on neural networks in astronomy

Smith

2023

R. Soc. open sci.

View full text Add to dashboard Cite

show abstract

“…A more dangerous scenario is if the model chooses a prediction to manipulate the outcome of that prediction (or other predictions), 32 e.g. if the model directly tries to find a self-consistent prediction by solving for a fixed point where the world is consistent with the model's own prediction [Tre22]. 33 Even if the model is myopically maximizing predictive accuracy, it has an incentive to find a fixed point that is both stable and likely-that is, a situation where the world state that results from the model outputting its prediction is highly overdetermined-since that's what makes for the best individual prediction.…”

Section: Major Challenge: Self-fulfilling Propheciesmentioning

confidence: 99%

Conditioning Predictive Models: Risks and Strategies

Evan¹,

Jermyn²,

Johannes³

et al. 2023

Preprint

View full text Add to dashboard Cite

Our intention is to provide a definitive reference on what it would take to safely make use of generative/predictive models in the absence of a solution to the Eliciting Latent Knowledge[CXC21] problem. Furthermore, we believe that large language models can be understood as such predictive models of the world, and that such a conceptualization raises significant opportunities for their safe yet powerful use via carefully conditioning them to predict desirable outputs.Unfortunately, such approaches also raise a variety of potentially fatal safety problems, particularly surrounding situations where predictive models predict the output of other AI systems, potentially unbeknownst to us. There are numerous potential solutions to such problems, however, primarily via carefully conditioning models to predict the things we want-e.g. humans-rather than the things we don't-e.g. malign AIs.Furthermore, due to the simplicity of the prediction objective, we believe that predictive models present the easiest inner alignment[HvMM + 19] problem that we are aware of.

show abstract

“…As our state-of-the-art learning methods approach asymptotically-optimal performance for static training sets and benchmarks, the primary lever for improving our models lies in improving the training data itself. Several results support this data-centric view of ML: for large neural models, continually scaling up the quantity of training data empirically improves generalization performance [17,56,136]. Generating additional supervised samples through methods like data programming —which generates new labels based on human-provided, approximate labelling functions, results in significant performance gains in several domains [137,138].…”

Section: A Unified View Of Explorationmentioning

confidence: 99%

General intelligence requires rethinking exploration

2023

View full text Add to dashboard Cite

We are at the cusp of a transition from ‘learning from data’ to ‘learning what data to learn from’ as a central focus of artificial intelligence (AI) research. While the first-order learning problem is not completely solved, large models under unified architectures, such as transformers, have shifted the learning bottleneck from how to effectively train models to how to effectively acquire and use task-relevant data. This problem, which we frame as exploration , is a universal aspect of learning in open-ended domains like the real world. Although the study of exploration in AI is largely limited to the field of reinforcement learning, we argue that exploration is essential to all learning systems, including supervised learning. We propose the problem of generalized exploration to conceptually unify exploration-driven learning between supervised learning and reinforcement learning, allowing us to highlight key similarities across learning settings and open research challenges. Importantly, generalized exploration is a necessary objective for maintaining open-ended learning processes, which in continually learning to discover and solve new problems, provides a promising path to more general intelligence.

show abstract

Emergent Structures and Training Dynamics in Large Language Models

Cited by 6 publications

References 0 publications

Astronomia ex machina: a history, primer and outlook on neural networks in astronomy

Astronomia ex machina: a history, primer and outlook on neural networks in astronomy

Conditioning Predictive Models: Risks and Strategies

General intelligence requires rethinking exploration

Contact Info

Product

Resources

About