A Simple Explanation for the Phase Transition in Large Language Models with List Decoding

Abstract

Various recent experimental results show that large language models (LLM) exhibit emergent abilities that are not present in small models. System performance is greatly improved after passing a certain critical threshold of scale. In this letter, we provide a simple explanation for such a phase transition phenomenon. For this, we model an LLM as a sequence-to-sequence random function. Instead of using instant generation at each step, we use a list decoder that keeps a list of candidate sequences at each step and defers the generation of the output sequence at the end. We show that there is a critical threshold such that the expected number of erroneous candidate sequences remains bounded when an LLM is below the threshold, and it grows exponentially when an LLM is above the threshold. Such a threshold is related to the basic reproduction number in a contagious disease.

Absence of phase transition in random language model

“…Theoretical analyses of phase transitions in language models: Some theoretical studies have examined phase transitions in mathematical models to explain phenomena such as emergent abilities [Chang, 2023] and grokking [Žunkovič andIlievski, 2022, Rubin et al, 2024], although some of them are not necessarily limited to LLMs. Cui et al [2024] demonstrated that a simple model with a self-attention mechanism exhibits a phase transition between phases corresponding to positional and semantic mechanisms.…”