Hungry Hungry Hippos: Towards Language Modeling with State Space Models

Dao, Tri; Fu, Daniel Y.; Saab, Khaled K.; Thomas, Armin W.; Rudra, Atri; Ré, Christopher

doi:10.48550/arxiv.2212.14052

Cited by 12 publications

(35 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Gated State Spaces (GSS) instead compose the operator via gating and a long convolution parametrized via SSMs. Taking this idea further, Hungry Hungry Hippo (H3) (Dao et al, 2022c), motivated by gaps of GSS on associative recall, extend the mechanism to include an additional gate and a short convolution obtained via a shift SSM. Hyena generalizes this body of work by introducing a recurrence of gates and implicit long convolutions, evaluated efficiently.…”

Section: Subquadratic Operatorsmentioning

confidence: 99%

“…Hyena operators build on the H3 mechanism developed by (Dao et al, 2022c). For clarity of exposition, we once again consider the SISO case (D = 1).…”

Section: Hyena Matricesmentioning

confidence: 99%

“…Efforts to reduce the computational cost of attention in models primarily involve the use of linearized, low-rank, and sparse approximations (Child et al, 2019;Wang et al, 2020;Kitaev et al, 2020;Zhai et al, 2021;Roy et al, 2021;Schlag et al, 2021;Tu et al, 2022). These approaches introduce a trade-off between expressivity and speed, requiring hybridization with standard attention layers to reach Transformer quality (Mehta et al, 2022;Dao et al, 2022c).…”

Section: Introductionmentioning

confidence: 99%

“…A growing amount of evidence suggests that attention mechanisms only utilize a small portion of their quadratic capabilities for language processing (Olsson et al, 2022;Dao et al, 2022c), leading us to question its role as the gold-standard operator for deep learning at scale. Specifically, we ask:…”

Section: Introductionmentioning

confidence: 99%

“…A specified depth (i.e., number of steps) of the recurrence controls the size of the operator. For short recurrences, existing models are recovered as special cases (Mehta et al, 2022;Dao et al, 2022c). By mapping each step in the Hyena recurrence to its corresponding matrix form, we reveal Hyena operators to be equivalently defined as a decomposition of a data-controlled matrix i.e., a matrix whose entries are functions of the input.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Hyena Hierarchy: Towards Larger Convolutional Language Models

Poli¹,

Massaroli²,

Nguyen³

et al. 2023

Preprint

View full text Add to dashboard Cite

Recent advances in deep learning have relied heavily on the use of large Transformers due to their ability to learn at scale. However, the core building block of Transformers, the attention operator, exhibits quadratic cost in sequence length, limiting the amount of context accessible. Existing subquadratic methods based on low-rank and sparse approximations need to be combined with dense attention layers to match Transformers, indicating a gap in capability. In this work, we propose Hyena, a subquadratic drop-in replacement for attention constructed by interleaving implicitly parametrized long convolutions and data-controlled gating. In recall and reasoning tasks on sequences of thousands to hundreds of thousands of tokens, Hyena improves accuracy by more than 50 points over operators relying on statespaces and other implicit and explicit methods, matching attention-based models. We set a new state-ofthe-art for dense-attention-free architectures on language modeling in standard datasets (WikiText103 and The Pile), reaching Transformer quality with a 20% reduction in training compute required at sequence length 2K. Hyena operators are twice as fast as highly optimized attention at sequence length 8K, and 100× faster at sequence length 64K.

show abstract

Section: Subquadratic Operatorsmentioning

confidence: 99%

“…Hyena operators build on the H3 mechanism developed by (Dao et al, 2022c). For clarity of exposition, we once again consider the SISO case (D = 1).…”

Section: Hyena Matricesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Hyena Hierarchy: Towards Larger Convolutional Language Models

Poli¹,

Massaroli²,

Nguyen³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

A survey on deep reinforcement learning for audio-based applications

Latif

Cuayáhuitl

Pervez³

et al. 2022

Artif Intell Rev

View full text Add to dashboard Cite

Deep reinforcement learning (DRL) is poised to revolutionise the field of artificial intelligence (AI) by endowing autonomous systems with high levels of understanding of the real world. Currently, deep learning (DL) is enabling DRL to effectively solve various intractable problems in various fields including computer vision, natural language processing, healthcare, robotics, to name a few. Most importantly, DRL algorithms are also being employed in audio signal processing to learn directly from speech, music and other sound signals in order to create audio-based autonomous systems that have many promising applications in the real world. In this article, we conduct a comprehensive survey on the progress of DRL in the audio domain by bringing together research studies across different but related areas in speech and music. We begin with an introduction to the general field of DL and reinforcement learning (RL), then progress to the main DRL methods and their applications in the audio domain. We conclude by presenting important challenges faced by audio-based DRL agents and by highlighting open areas for future research and investigation. The findings of this paper will guide researchers interested in DRL for the audio domain.

show abstract

Multiplicative processing in the modeling of cognitive activities in large neural networks

Valle-Lisboa

Mizraji

2023

Biophys Rev

View full text Add to dashboard Cite

Explaining the foundation of cognitive abilities in the processing of information by neural systems has been in the beginnings of biophysics since McCulloch and Pitts pioneered work within the biophysics school of Chicago in the 1940s and the interdisciplinary cybernetists meetings in the 1950s, inseparable from the birth of computing and artificial intelligence. Since then, neural network models have traveled a long path, both in the biophysical and the computational disciplines. The biological, neurocomputational aspect reached its representational maturity with the Distributed Associative Memory models developed in the early 70 s. In this framework, the inclusion of signal-signal multiplication within neural network models was presented as a necessity to provide matrix associative memories with adaptive, context-sensitive associations, while greatly enhancing their computational capabilities. In this review, we show that several of the most successful neural network models use a form of multiplication of signals. We present several classical models that included such kind of multiplication and the computational reasons for the inclusion. We then turn to the different proposals about the possible biophysical implementation that underlies these computational capacities. We pinpoint the important ideas put forth by different theoretical models using a tensor product representation and show that these models endow memories with the context-dependent adaptive capabilities necessary to allow for evolutionary adaptation to changing and unpredictable environments. Finally, we show how the powerful abilities of contemporary computationally deep-learning models, inspired in neural networks, also depend on multiplications, and discuss some perspectives in view of the wide panorama unfolded. The computational relevance of multiplications calls for the development of new avenues of research that uncover the mechanisms our nervous system uses to achieve multiplication.

show abstract

Hungry Hungry Hippos: Towards Language Modeling with State Space Models

Cited by 12 publications

References 0 publications

Hyena Hierarchy: Towards Larger Convolutional Language Models

Hyena Hierarchy: Towards Larger Convolutional Language Models

A survey on deep reinforcement learning for audio-based applications

Multiplicative processing in the modeling of cognitive activities in large neural networks

Contact Info

Product

Resources

About