How Can Self-Attention Networks Recognize Dyck-n Languages?

Abstract

We focus on the recognition of Dyck-n (D n ) languages with self-attention (SA) networks, which has been deemed to be a difficult task for these networks. We compare the performance of two variants of SA, one with a starting symbol (SA + ) and one without (SA − ). Our results show that SA + is able to generalize to longer sequences and deeper dependencies. For D 2 , we find that SA − completely breaks down on long sequences whereas the accuracy of SA + is 58.82%. We find attention maps learned by SA + to be amenable to interpretation and compatible with a stack-based language recognizer. Surprisingly, the performance of SA networks is at par with LSTMs, which provides evidence on the ability of SA to learn hierarchies without recursion.

Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functions

“…Connections between LSTMs and counter automata have also been established empirically (Suzgun et al, 2019a) and theoretically (Merrill et al, 2020). More recently, multiple works have investigated the ability of Transformers to recognize various regular, context-free (Ebrahimi et al, 2020;Yao et al, 2021;Bhattamishra et al, 2020b), and mildly context-sensitive languages (Wang, 2021).…”