No free lunch in linguistics or machine learning: Response to Pater

Rawski, Jonathan; Heinz, Jeffrey

doi:10.1353/lan.2019.0021

“…These models were heavily influenced by the early advances in neural network research (Alderete and Tupper, 2018a ; Pater, 2019 ). Modeling linguistic data with neural networks has seen a rapid increase in the past few years (Alderete et al, 2013 ; Avcu et al, 2017 ; Kirov, 2017 ; Alderete and Tupper, 2018a ; Dupoux, 2018 ; Mahalunkar and Kelleher, 2018 ; Weber et al, 2018 ; Prickett et al, 2019 , for cautionary notes, see Rawski and Heinz, 2019 ). One of the promising implications of neural network modeling is the ability to test generalizations that the models produce without language-specific assumptions (Pater, 2019 ).…”

Section: Previous Workmentioning

confidence: 99%

“…This paper also proposes a technique for establishing the Generator's internal representations. The inability to uncover networks' representations has been used as an argument against neural network approaches to linguistic data (among others in Rawski and Heinz, 2019 ). We argue that the internal representation of a network can be, at least partially, uncovered.…”

Section: Introductionmentioning

confidence: 99%

Generative Adversarial Phonology: Modeling Unsupervised Phonetic and Phonological Learning With Neural Networks

Beguš

¹

2020

Front. Artif. Intell.

View full text Add to dashboard Cite

Training deep neural networks on well-understood dependencies in speech data can provide new insights into how they learn internal representations. This paper argues that acquisition of speech can be modeled as a dependency between random space and generated speech data in the Generative Adversarial Network architecture and proposes a methodology to uncover the network's internal representations that correspond to phonetic and phonological properties. The Generative Adversarial architecture is uniquely appropriate for modeling phonetic and phonological learning because the network is trained on unannotated raw acoustic data and learning is unsupervised without any language-specific assumptions or pre-assumed levels of abstraction. A Generative Adversarial Network was trained on an allophonic distribution in English, in which voiceless stops surface as aspirated word-initially before stressed vowels, except if preceded by a sibilant [s]. The network successfully learns the allophonic alternation: the network's generated speech signal contains the conditional distribution of aspiration duration. The paper proposes a technique for establishing the network's internal representations that identifies latent variables that correspond to, for example, presence of [s] and its spectral properties. By manipulating these variables, we actively control the presence of [s] and its frication amplitude in the generated outputs. This suggests that the network learns to use latent variables as an approximation of phonetic and phonological representations. Crucially, we observe that the dependencies learned in training extend beyond the training interval, which allows for additional exploration of learning representations. The paper also discusses how the network's architecture and innovative outputs resemble and differ from linguistic behavior in language acquisition, speech disorders, and speech errors, and how well-understood dependencies in speech data can help us interpret how neural networks learn their representations.

show abstract

“…These models were heavily influenced by the early advances in neural network research (Alderete and Tupper, 2018a;Pater, 2019). Modeling linguistic data with neural networks has seen a rapid increase in the past few years (Alderete et al 2013;Avcu et al 2017;Kirov 2017;Alderete and Tupper 2018a;Mahalunkar and Kelleher 2018;Weber et al 2018;Dupoux 2018;Prickett et al 2019, for cautionary notes, see Rawski and Heinz 2019). One of the promising implications of neural network modeling is the ability to test generalizations that the models produce without language-specific assumptions (Pater, 2019).…”

Section: Previous Workmentioning

confidence: 99%

Generative Adversarial Phonology: Modeling unsupervised allophonic learning with neural networks

Beguš¹

2019

Preprint

0

View full text Add to dashboard Cite

Training deep neural networks on well-understood dependencies in speech data can provide new insights into how they learn internal representations. This paper argues that acquisition of speech can be modeled as a dependency between random space and generated speech data in the Generative Adversarial Network architecture and proposes a methodology to uncover the network's internal representations that correspond to phonetic and phonological properties. The Generative Adversarial architecture is uniquely appropriate for modeling phonetic and phonological learning because the network is trained on unannotated raw acoustic data and learning is unsupervised without any language-specific assumptions or pre-assumed levels of abstraction. A Generative Adversarial Network was trained on an allophonic distribution in English, in which voiceless stops surface as aspirated word-initially before stressed vowels, except if preceded by a sibilant [s]. The network successfully learns the allophonic alternation: the network's generated speech signal contains the conditional distribution of aspiration duration. The paper proposes a technique for establishing the network's internal representations that identifies latent variables that correspond to, for example, presence of [s] and its spectral properties. By manipulating these variables, we actively control the presence of [s] and its frication amplitude in the generated outputs. This suggests that the network learns to use latent variables as an approximation of phonetic and phonological representations. Crucially, we observe that the dependencies learned in training extend beyond the training interval, which allows for additional exploration of learning representations. The paper also discusses how the network's architecture and innovative outputs resemble and differ from linguistic behavior in language acquisition, speech disorders, and speech errors, and how well-understood dependencies in speech data can help us interpret how neural networks learn their representations.

show abstract

“…He argues that these architectures may supply the theory of learning that linguistics currently lacks. In response, Rawski and Heinz (2019) invoke the no-free-lunch theorems (Wolpert and Macready, 1997) and poverty-of-the-stimulus arguments (Chomsky, 1986) to question whether neural models actually have the right inductive biases.…”

Section: Introductionmentioning

confidence: 99%

Priorless Recurrent Networks Learn Curiously

Mitchell¹,

Bowers²

2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

Recently, domain-general recurrent neural networks, without explicit linguistic inductive biases, have been shown to successfully reproduce a range of human language behaviours, such as accurately predicting number agreement between nouns and verbs. We show that such networks will also learn number agreement within unnatural sentence structures, i.e. structures that are not found within any natural languages and which humans struggle to process. These results suggest that the models are learning from their input in a manner that is substantially different from human language acquisition, and we undertake an analysis of how the learned knowledge is stored in the weights of the network. We find that while the model has an effective understanding of singular versus plural for individual sentences, there is a lack of a unified concept of number agreement connecting these processes across the full range of inputs. Moreover, the weights handling natural and unnatural structures overlap substantially, in a way that underlines the non-human-like nature of the knowledge learned by the network.

show abstract

No free lunch in linguistics or machine learning: Response to Pater

Cited by 12 publications

References 36 publications

Generative Adversarial Phonology: Modeling Unsupervised Phonetic and Phonological Learning With Neural Networks

Generative Adversarial Phonology: Modeling Unsupervised Phonetic and Phonological Learning With Neural Networks

Generative Adversarial Phonology: Modeling unsupervised allophonic learning with neural networks

Priorless Recurrent Networks Learn Curiously

Contact Info

Product

Resources

About