Structured Semantic Knowledge Can Emerge Automatically from Predicting Word Sequences in Child-Directed Speech

Abstract

Previous research has suggested that distributional learning mechanisms may contribute to the acquisition of semantic knowledge. However, distributional learning mechanisms, statistical learning, and contemporary “deep learning” approaches have been criticized for being incapable of learning the kind of abstract and structured knowledge that many think is required for acquisition of semantic knowledge. In this paper, we show that recurrent neural networks, trained on noisy naturalistic speech to children, do in fact learn what appears to be abstract and structured knowledge. We trained two types of recurrent neural networks (Simple Recurrent Network, and Long Short-Term Memory) to predict word sequences in a 5-million-word corpus of speech directed to children ages 0–3 years old, and assessed what semantic knowledge they acquired. We found that learned internal representations are encoding various abstract grammatical and semantic features that are useful for predicting word sequences. Assessing the organization of semantic knowledge in terms of the similarity structure, we found evidence of emergent categorical and hierarchical structure in both models. We found that the Long Short-term Memory (LSTM) and SRN are both learning very similar kinds of representations, but the LSTM achieved higher levels of performance on a quantitative evaluation. We also trained a non-recurrent neural network, Skip-gram, on the same input to compare our results to the state-of-the-art in machine learning. We found that Skip-gram achieves relatively similar performance to the LSTM, but is representing words more in terms of thematic compared to taxonomic relations, and we provide reasons why this might be the case. Our findings show that a learning system that derives abstract, distributed representations for the purpose of predicting sequential dependencies in naturalistic language may provide insight into emergence of many properties of the developing semantic system.

The Role of Co-Occurrence Statistics in Developing Semantic Knowledge

“…In this section, we review three sources of evidence that underline the possibility that cooccurrence regularities contribute to forming links between concepts. First, we present evidence that co-occurrence regularities that can support the formation of semantic relations are ubiquitous in environmental input, including both language and visual scenes (Asr, Willits, & Jones, 2016;Frermann & Lapata, 2015;Hofmann, Biemann, Westbury et al, 2018;Huebner & Willits, 2018;Jones & Mewhort, 2007;Landauer & Dumais, 1997;Rohde, Gonnerman, & Plaut, 2004;Sadeghi, McClelland, & Hoffman, 2015). Second, we underline the plausibility that these ubiquitous co-occurrence regularities contribute to the development of semantic organization by reviewing evidence that human learners are sensitive to these regularities from an early age (e.g., Pelucchi, Hay, & Saffran, 2009;Saffran, Aslin, & Newport, 1996).…”

Understanding the role of linguistic distributional knowledge in cognition

“…The cognitive plausibility of LDMs has been a concern since their inception and LINGUISTIC DISTRIBUTIONAL KNOWLEDGE IN COGNITION 7 continues to be a matter of debate (Barsalou, 2017;Boleda & Herbelot, 2017;Glenberg & Robertson, 2000;Günther et al, 2019;Perfetti, 1998). Some critics have targeted low-level implementational details of specific models, such as the use of supervised learning in word2vec models (e.g., Huebner & Willits, 2018;cf. Hollis, 2017).…”

Quantity and Diversity: Simulating Early Word Learning Environments

“…This corpus consists of a total of 4,432 individual conversations (contiguous recording sessions) containing a total of about 6.5 million words. We used a version of the CHILDES corpus that had been processed to (1) remove a number of the special transcription characters and other artifacts of the CHILDES coding system and (2) systematize words with idiosyncratic spellings (e.g., replace all instances of "doggy" with "doggie" to maintain consistent spelling) (Huebner & Willits, 2017). We first describe the properties of this baseline environment and then the relations between types and tokens in simulated environments derived from this baseline environment.…”