This is a BERT. Now there are several of them. Can they generalize to novel words?

Abstract

Recently, large-scale pre-trained neural network models such as BERT have achieved many state-of-the-art results in natural language processing. Recent work has explored the linguistic capacities of these models. However, no work has focused on the ability of these models to generalize these capacities to novel words. This type of generalization is exhibited by humans (Berko, 1958), and is intimately related to morphology-humans are in many cases able to identify inflections of novel words in the appropriate context. This type of morphological capacity has not been previously tested in BERT models, and is important for morphologically-rich languages, which are under-studied in the literature regarding BERT's linguistic capacities. In this work, we investigate this by considering monolingual and multilingual BERT models' abilities to agree in number with novel plural words in English, French, German, Spanish, and Dutch. We find that many models are not able to reliably determine plurality of novel words, suggesting potential deficiencies in the morphological capacities of BERT models.

Accurate and Reliable Classification of Unstructured Reports on Their Diagnostic Goal Using BERT Models

“…While BERT was created for tasks with an English corpus, mBERT was created using all Wikipedia pages with 104 different languages, making it able to handle tasks in multiple languages [ 8 ]. However, this resulted in worse or comparable performance to monolingual models with BERT’s architecture settings [ 14 ]. It has been speculated that this is because Wikipedia is not representative of general language use [ 15 ].…”

Proceedings of the 9th Workshop on Slavic Natural Language Processing 2023 (SlavicNLP 2023)

“…Multilingual models have also demonstrated that synthetic transfer can occur between languages (Guarasci et al, 2022). Meanwhile, Haley (2020) show that BERT can perform the Wug test, a standard grammatical generalisation test (Berko Gleason, 1958), significantly better than chance in English, French, Spanish and Dutch. However, there are still many gaps in this research.…”

Artefact Retrieval: Overview of NLP Models with Knowledge Base Access

“…Pre-trained language models, such as BERT or GPT-3, are either fine-tuned on a new task or are primed on prompts either crafted by humans [Haley, 2020, Misra et al, 2020, Petroni et al, 2019 or found automatically [Shin et al, 2020, Jiang et al, 2020. These prompts (also called stimuli, constraints, or demonstrations) are then dependent only on the specific task at hand and nothing else.…”