A Closer Look at Linguistic Knowledge in Masked Language Models: The Case of Relative Clauses in American English

Abstract

Transformer-based language models achieve high performance on various tasks, but we still lack understanding of the kind of linguistic knowledge they learn and rely on. We evaluate three models (BERT, RoBERTa, and ALBERT), testing their grammatical and semantic knowledge by sentence-level probing, diagnostic cases, and masked prediction tasks. We focus on relative clauses (in American English) as a complex phenomenon needing contextual information and antecedent identification to be resolved. Based on a naturalistic dataset, probing shows that all three models indeed capture linguistic knowledge about grammaticality, achieving high performance. Evaluation on diagnostic cases and masked prediction tasks considering fine-grained linguistic knowledge, however, shows pronounced model-specific weaknesses especially on semantic knowledge, strongly impacting models' performance. Our results highlight the importance of (a) model comparison in evaluation task and (b) building up claims of model performance and the linguistic knowledge they capture beyond purely probing-based evaluations. * equal contribution.

Uncovering patterns of semantic predictability in sentence processing

“…Our approach extracts word vectors for input to a Gaussian mixture model from the Transformer language model RoBERTa 1 (Liu et al, 2019). While RoBERTa has not been used as extensively in psycholinguistics as models such as GPT-2 (Radford et al, 2019), it has nevertheless proven useful for modeling a wide variety of phenomena, ranging from reading times (Oh, 2021), to neural responses (Mikolov, Coulson, & Bergen, 2022), to linguistic judgments (Mosbach et al, 2020;Lau et al, 2020). We focus on RoBERTa partly because it "encodes" language and can represent how a word fits with the preceding and following context surrounding a particular token.…”