Explaining pretrained language models' understanding of linguistic structures using construction grammar

Abstract

Construction Grammar (CxG) is a paradigm from cognitive linguistics emphasizing the connection between syntax and semantics. Rather than rules that operate on lexical items, it posits constructions as the central building blocks of language, i.e., linguistic units of different granularity that combine syntax and semantics. As a first step toward assessing the compatibility of CxG with the syntactic and semantic knowledge demonstrated by state-of-the-art pretrained language models (PLMs), we present an investigation of their capability to classify and understand one of the most commonly studied constructions, the English comparative correlative (CC). We conduct experiments examining the classification accuracy of a syntactic probe on the one hand and the models' behavior in a semantic application task on the other, with BERT, RoBERTa, and DeBERTa as the example PLMs. Our results show that all three investigated PLMs, as well as OPT, are able to recognize the structure of the CC but fail to use its meaning. While human-like performance of PLMs on many NLP tasks has been alleged, this indicates that PLMs still suffer from substantial shortcomings in central domains of linguistic knowledge.

Enhancing Language Representation with Constructional Information for Natural Language Understanding

“…[•] N refers to the aspects with negative sentiment polarity. 2020) while the meaning is not attached to the words instantiating the construction, but rather to the abstract pattern itself (Weissweiler et al, 2023). Learning the representation of more complex and abstract linguistic units (called constructions) can be more challenging, such as collocations, argument structures and sentence patterns.…”

“…Weissweiler et al (2022) discover that PLMs can recognize the comparative correlative constructions but fail to utilize their meaning. Furthermore, Weissweiler et al (2023) argue that fine-tuning on downstream tasks necessitates explicit access to constructional information. Our work is the first attempt to exploit the constructional information for representation enhancement on NLU tasks.…”

Can Language Models Be Tricked by Language Illusions? Easier with Syntax, Harder with Semantics

“…For additional critiques of the role of ideal grammatical knowledge in evaluations of LMs, seePannitto and Herbelot (2020);Weissweiler et al (2023).…”