John praised Mary because _he_? Implicit Causality Bias and Its Interaction with Explicit Cues in LMs

Abstract

Some interpersonal verbs can implicitly attribute causality to either their subject or their object and are therefore said to carry an implicit causality (IC) bias. Through this bias, causal links can be inferred from a narrative, aiding language comprehension. We investigate whether pre-trained language models (PLMs) encode IC bias and use it at inference time. We find that to be the case, albeit to different degrees, for three distinct PLM architectures. However, causes do not always need to be implicit-when a cause is explicitly stated in a subordinate clause, an incongruent IC bias associated with the verb in the main clause leads to a delay in human processing. We hypothesize that the temporary challenge humans face in integrating the two contradicting signals, one from the lexical semantics of the verb, one from the sentence-level semantics, would be reflected in higher error rates for models on tasks dependent on causal links. The results of our study lend support to this hypothesis, suggesting that PLMs tend to prioritize lexical patterns over higher-order signals.

Beyond the Bias: Unveiling the Quality of Implicit Causality Prompt Continuations in Language Models

“…Overall, the performance varies across different models, decoding procedures, and bias types. However, in general, the models are more likely to capture the object bias, as can be noted by the (almost) overall higher CS values for the ES verbs, aligning with results from, for example, Kementchedjhieva et al, 2021 andZarrieß et al, 2022 which as well point towards a general tendency of LLMs to establish coreference to the object. Moreover, it is noticeable that for each decoding procedure the ability to capture the IC bias of SE verbs tends to improve when prompts are augmented with adverbial modifiers.…”

“…And, as several psycholinguistic studies have demonstrated that the IC bias is not only highly reliable but also robust across different languages (Ferstl et al, 2011;Goikoetxea et al, 2008;Hartshorne et al, 2013;Bott and Solstad, 2014), it has become an an intriguing domain for testing language models. Earlier studies, including those conducted by Upadhye et al, 2020, Davis and van Schijndel, 2020, Kementchedjhieva et al, 2021and Zarrieß et al, 2022, have examined the performance of LLMs in capturing the IC coreference bias. I.e., they concentrated on single-word prediction tasks and evaluated the models' ability to generate continuations of such classic prompts, like examples (1) and (2), and predominantly found that LLMs display limited ability to systematically incorporate the IC coreference bias in their genera-tions.…”

“…A range of recent studies investigated LLMs' predictions of the next mention in examples like (1) and whether these mentions (i.e. pronouns) follow the same coreference biases as can be found in human data (e.g., Upadhye et al, 2020;Davis and van Schijndel, 2020;Kementchedjhieva et al, 2021;Zarrieß et al, 2022). These studies predominantly indicated that LLMs are not generally congruent with the human IC bias, which has been interpreted as evidence for LLMs struggling with certain aspects of discourse understanding (but see Cai et al, 2023).…”

Incremental Processing of Principle B: Mismatches Between Neural Models and Humans

“…1 Within work in natural language processing, existing models have been claimed to capture aspects of Principle A (e.g., Warstadt et al, 2020;Hu et al, 2020). Principle C has received less attention, though see Mitchell et al (2019) which found that LSTM language models failed to obey Principle C. Coreference, more broadly, has also been explored, with results suggesting that models encode features of coreference resolution (e.g., Sorodoc et al, 2020) and the interaction of implicit causality and pronouns (verb biases that influence preferred antecedents for pronouns; Upadhye et al, 2020;Davis and van Schijndel, 2021;Kementchedjhieva et al, 2021).…”