We introduce Dynabench, an open-source platform for dynamic dataset creation and model benchmarking. Dynabench runs in a web browser and supports human-and-model-inthe-loop dataset creation: annotators seek to create examples that a target model will misclassify, but that another person will not. In this paper, we argue that Dynabench addresses a critical need in our community: contemporary models quickly achieve outstanding performance on benchmark tasks but nonetheless fail on simple challenge examples and falter in real-world scenarios. With Dynabench, dataset creation, model development, and model assessment can directly inform each other, leading to more robust and informative benchmarks. We report on four initial NLP tasks, illustrating these concepts and highlighting the promise of the platform, and address potential objections to dynamic benchmarking as a new standard for the field.

Neural language models (LMs) perform well on tasks that require sensitivity to syntactic structure. Drawing on the syntactic priming paradigm from psycholinguistics, we propose a novel technique to analyze the representations that enable such success. By establishing a gradient similarity metric between structures, this technique allows us to reconstruct the organization of the LMs' syntactic representational space. We use this technique to demonstrate that LSTM LMs' representations of different types of sentences with relative clauses are organized hierarchically in a linguistically interpretable manner, suggesting that the LMs track abstract properties of the sentence.1 Wells et al. (2009) measured priming effects for relative clauses, not dative constructions. For work on priming in production with dative constructions, see Kaschak et al. (2011).

Syntactically ambiguous sentences that are disambiguated in favor of a less preferred parse are read more slowly than their unambiguous counterparts. This is called a garden path effect. Recent self-paced reading studies have found that this effect decreased as participants are exposed to many such syntactically ambiguous sentences over the course of an experiment. This decrease has been interpreted as evidence that readers rapidly calibrate their expectations to a new environment in order to minimize their surprise when they encounter these initially unexpected syntactic structures (Fine, Jaeger, Farmer, & Qian, 2013). However, such expectation recalibration, referred to as syntactic adaptation, is only one possible explanation for the observed decrease in garden-path effect: recent studies have argued that this decrease may be driven by increased familiarity with the self-paced reading paradigm (task adaptation; Harrington Stack, James, and Watson, 2018). The goal of this paper is to tease apart these two explanations. We demonstrate that there is evidence for rapid syntactic adaptation over and above task adaptation. In a series of power analyses, we should that a large number of participants are necessary in order to have adequate power to detect this effect of syntactic adaptation. The issue of power is exacerbated in experiments designed to detect modulations of the basic syntactic adaptation effect. Such experiments are likely to be underpowered with even 1200 participants. We conclude that syntactic adaptation can in fact be detected using self-paced reading, but that other paradigms (eye-tracking) may be more effective for studying this phenomenon.

We introduce Dynabench, an open-source platform for dynamic dataset creation and model benchmarking. Dynabench runs in a web browser and supports human-and-model-inthe-loop dataset creation: annotators seek to create examples that a target model will misclassify, but that another person will not. In this paper, we argue that Dynabench addresses a critical need in our community: contemporary models quickly achieve outstanding performance on benchmark tasks but nonetheless fail on simple challenge examples and falter in real-world scenarios. With Dynabench, dataset creation, model development, and model assessment can directly inform each other, leading to more robust and informative benchmarks. We report on four initial NLP tasks, illustrating these concepts and highlighting the promise of the platform, and address potential objections to dynamic benchmarking as a new standard for the field.

The stimuli for all experiments and the scripts used to create the experiments, generate the plots and run all the analyses are available on OSF: https://osf.io/qd8ye/

There has been increased awareness that individuals need not have a binary gender identity (i.e., male or female), but rather, gender identities exist on a spectrum. With this increased awareness, there has also been an increase in the use of they as a singular pronoun when referring to individuals with a non-binary gender identity. Has the processing of singular they changed along with a change in its usage? Previous studies have demonstrated that sentences in which they is co-indexed with singular antecedents, are judged acceptable and are easy to process, but only if the antecedents are non-referential and/or have ambiguous gender; co-indexing they with referential antecedents with unambiguous gender (e.g., Mary) results in lower acceptability ratings and greater processing effort. We investigated whether participants who frequently interacted with individuals with a non-binary gender identity and/or identified as having a non-binary gender themselves would process sentences in which themselves was co-indexed with singular antecedents similarly. We found a significant P600 effect for sentences in which themselves was co-indexed with singular referential antecedents with unambiguous gender, but failed to find a P600 effect when the antecedents were non-referential and/or had an ambiguous gender. This pattern of results is consistent with behavioural results from previous studies, suggesting that the change in the usage of singular they has not resulted in a corresponding change in the way in which this pronoun is processed.

Prediction has been proposed as an overarching principle that explains human information processing in language and beyond. To what degree can processing difficulty in syntactically complex sentences - one of the major concerns of psycholinguistics - be explained by predictability, as estimated using computational language models? A precise, quantitative test of this question requires a much larger scale data collection effort than has been done in the past. We present the Syntactic Ambiguity Processing Benchmark, a dataset of self-paced reading times from 2000 participants, who read a diverse set of complex English sentences. This dataset makes it possible to measure processing difficulty associated with individual syntactic constructions, and even individual sentences, precisely enough to rigorously test the predictions of computational models of language comprehension. We find that the predictions of language models with two different architectures sharply diverge from the reading time data, dramatically underpredicting processing difficulty, failing to predict relative difficulty among different syntactic ambiguous constructions, and only partially explaining item-wise variability. These findings suggest that prediction is most likely insufficient on its own to explain human syntactic processing.

When language models process syntactically complex sentences, do they use their representations of syntax in a manner that is consistent with the grammar of the language? We propose AlterRep, an intervention-based method to address this question. For any linguistic feature of a given sentence, AlterRep generates counterfactual representations by altering how the feature is encoded, while leaving intact all other aspects of the original representation. By measuring the change in a model's word prediction behavior when these counterfactual representations are substituted for the original ones, we can draw conclusions about the causal effect of the linguistic feature in question on the model's behavior. We apply this method to study how BERT models of different sizes process relative clauses (RCs). We find that BERT variants use RC boundary information during word prediction in a manner that is consistent with the rules of English grammar; this RC boundary information generalizes to a considerable extent across different RC types, suggesting that BERT represents RCs as an abstract linguistic category.