Quantifying sentence acceptability measures: Reliability, bias, and variability

Abstract

Understanding and measuring sentence acceptability is of fundamental importance for linguists, but although many measures for doing so have been developed, relatively little is known about some of their psychometric properties. In this paper we evaluate within-and between-participant test-retest reliability on a wide range of measures of sentence acceptability. Doing so allows us to estimate how much of the variability within each measure is due to factors including participantlevel individual differences, sample size, response styles, and item effects. The measures examined include Likert scales, two versions of forced-choice judgments, magnitude estimation, and a novel measure based on Thurstonian approaches in psychophysics. We reproduce previous findings of high between-participant reliability within and across measures, and extend these results to a generally high reliability within individual items and individual people. Our results indicate that Likert scales and the Thurstonian approach produce the most stable and reliable acceptability measures and do so with smaller sample sizes than the other measures. Moreover, their agreement with each other suggests that the limitation of a discrete Likert scale does not impose a significant degree of structure on the resulting acceptability judgments.

The Application of Signal Detection Theory to Acceptability Judgments

“…This formal procedure will increase the sample size of participants and items, will better control for confounds, and avoids bias based on adherence to a particular linguistic theory. While the reliability of the informal procedure has been much debated (Gibson and Fedorenko, 2010;Sprouse and Almeida, 2012;Gibson et al, 2013a,b), it has been shown that acceptability judgments are generally reliable when formal data collection procedures are used that conform to the standards of experimental psychology (Langsford et al, 2018;Linzen and Oseki, 2018). Therefore, in this paper, we restrict our discussion to formal data collection procedures.…”

“…Indeed, if we ask the same participant to judge different items in the same condition or if we ask different participants to judge the same item, we would not necessarily expect the same response from every participant on every item. If we look at the results from studies that test the reliability of acceptability judgments, we can see that there is indeed between-subject and between-item variability (e.g., Langsford et al, 2018).…”

Viewing dialect change through acceptability judgments: A case study in Shetland dialect

“…Gradient judgment tasks have been shown to be reliable indicators of acceptability in cases where there is no expected variation. Here, inter-speaker judgment reliability is high (Sprouse, Schütze & Almeida 2013;Sprouse & Almeida 2017;Langsford et al 2018, though see Linzen & Oseki 2018 on Hebrew and Japanese judgments). Gradient judgment tasks have also been shown to be reliable in cases of stable variation in a community, where speakers using both a dialect variant and a standard variant (Thoms 2014;Zanuttini et al 2018).…”

Variation is information: Analyses of variation across items, participants, time, and methods in metalinguistic judgment data

“…a binary scale, a Likert scale, and an open-ended scale constructed in Magnitude Estimation), to see whether the scales differ in perceived ease of use and expressivity, and in the judgment data they provide (e.g. Bader and Häussler 2010;Langsford et al 2018;Preston and Colman 2000).…”

“…Some studies (e.g. Langsford et al 2018) do examine test-retest reliability of judgments expressed on various scales, thus examining variation across time and across methods, but all analyses are performed on mean ratings. We will demonstrate how all four types of variation can be investigated in judgment data, and how they can be used as sources of information.…”