A Test of the Cognitive Assumptions of Magnitude Estimation: Commutativity does not Hold for Acceptability Judgments

Abstract

The introduction of the psychophysical technique of magnitude estimation to the study of acceptability judgments (Bard et al. 1996) has led to a surge of interest in formal acceptability-judgment experiments over the past fifteen years. One of the primary reasons for its popularity is that it was developed as a tool to measure actual units of perception, offering the possibility of data that is inherently more informative than previous scaling tasks. However, there are several untested cognitive assumptions that must hold in order for ME to be the perceptual measurement test that it is purported to be. Building on the recent formalization of these assumptions in the psychophysics literature (Narens 1996, Luce 2002), this article presents two experiments designed to test whether these assumptions hold for acceptability-judgment experiments. The results suggest that the cognitive assumptions of magnitude estimation do not hold for participants in acceptability-judgment experiments, eliminating any reason to believe that ME could deliver inherently more meaningful data than other acceptability-judgment tasks.

A comparison of informal and formal acceptability judgments using a random sample from Linguistic Inquiry 2001–2010

“…There are only 2 divergent sentence types for LS that were not divergent for ME, and only 3 divergent sentence types for ME that were not divergent for LS. This is perhaps unsurprising given the similarities between the two tasks (Weskott and Fanselow, 2011;Sprouse, 2011b). The second property of Table 10 that may be of interest is how consistent the divergences in the threshold analysis are with the divergences in the pairwise analyses.…”

“…This development means that there are two methods for collecting acceptability judgments currently in widespread use in the field of syntax: the relatively informal traditional methods that have largely established the foundation of the field for the past 60 years (henceforth informal methods), and the more formal experimental methods that have been gaining popularity over the past 15 years (henceforth formal methods). This methodological dichotomy has led a number of researchers to ask which method is empirically superior (e.g., Bard et al, 1996;Keller, 2000;Edelman and Christiansen, 2003;Phillips and Lasnik, 2003;Featherston, 2005aFeatherston, , 2005bFeatherston, , 2007Featherston, , 2008Featherston, , 2009Ferreira, 2005;Sorace and Keller, 2005;Wasow and Arnold, 2005;den Dikken et al, 2007;Alexopoulou and Keller, 2007;Bornkessel-Schlesewsky and Schlesewsky, 2007;Fanselow, 2007;Grewendorf, 2007;Haider, 2007;Newmeyer, 2007;Sprouse, 2007;Culbertson and Gross, 2009;Myers, 2009aMyers, , 2009bPhillips, 2010;Bader and Häussler, 2010;Dąbrowska, 2010;Gibson and Fedorenko, 2010;Culicover and Jackendoff, 2010;Gross and Culberton, 2011;Sprouse, 2011b;Weskott and Fanselow, 2011;Gibson et al, 2011;Almeida, 2012, 2013;Gibson and Fedorenko, 2013). Our goal in this paper is to substantially increase the empirical basis of this line of research by comparing the results of informal and formal methods for a very large and random s...…”

“…One of the proposed benefits of ME is that it asks participants to use the standard as a unit of measure to rate the target sentences, potentially resulting in more accurate ratings than are possible with Likert scale tasks (Stevens, 1956). Recent research suggests that this particular benefit may not hold for acceptability judgments, as participants do not appear to use the standard as a unit of measure (Sprouse, 2011b). A second possible benefit of ME concerns the continuous nature of the response scale (i.e., the positive number line), which could in principle allow participants to distinguish finer grained differences in acceptability than the fixed response scales of Likert scale tasks.…”

“…Odd numbered scales such as the one used here allow participants to easily define the most acceptable rating, the least acceptable rating, and a rating that is exactly in the middle. Although the ME task was originally intended by Stevens to supplant the LS task, the fact that participants in acceptability judgment ME tasks do not complete the task in the way envisioned by Stevens (they treat it as an open scale LS task; see Sprouse, 2011b), and the fact that LS and ME yield no difference in statistical power in syntactic experiments (see Sprouse and Almeida submitted), suggests that the LS task remains a viable alternative to ME.…”

Design sensitivity and statistical power in acceptability judgment experiments

“…Participants are asked to indicate the acceptability of target sentences as a multiple of the acceptability of the standard by providing a rating that is a multiple of the modulus. However, it should be noted that recent research suggests that participants do not actually use the standard to make ratio judgments of the target sentences (Sprouse 2011b). …”