The design and analysis of small-scale syntactic judgment experiments

A comparison of informal and formal acceptability judgments using a random sample from Linguistic Inquiry 2001–2010

“…The other 21% contained discussion in the text surrounding the data point that implied a control condition that any professional syntactician could construct for themselves. Furthermore, the relative frequency of pairwise phenomena has been explicitly recognized in both the experimental syntax literature (e.g., Bard et al, 1996;Myers, 2009a), and the theoretical syntax literature (as Bošković and Lasnik, 2003:527 put it, "As is standard in the literature, the judgments reported in this article are intended as relative rather than absolute, and most of the data was collected by soliciting relative judgments between pairs of examples."). Beyond being a relatively frequent source of evidence in syntactic theory, pairwise phenomena are also a relatively useful source of evidence.…”

“…This development means that there are two methods for collecting acceptability judgments currently in widespread use in the field of syntax: the relatively informal traditional methods that have largely established the foundation of the field for the past 60 years (henceforth informal methods), and the more formal experimental methods that have been gaining popularity over the past 15 years (henceforth formal methods). This methodological dichotomy has led a number of researchers to ask which method is empirically superior (e.g., Bard et al, 1996;Keller, 2000;Edelman and Christiansen, 2003;Phillips and Lasnik, 2003;Featherston, 2005aFeatherston, , 2005bFeatherston, , 2007Featherston, , 2008Featherston, , 2009Ferreira, 2005;Sorace and Keller, 2005;Wasow and Arnold, 2005;den Dikken et al, 2007;Alexopoulou and Keller, 2007;Bornkessel-Schlesewsky and Schlesewsky, 2007;Fanselow, 2007;Grewendorf, 2007;Haider, 2007;Newmeyer, 2007;Sprouse, 2007;Culbertson and Gross, 2009;Myers, 2009aMyers, , 2009bPhillips, 2010;Bader and Häussler, 2010;Dąbrowska, 2010;Gibson and Fedorenko, 2010;Culicover and Jackendoff, 2010;Gross and Culberton, 2011;Sprouse, 2011b;Weskott and Fanselow, 2011;Gibson et al, 2011;Almeida, 2012, 2013;Gibson and Fedorenko, 2013). Our goal in this paper is to substantially increase the empirical basis of this line of research by comparing the results of informal and formal methods for a very large and random s...…”

A Test of the Cognitive Assumptions of Magnitude Estimation: Commutativity does not Hold for Acceptability Judgments

“…The ability to test the fundamental cognitive assumptions of magnitude estimation is particularly relevant given the recent discussion about the statistical power of magnitude estimation (compared to the statistical power of other tasks) in the literature (Featherston 2008, Myers 2009, Bader & Häussler 2010, Weskott & Fanselow 2011. These discussions are direct consequences of the early claims that magnitude estimation is a fundamentally different type of measurement task.…”

“…Featherston 2008, Myers 2009, Bader & Häussler 2010, Weskott & Fanselow 2011. However, because such comparisons are dependent on the sentence types chosen, the sample sizes tested, and the statistical tests employed, it is logically possible that future experiments could reveal that magnitude estimation does indeed yield superior data.…”

“…This raises the possibility that a participant may wish to distinguish more levels of acceptability than the scale allows. Whether this is a problem in practice is still a matter of debate given the standard practice of averaging across samples of participants, though several recent studies have demonstrated that there is no obvious sensitivity difference between ME and other tasks, at least for the constructions and sample sizes investigated (Myers 2009, Bader & Häus-sler 2010, Weskott & Fanselow 2011.…”