A streamlined approach to online linguistic surveys

Abstract

More and more researchers in linguistics use large-scale experiments to test hypotheses about the data they research, in addition to more traditional informant work. In this paper we describe a new set of free, open-source tools that allow linguists to post studies online, turktools. These tools allow for the creation of a wide range of linguistic tasks, including grammaticality surveys, sentence completion tasks, and picture-matching tasks, allowing for easily implemented largescale linguistic studies. Our tools further help streamline the design of such experiments and assist in the extraction and analysis of the resulting data. Surveys created using the tools described in this paper can be posted on Amazon's Mechanical Turk service, a popular crowdsourcing platform that mediates between 'Requesters' who can post surveys online and 'Workers' who complete them. This allows many linguistic surveys to be completed within hours or days and at relatively low costs. Alternatively, researchers can host these randomized experiments on their own servers using a supplied server-side component.

Quantifying sentence acceptability measures: Reliability, bias, and variability

“…In this procedure, adapted from psychophysics (Stevens 1956), participants are given an initial reference item to calibrate their judgments and then asked to compare other items by assigning them any positive real number. Although unable to provide true ratio data as initially claimed (Sprouse 2008;Weskott & Fanselow 2008;Sprouse 2011a), ME is still commonly used (Cowart 1997;Keller 2003;Featherston 2005;Murphy et al 2006;Johnson 2011;Schütze 2011;Erlewine & Kotek 2016). Interpreted as a linear scaling task rather than a direct recording of people's mental representations, it is distinct from other measures in the extreme freedom it gives for arbitrarily precise responses, although whether that extra variability actually encodes information about linguistic effects has been questioned (Weskott & Fanselow 2011).…”

Syntax

“…Experimental approaches are sometimes necessary. [17][18][19] Similar comments are true for corpus data, which have advantages in providing frequency information, but disadvantages in the rarity of the crucial examples. 14 The lack of statistical and other information about data provenance and analysis does raise a communication difficulty: psychologists looking at syntax papers and seeking the kind of statistical information they are used to in their own discipline will generally find it lacking.…”

“…This can be tackled via more careful design of the materials to weed out pragmatic or semantic factors that interfere but sometimes data will simply not be considered to have a strong enough effect size to be used in theoretical or analytical argumentation. Experimental approaches are sometimes necessary . Similar comments are true for corpus data, which have advantages in providing frequency information, but disadvantages in the rarity of the crucial examples …”

Breaking de Morgan's law in counterfactual antecedents

“…We implemented our experiments and tests as web surveys using TurkTools (Erlewine & Kotek 2016), which relies on the platform Amazon Mechanical Turk (http://www.mturk.com). Participants were required to be located in the U.S. and have a Mechanical Turk approval rate (an indication of reliability) of at least 95%.…”