Crowdsourcing the acquisition of natural language corpora: Methods and observations

Abstract

We study the opportunity for using crowdsourcing methods to acquire language corpora for use in natural language processing systems. Specifically, we empirically investigate three methods for eliciting natural language sentences that correspond to a given semantic form. The methods convey frame semantics to crowd workers by means of sentences, scenarios, and list-based descriptions. We discuss various performance measures of the crowdsourcing process, and analyze the semantic correctness, naturalness, and biases of the collected language. We highlight research challenges and directions in applying these methods to acquire corpora for natural language processing applications.

Proceedings of the 9th International Natural Language Generation conference

“…Such a workload distribution was previously described in (Wang et al, 2012) as appropriate for a betweensubject design. Each batch corresponded to one of two conditions: the first batch contained only textual/logical MRs, and the second one used only pictorial MRs.…”

“…For example, (Zaidan and Callison-Burch, 2011) showed that crowdsourcing can result in datasets of comparable quality to those created by professional translators given appropriate quality control methods. (Mairesse et al, 2010) demonstrate that crowd workers can produce NL descriptions from abstract MRs, a method which also has shown success in related NLP tasks, such as Spoken Dialogue Systems (Wang et al, 2012) or Semantic Parsing . However, when collecting corpora for training NLG systems, new challenges arise: (1) How to ensure the required high quality of the collected data?…”

“…Williams and Young, 2007) for evaluating spoken dialogue systems. We compare these pictorial MRs to text-based MRs used by previous crowd-sourcing work (Mairesse et al, 2010;Wang et al, 2012). These text-based MRs take the form of Dialogue Acts, such as inform(type [hotel],pricerange[expensive]).…”

NLify

“…Crowdsourcing services such as Amazon Mechanical Turk have been used to collect paraphrase sets that serve as NLP benchmarks [8,9,24]. Essentially, workers worldwide are paid tiny amounts to paraphrase individual example sentences or concepts.…”

Spoken dialogue grammar induction from crowdsourced data

“…Crowdsourcing is a very popular method for various natural language and speech processing tasks [9,10,11]. Examples include sentence translation from one language to another or gathering annotations on bilingual lexical entries [12,13], as well as paraphrasing applications [14,15].…”