Perspectives on crowdsourcing annotations for natural language processing

Abstract

Crowdsourcing has emerged as a new method for obtaining annotations for training models for machine learning. While many variants of this process exist, they largely differ in their method of motivating subjects to contribute and the scale of their applications. To date, however, there has yet to be a study that helps the practitioner to decide what form an annotation application should take to best reach its objectives within the constraints of a project. We first provide a faceted analysis of existing crowdsourcing annotation applications. We then use our analysis to discuss our recommendations on how practitioners can take advantage of crowdsourcing and discuss our view on potential opportunities in this area.

Assisting non-expert speakers of under-resourced languages in assigning stems and inflectional paradigms to new word entries of morphological dictionaries

“…As already pointed out, this approach could be integrated into a rule-based MT system and users could be asked to help insert the words into the sentences to be translated that are not found in the system's dictionaries. Moreover, users could also be contacted by means of a crowdsourcing platform (Wang et al, 2013).…”

“…Ambati et al (2010) propose asking non-expert bilingual informants to translate SL sentences in order to create a parallel corpus from which a statistical MT system (Koehn, 2010) is eventually built. Users interact through a crowdsourcing platform (Wang et al, 2013). An efficient active learning strategy (Haffari et al, 2009) is critical in this scenario: the SL sentences to be translated by the users should be those that, when included in the parallel corpus, cause the largest possible increase in the performance of the resulting statistical MT system.…”

“…For instance, other non-expert individuals, who are not users of the MT system, can be recruited by means of crowdsourcing platforms (Wang et al, 2013) to collaboratively perform the task of inserting new entries into morphological dictionaries. Moreover, linguists themselves can also benefit from the approach because validating inflected forms of words may often be faster than choosing from among a list of paradigms.…”

Assessing OpenStreetMap Data Using Intrinsic Quality Indicators: An Extension to the QGIS Processing Toolbox

“…For measuring the similarity or lexicographical error between data, the text similarity approach is used. The results obtained for attribute accuracy are presented as the percentage correctness of numerical or text-based values associated with an attribute [34,46].…”

Influence of Weak Labels for Emotion Recognition of Tweets

“…Current state-of-the-art emotion recognition techniques use lexicon based methods in order to classify text [10], [16]. Lexicon based techniques use dictionaries of words with pre-computed scores expressing the emotional value of the word.…”