Ebaluatoia: crowd evaluation for English–Basque machine translation

Aranberri, Nora; Labaka, Gorka; Ilarraza, Arantza Díaz de; Sarasola, Kepa

doi:10.1007/s10579-016-9335-x

Cited by 5 publications

(6 citation statements)

References 28 publications

(21 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…2 Their system was tested on ranking documents by their reading difficulty. Relative preference annotations have also been heavily employed in assessing machine translation (Aranberri et al, 2016). By contrast to our work, the underlying relations (reading difficulty 1-12 or better translation) have well known properties of total ordering, while convincingness of arguments is a yet unexplored task, thus no assumptions can be made apriori.…”

Section: Related Workmentioning

confidence: 92%

Which argument is more convincing? Analyzing and predicting convincingness of Web arguments using bidirectional LSTM

Habernal¹,

Gurevych²

2016

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

158

185

View full text Add to dashboard Cite

We propose a new task in the field of computational argumentation in which we investigate qualitative properties of Web arguments, namely their convincingness. We cast the problem as relation classification, where a pair of arguments having the same stance to the same prompt is judged. We annotate a large datasets of 16k pairs of arguments over 32 topics and investigate whether the relation "A is more convincing than B" exhibits properties of total ordering; these findings are used as global constraints for cleaning the crowdsourced data. We propose two tasks: (1) predicting which argument from an argument pair is more convincing and (2) ranking all arguments to the topic based on their convincingness. We experiment with feature-rich SVM and bidirectional LSTM and obtain 0.76-0.78 accuracy and 0.35-0.40 Spearman's correlation in a cross-topic evaluation. We release the newly created corpus UKPConvArg1 and the experimental software under open licenses.

show abstract

Section: Related Workmentioning

confidence: 92%

Which argument is more convincing? Analyzing and predicting convincingness of Web arguments using bidirectional LSTM

Habernal¹,

Gurevych²

2016

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

158

185

View full text Add to dashboard Cite

show abstract

“…Indeed, a comparative approach is widely used in many NLP applications, e.g. in Chen et al (2013) for assessing reading difficulty of documents and in Aranberri et al (2017) for machine translation. In light of these considerations, here we decided to investigate and compare both approaches.…”

Section: Approaches To Argument Quality Labelingmentioning

confidence: 99%

Automatic Argument Quality Assessment - New Datasets and Methods

Toledo¹,

Gretz²,

Cohen-Karlik³

et al. 2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

View full text Add to dashboard Cite

We explore the task of automatic assessment of argument quality. To that end, we actively collected 6.3k arguments, more than a factor of five compared to previously examined data. Each argument was explicitly and carefully annotated for its quality. In addition, 14k pairs of arguments were annotated independently, identifying the higher quality argument in each pair. In spite of the inherent subjective nature of the task, both annotation schemes led to surprisingly consistent results. We release the labeled datasets to the community. Furthermore, we suggest neural methods based on a recently released language model, for argument ranking as well as for argument-pair classification. In the former task, our results are comparable to state-of-the-art; in the latter task our results significantly outperform earlier methods. * These authors equally contributed to this work. 1 For more details:https://www.research. ibm.com/artificial-intelligence/ project-debater/live/

show abstract

“…Crowdsourcing is a method by which researchers acquire services by asking for contributions from a large number of people, often from an online community (Chandler et al 2019;Chmielewski and Kucker 2019;Ortega-Santos 2019). Some crowdsourced language studies have detected professional respondents in all languages, including Spanish (Aranberri et al 2017;Ortega-Santos 2019). Our article contributes to a gap in the literature relating to Spanish-language responses to online opt-in panels as compared to English-speaking respondents.…”

Section: Literature Reviewmentioning

confidence: 91%

Identifying Data Quality Challenges in Online Opt-In Panels Using Cognitive Interviews in English and Spanish

Trejo

Meyers

Martinez

et al. 2022

Journal of Official Statistics

View full text Add to dashboard Cite

In this article, we evaluate how the analysis of open-ended probes in an online cognitive interview can serve as a metric to identify cases that should be excluded due to disingenuous responses by ineligible respondents. We analyze data collected in 2019 via an online opt-in panel in English and Spanish to pretest a public opinion questionnaire (n = 265 in English and 199 in Spanish). We find that analyzing open-ended probes allowed us to flag cases completed by respondents who demonstrated problematic behaviors (e.g., answering many probes with repetitive textual patterns, by typing random characters, etc.), as well as to identify cases completed by ineligible respondents posing as eligible respondents (i.e., non-Spanish-speakers posing as Spanish-speakers). These findings indicate that data collected for multilingual pretesting research using online opt-in panels likely require additional evaluations of data quality. We find that open-ended probes can help determine which cases should be replaced when conducting pretesting using opt-in panels. We argue that open-ended probes in online cognitive interviews, while more time consuming and expensive to analyze than close-ended questions, serve as a valuable method of verifying response quality and respondent eligibility, particularly for researchers conducting multilingual surveys with online opt-in panels.

show abstract

Ebaluatoia: crowd evaluation for English–Basque machine translation

Cited by 5 publications

References 28 publications

Which argument is more convincing? Analyzing and predicting convincingness of Web arguments using bidirectional LSTM

Which argument is more convincing? Analyzing and predicting convincingness of Web arguments using bidirectional LSTM

Automatic Argument Quality Assessment - New Datasets and Methods

Identifying Data Quality Challenges in Online Opt-In Panels Using Cognitive Interviews in English and Spanish

Contact Info

Product

Resources

About