Ethical Considerations in NLP Shared Tasks

Abstract

Shared tasks are increasingly common in our field, and new challenges are suggested at almost every conference and workshop. However, as this has become an established way of pushing research forward, it is important to discuss how we researchers organise and participate in shared tasks, and make that information available to the community to allow further research improvements. In this paper, we present a number of ethical issues along with other areas of concern that are related to the competitive nature of shared tasks. As such issues could potentially impact on research ethics in the Natural Language Processing community, we also propose the development of a framework for the organisation of and participation in shared tasks that can help mitigate against these issues arising.

Proceedings of the Fifth Workshop on Computational Linguistics and Clinical Psychology: From Keyboard to Clinic

“…For future work, since the Socio Demographic performed best, we could apply methods such as User-Factor Adaptation which focus on the author of the content in addition to the content (Lynn et al, 2017;Zhu et al, 2018). It would also be interesting to investigate if word clusters trained on historical sources (for e.g.…”

“…For example, Homan et al (2014) found that two novice annotators were more likely to assign their expert's "low distress" tweets to the "no distress" category. Conversely, on a related but coarser-grained categorization task, Liu et al (2017) find "some evidence that multiple crowdsourcing workers, when they reach high inter-annotator agreement, can provide reliable quality of annotations". …”

“…defining a positive instance only as someone whose risk level is moderate or severe, which is why we limit our universe population here to those for whom we have risk ratings. See also Liu et al (2017) on aggregation of annotator labels for supervised learning in this domain. lecting sets of posts starting 7, 5, 2, and 1 day before that one.…”

“…Nguyen et al (2011) also use regression to predict author age, using l1 regularisation to induce a sparser model, selecting a subset of informative features to further analyse. Ethical considerations are also extremely important in clinical NLP (Suster et al, 2017), health research and shared tasks in general (Parra Escartín et al, 2017). While (mercifully) much of the data access logistics for the CLPsych18 shared task were handled by the organisers, it is still critical to consider how raw data and interim results are distributed amongst the team, whether models perform unusually poorly for subsets of participants, potential dual use of any developed technology, and suitability of different deployment techniques.…”

Towards Replicability in Parsing

“…Replicability has started to appear as a topic in NLP and machine learning, for example as an IJ-CAI 2015 Workshop on Replicability and Reproducibility in Natural Language Processing 1 , and it has been described as one of the potentially negative factors of shared tasks in NLP by Parra Escartín et al (2017). However, there is little work on more specific areas such as parsing.…”