Shared tasks are increasingly common in our field, and new challenges are suggested at almost every conference and workshop. However, as this has become an established way of pushing research forward, it is important to discuss how we researchers organise and participate in shared tasks, and make that information available to the community to allow further research improvements. In this paper, we present a number of ethical issues along with other areas of concern that are related to the competitive nature of shared tasks. As such issues could potentially impact on research ethics in the Natural Language Processing community, we also propose the development of a framework for the organisation of and participation in shared tasks that can help mitigate against these issues arising.

Machine Translation (MT) quality is typically assessed using automatic evaluation metrics such as BLEU and TER. Despite being generally used in the industry for evaluating the usefulness of Translation Memory (TM) matches based on text similarity, fuzzy match values are not as widely used for this purpose in MT evaluation. We designed an experiment to test if this fuzzy score applied to MT output stands up against traditional methods of MT evaluation. The results obtained seem to confirm that this metric performs at least as well as traditional methods for MT evaluation.

No abstract

This paper reports on the organization and results of the rst Automatic\ud Translation Memory Cleaning Shared Task. This shared task is aimed\ud at nding automatic ways of cleaning translation memories (TMs) that have\ud not been properly curated and thus include incorrect translations. As a follow\ud up of the shared task, we also conducted two surveys, one targeting the teams\ud participating in the shared task, and the other one targeting professional translators.\ud While the researchers-oriented survey aimed at gathering information\ud about the opinion of participants on the shared task, the translators-oriented\ud survey aimed to better understand what constitutes a good TM unit and inform\ud decisions that will be taken in future editions of the task. In this paper, we\ud report on the process of data preparation and the evaluation of the automatic\ud systems submitted, as well as on the results of the collected surveys

Normes et usages en anglais de spécialité La traduction automatique comme outil d'aide à la rédaction scientifique en anglais langue seconde : résultats d'une étude exploratoire sur la qualité linguistique Using machine translation for academic writing in English as a second language: results of an exploratory study on linguistic quality

Post-Editing of Machine Translation (MT) has become a reality in professional translation workflows. In order to optimize the management of projects that use post-editing and avoid underpayments and mistrust from professional translators, effective tools to assess the quality of Machine Translation (MT) systems need to be put in place. One field of study that could address this problem is Machine Translation Quality Estimation (MTQE), which aims to determine the quality of MT without an existing reference. Accurate and reliable MTQE can help project managers and translators alike, as it would allow estimating more precisely the cost of post-editing projects in terms of time and adequate fares by discarding those segments that are not worth post-editing (PE) and have to be translated from scratch.In this paper, we report on the results of an impact study which engages professional translators in PE tasks using MTQE. We measured translators' productivity in different scenarios: translating from scratch, post-editing without using MTQE, and post-editing using MTQE. Our results show that QE information, when accurate, improves post-editing efficiency.

No abstract

When a sudden-onset emergency occurs, the language needs of those affected and those involved in the relief efforts cannot be foreseen. Provided that access to online communication is still available, it is not unlikely that many involved in the crisis will resort to language technologies such as machine translation and initiatives such as crowdsourcing to assist in the urgent need for multilingual communication. This may be done in an attempt to understand the key messages from official bodies, or relief organisations, when there is a lack of professional translators to assist in the multilingual communication process. This approachmachine translation and crowdsourcing -was successfully used in a previous crisis, i.e. the 2010 earthquake in Haiti. However, the use of technologies cannot be taken for granted. Even if they are supposedly used for good, a number of ethical issues should be given consideration before using these technologies, when using them, and in the aftermath of a crisis. In this chapter, we describe those issues by having a closer look at potential crisis translation workflows which rely on machine translation and crowdsourcing.