Asiya: An Open Toolkit for Automatic Machine Translation (Meta-)Evaluation

Giménez, Jesús; Màrquez, Lluı́s

doi:10.2478/v10108-010-0022-6

Cited by 49 publications

(34 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

Section: Introductionmentioning

confidence: 99%

“…1 It provides a rich set of heterogeneous metrics and tools to evaluate and analyse the quality of automatic translations. The ASIYA core toolkit was first released in 2009 (Giménez andMàrquez, 2010a) and has been continuously improved and extended since then (Gonzàlez et al, 2012;Gonzàlez et al, 2013).In this paper we first describe the most recent enhancements to ASIYA: (i) linguistic-based metrics for French and German; (ii) an extended set of source-based metrics for English, Spanish, German, French, Russian, and Czech; and (iii) the integration of mechanisms to exploit the alignments between sources and translations. These enhancements are all available in ASIYA since version 3.0.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Proceedings of the Ninth Workshop on Statistical Machine Translation

Bojar

Buck

Federmann

et al. 2014

View full text Add to dashboard Cite

The focus of our workshop was to use parallel corpora for machine translation. Recent experimentation has shown that the performance of SMT systems varies greatly with the source language. In this workshop we encouraged researchers to investigate ways to improve the performance of SMT systems for diverse languages, including morphologically more complex languages, languages with partial free word order, and low-resource languages.Prior to the workshop, in addition to soliciting relevant papers for review and possible presentation, we conducted four shared tasks: a general translation task, a medical translation task, a quality estimation task, and a task to test automatic evaluation metrics. The medical translation task was introduced this year to address the important issue of domain adaptation within SMT. The results of the shared tasks were announced at the workshop, and these proceedings also include an overview paper for the shared tasks that summarizes the results, as well as provides information about the data used and any procedures that were followed in conducting or scoring the task. In addition, there are short papers from each participating team that describe their underlying system in greater detail.Like in previous years, we have received a far larger number of submission than we could accept for presentation. This year we have received 27 full paper submissions and 49 shared task submissions. In total WMT 2014 featured 12 full paper oral presentations and 49 shared task poster presentations.The invited talk was given by Alon Lavie (Carnegie Mellon University and Safaba Translation Solutions, Inc.) entitled "Machine Translation in Academia and in the Commercial World -a Contrastive Perspective".We would like to thank the members of the Program Committee for their timely reviews. We also would like to thank the participants of the shared task and all the other volunteers who helped with the evaluations. is a ranking of the systems that participated in its shared translation tasks, produced by aggregating pairwise sentencelevel comparisons collected from human judges. Over the past few years, there have been a number of tweaks to the aggregation formula in attempts to address issues arising from the inherent ambiguity and subjectivity of the task, as well as weaknesses in the proposed models and the manner of model selection.We continue this line of work by adapting the TrueSkill TM algorithm -an online approach for modeling the relative skills of players in ongoing competitions, such as Microsoft's Xbox Live -to the human evaluation of machine translation output. Our experimental results show that TrueSkill outperforms other recently proposed models on accuracy, and also can significantly reduce the number of pairwise annotations that need to be collected by sampling non-uniformly from the space of system competitions. IntroductionThe Workshop on Statistical Machine Translation (WMT) has long been a central event in the machine translation (MT) community for the evaluation of MT output. It hosts...

show abstract

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

Proceedings of the Ninth Workshop on Statistical Machine Translation

Bojar

Buck

Federmann

et al. 2014

View full text Add to dashboard Cite

show abstract

“…DPMFcomb allows the combination of the advantages of a set of arbitrary metrics resulting in a metric with a high correlation with human assessment. DPMFcomb includes default metrics provided by Asiya MT evaluation toolkit (Giménez and Màrquez, 2010), as well as three other metrics, namely ENTF (Yu et al, 2015c), REDp (Yu et al, 2014) and DPMF (Yu et al, 2015b). Over the past two years of WMT metrics tasks, DPMFcomb has achieved the best performance for evaluation of MT of to-English language pairs.…”

Section: Metrics 21 Review Of Dpmfcombmentioning

confidence: 99%

Blend: a Novel Combined MT Metric Based on Direct Assessment — CASICT-DCU submission to WMT17 Metrics Task

Ma¹,

Graham²,

Wang³

et al. 2017

Proceedings of the Second Conference on Machine Translation

View full text Add to dashboard Cite

Existing metrics to evaluate the quality of Machine Translation hypotheses take different perspectives into account. DPMFcomb, a metric combining the merits of a range of metrics, achieved the best performance for evaluation of to-English language pairs in the previous two years of WMT Metrics Shared Tasks. This year, we submit a novel combined metric, Blend, to WMT17 Metrics task. Compared to DPMFcomb, Blend includes the following adaptations: i) We use DA human evaluation to guide the training process with a vast reduction in required training data, while still achieving improved performance when evaluated on WMT16 to-English language pairs; ii) We carry out experiments to explore the contribution of metrics incorporated in Blend, in order to find a trade-off between performance and efficiency.

show abstract

“…The Asiya Toolkit (Giménez et al, 2010) is a visualisation tool that can be used online or as a stand-alone tool. It offers a comprehensive suite of metrics, including many linguistically motivated ones.…”

Section: Related Toolsmentioning

confidence: 99%

Vis-Eval Metric Viewer: A Visualisation Tool for Inspecting and Evaluating Metric Scores of Machine Translation Output

Steele¹,

Specia²

2018

Proceedings of the 2018 Conference of the North American Chapter Of the Association for Computational Linguistics: De

View full text Add to dashboard Cite

Machine Translation systems are usually evaluated and compared using automated evaluation metrics such as BLEU and METEOR to score the generated translations against human translations. However, the interaction with the output from the metrics is relatively limited and results are commonly a single score along with a few additional statistics. Whilst this may be enough for system comparison it does not provide much useful feedback or a means for inspecting translations and their respective scores. Vis-Eval Metric Viewer (VEMV) is a tool designed to provide visualisation of multiple evaluation scores so they can be easily interpreted by a user. VEMV takes in the source, reference, and hypothesis files as parameters, and scores the hypotheses using several popular evaluation metrics simultaneously. Scores are produced at both the sentence and dataset level and results are written locally to a series of HTML files that can be viewed on a web browser. The individual scored sentences can easily be inspected using powerful search and selection functions and results can be visualised with graphical representations of the scores and distributions.

show abstract

Asiya: An Open Toolkit for Automatic Machine Translation (Meta-)Evaluation

Cited by 49 publications

References 22 publications

Proceedings of the Ninth Workshop on Statistical Machine Translation

Proceedings of the Ninth Workshop on Statistical Machine Translation

Blend: a Novel Combined MT Metric Based on Direct Assessment — CASICT-DCU submission to WMT17 Metrics Task

Vis-Eval Metric Viewer: A Visualisation Tool for Inspecting and Evaluating Metric Scores of Machine Translation Output

Contact Info

Product

Resources

About