As Little as Possible, as Much as Necessary: Detecting Over- and Undertranslations with Contrastive Conditioning

Vamvas, Jannis; Sennrich, Rico

doi:10.18653/v1/2022.acl-short.53

Cited by 5 publications

(2 citation statements)

References 83 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These, however, were shown not to be indicative of hallucinations (Guerreiro et al, 2023), which highlights the importance of our human-annotated data. For omissions, previous work mostly focused on empty translations (Stahlberg and Byrne, 2019;Vijayakumar et al, 2016) with some work using artificially created undertranslations (Vamvas and Sennrich, 2022). As we saw in Section 6, the latter is unlikely to be helpful when evaluating detection methods.…”

Section: Additional Related Workmentioning

confidence: 99%

See 1 more Smart Citation

HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translation

Dale,

Voita,

Lam

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Hallucinations in machine translation are translations that contain information completely unrelated to the input. Omissions are translations that do not include some of the input information. While both cases tend to be catastrophic errors undermining user trust, annotated data with these types of pathologies is extremely scarce and is limited to a few high-resource languages. In this work, we release an annotated dataset for the hallucination and omission phenomena covering 18 translation directions with varying resource levels and scripts. Our annotation covers different levels of partial and full hallucinations as well as omissions both at the sentence and at the word level. Additionally, we revisit previous methods for hallucination and omission detection, show that conclusions made based on a single language pair largely do not hold for a large-scale evaluation, and establish new solid baselines.

show abstract

Section: Additional Related Workmentioning

confidence: 99%

“…In contrast to sentence-level detection, detecting pathologies at the word level received much less attention. In terms of both available data and detectors, previous attempts were rather limited (Zhou et al, 2021;Vamvas and Sennrich, 2022). Here, we want to facilitate future research in this direction.…”

Section: Word-level Detectionmentioning

confidence: 99%

HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translation

Dale,

Voita,

Lam

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

Automatic Reference-Free Fine-Grained Machine Translation Error Detection via Named Entity Recognition and Back-Translation

Yan,

Song,

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Learning From Crowds With Contrastive Representation

Pedrycz

2023

IEEE Access

View full text Add to dashboard Cite

Crowdsourcing provides a practical approach to obtaining annotated data for data-hungry deep models. Due to its simplicity and practicality, simultaneously learning the annotation correction mechanism and the target classifier is widely studied and applied. Existing work has improved performance from the annotator and annotation process modeling perspective. However, the instance representation, which most directly affects model training, has been neglected. In this work, we investigate contrastive representation to improve learning from crowds. Specifically, we first sample confident instances and positive pairs using the pre-trained representation and human annotations. Then, we extend the supervised contrastive loss to obtain a noise-tolerant version that supports continuous consistency between labels. After that, we leverage the learned representations to train the classifier and annotator parameters. The process is generally designed as an end-to-end framework, CrowdCons, compatible with existing crowdsourcing models. Our approach is evaluated on three real-world crowdsourcing datasets; LabelMe, CIFAR10-H and Music. The experimental results show that it can significantly improve prediction accuracy, and the case study demonstrates the robustness of the model regarding noisy annotations.

show abstract

As Little as Possible, as Much as Necessary: Detecting Over- and Undertranslations with Contrastive Conditioning

Cited by 5 publications

References 83 publications

HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translation

HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translation

Automatic Reference-Free Fine-Grained Machine Translation Error Detection via Named Entity Recognition and Back-Translation

Learning From Crowds With Contrastive Representation

Contact Info

Product

Resources

About