VERTa: a Linguistically-motivated Metric at the WMT15 Metrics Task

Comelles, Elisabet; Atserias, Jordi

doi:10.18653/v1/w15-3045

“…Metric Participant BEER, BEER TREEPEL ILLC -University of Amsterdam (Stanojević and Sima'an, 2015) BS University of Zurich (Mark Fishel; no corresponding paper) CHRF, CHRF3 DFKI (Popović, 2015) DPMF, DPMFCOMB Chinese Academy of Sciences and Dublin City University (Yu et al, 2015) DREEM National Research Council Canada (Chen et al, 2015) LEBLEU-DEFAULT, LEBLEU-OPTIMIZED Lingsoft and Aalto University (Virpioja and Grönroos, 2015) METEOR-WSD, RATATOUILLE LIMSI-CNRS (Marie and Apidianaki, 2015) UOW-LSTM University of Wolverhampton (Gupta et al, 2015a) UPF-COBALT Universitat Pompeu Fabra (Fomicheva et al, 2015) USAAR-ZWICKEL-* Saarland University (Vela and Tan, 2015) VERTA-W, VERTA-EQ, VERTA-70ADEQ30FLU University of Barcelona (Comelles and Atserias, 2015) Table 1: Participants of WMT15 Metrics Shared Task…”

Section: Manual Mt Quality Judgementsmentioning

confidence: 99%

Results of the WMT15 Metrics Shared Task

Stanojević¹,

Kamran²,

Koehn³

et al. 2015

Proceedings of the Tenth Workshop on Statistical Machine Translation

View full text Add to dashboard Cite

This paper presents the results of the WMT13 Metrics Shared Task. We asked participants of this task to score the outputs of the MT systems involved in WMT13 Shared Translation Task. We collected scores of 16 metrics from 8 research groups. In addition to that we computed scores of 5 standard metrics such as BLEU, WER, PER as baselines. Collected scores were evaluated in terms of system level correlation (how well each metric's scores correlate with WMT13 official human scores) and in terms of segment level correlation (how often a metric agrees with humans in comparing two translations of a particular sentence).

show abstract

“…Measuring semantic similarity between words, terms, sentences, paragraphs, and documents is a vital aspect of natural language processing and computational linguistics [3]. It has numerous applications, including question answering systems, machine translation, information retrieval, fraud detection, and more [4][5][6][7][8]. In this article, we explore various techniques for determining semantic similarity across languages and introduce an improvement by proposing an ensemble method based on transformer models.…”

Section: Introductionmentioning

confidence: 99%

Ensemble Transformer for cross lingual semantic textual similarity

Piroozfar,

Abdous,

Bidgoli

2024

Preprint

0

View full text Add to dashboard Cite

Today, it is particularly important to recognize the semantic similarity between texts in different languages due to the emergence of new natural language processing models like ChatGPT and Bard. These models can provide more accurate and comprehensive answers to users' questions by identifying semantic similarity between two texts in different languages. Cross-lingual semantic similarity refers to the process of calculating similarity between two pieces of text in different languages. This paper aims to present an improved method for finding similarities between sentences in different languages. Some of the current methods create the same vector space to achieve this, while others use machine translation to translate the text into another language and then determine similarity between the two sentences using monolingual sentence similarity methods. The degree of similarity is expressed as a number between 0 and 5. Over the past few years, the progress in language models based on transformers has paved the way for improvements in detecting text similarity. This article discusses the utilization of ensemble models with transformers to determine the semantic similarity of sentences in Persian and English languages utilizing the Persian-English corpus. According to our findings, this ensemble approach has a correlation rate of 95.28% in detecting the extent of semantic similarity between cross-lingual sentences. These results indicate that our method surpasses previous techniques for discovering similarities between sentences in different languages.

show abstract

“…Measuring semantic similarity between words, terms, sentences, paragraphs, and documents is a vital aspect of natural language processing and computational linguistics [3]. It has numerous applications, including question answering systems, machine translation, information retrieval, fraud detection, and more [4][5][6][7][8]. In this article, we explore various techniques for determining semantic similarity across languages and introduce an improvement by proposing an ensemble method based on transformer models.…”

Section: Introductionmentioning

confidence: 99%

An ensemble method for cross lingual semantic textual similarity

Piroozfar,

Abdous,

Bidgoli

2023

Preprint

0

View full text Add to dashboard Cite

Today, it is particularly important to recognize the semantic similarity between texts in different languages due to the emergence of new natural language processing models like ChatGPT and Bard. These models can provide more accurate and comprehensive answers to users' questions by identifying semantic similarity between two texts in different languages. Cross-lingual semantic similarity refers to the process of calculating similarity between two pieces of text in different languages. This paper aims to present an improved method for finding similarities between sentences in different languages. Some of the current methods create the same vector space to achieve this, while others use machine translation to translate the text into another language and then determine similarity between the two sentences using monolingual sentence similarity methods. The degree of similarity is expressed as a number between 0 and 5. Over the past few years, the progress in language models based on transformers has paved the way for improvements in detecting text similarity. This article discusses the utilization of ensemble models with transformers to determine the semantic similarity of sentences in Persian and English languages utilizing the Persian-English corpus. According to our findings, this ensemble approach has a correlation rate of 95.28% in detecting the extent of semantic similarity between cross-lingual sentences. These results indicate that our method surpasses previous techniques for discovering similarities between sentences in different languages.

show abstract

VERTa: a Linguistically-motivated Metric at the WMT15 Metrics Task

Cited by 6 publications

References 9 publications

Results of the WMT15 Metrics Shared Task

Results of the WMT15 Metrics Shared Task

Ensemble Transformer for cross lingual semantic textual similarity

An ensemble method for cross lingual semantic textual similarity

Contact Info

Product

Resources

About