{bs,hr,sr}WaC - Web Corpora of Bosnian, Croatian and Serbian

This paper presents a quantitative fine-grained manual evaluation approach to comparing the performance of different machine translation (MT) systems. We build upon the well-established Multidimensional Quality Metrics (MQM) error taxonomy and implement a novel method that assesses whether the differences in performance for MQM error types between different MT systems are statistically significant. We conduct a case study for English-toCroatian, a language direction that involves translating into a morphologically rich language, for which we compare three MT systems belonging to different paradigms: pure phrase-based, factored phrase-based and neural. First, we design an MQM-compliant error taxonomy tailored to the relevant linguistic phenomena of Slavic languages, which made the annotation process feasible and accurate. Errors in MT outputs were then annotated by two annotators following this taxonomy. Subsequently, we carried out a statistical analysis which showed that the best-performing system (neural) reduces the errors produced by the worst system (pure phrase-based) by more than half (54%). Moreover, we conducted an additional analysis of agreement errors in which we distinguished between short (phrase-level) and long distance (sentence-level) errors. We discovered that phrase-based MT approaches are of limited use for long distance agreement phenomena, for which neural MT was found to be especially effective.

show abstract

“…To this end we concatenated the hrWaC corpus (Ljubešić and Klubička, 2014) with the target side of the aforementioned parallel corpora.…”

Section: Mt Systems and Datasetsmentioning

confidence: 99%

Quantitative fine-grained human evaluation of machine translation systems: a case study on English to Croatian

Klubička¹,

Toral

Sánchez-Cartagena³

2018

Machine Translation

Self Cite

View full text Add to dashboard Cite

show abstract

“…Baza po~iva na empirijski utvr|enim vrijednostima za kategorije konkretnosti, predo~ivosti, subjektivne ~esto}e i dobi usvajanja 3.000 leksema (imenica, glagola i pridjeva) hrvatskoga jezika iz korpusa hrWaC (Ljube{i} i Klubi~ka 2014). Kako slu~ajnim odabirom leksema iz korpusa nisu obuhva}eni mnogi naj~e{}i leksemi hrvatskoga jezika, baza }e tijekom 2018. biti nadopunjena s jo{ 1.500 naj~e{}ih leksema iz Hrvatskog ~estotnog rje~nika (Mogu{, Bratani} i Tadi} 1999) za koje jo{ nisu prikupljene procjene promatranih kategorija i jo{ 1.500 leksema ekscerpiranih iz ud`benika za hrvatski jezik, prirodu, matematiku, povijest i geografiju za 4., 5. i 6. razred osnovne {kole najzastupljenijih u izboru profesora koji ih upotrebljavaju u nastavi.…”

Section: Uvodunclassified

Psycholinguistic estimates of 3000 words of Croatian: concreteness and imageability

Stantić¹,

Anđel²,

Keresteš³

et al. 2018

suvlin

Self Cite

View full text Add to dashboard Cite

“…In the Croatian National Corpus 3.0 (CNC 3.0) [27] which contains about 234,000,000 tokens, we found altogether 2,908,182 occurrences of the identified homograms and 2,488 occurrences of unique homograms, which make up 72% of all homograms that we obtained in the lexicon. The Croatian web corpus hrWaC [28] is a web corpus that contains 1.9 billion tokens and is annotated with the lemma, morphosyntax, and dependency syntax layers. There were 20,694,430 total occurrences of the homograms and 3,180 of unique homograms (92% of the homograms from the lexicon) identified in the hrWaC corpus.…”

Section: A Identifying and Disambiguating Homograms In Corpusmentioning

confidence: 99%

The Role of Homograms in Machine Translation

Prskalo¹,

Bakarić²

2018

IJMLC

View full text Add to dashboard Cite

Abstract-TheCroatian language is a pitch-accent language, in which the tone contour realized in the stressed syllable carries the lexical information. Therefore, in some cases, a different lexical accent gives the word a different meaning. In such cases, the ambiguity of the word in written texts, where accents are not usually marked, can be solved by determining the appropriate accent. There are also cases when various basic and derived forms of words have different meanings, different morphosyntactic descriptions (MSDs), and possibly different accents. When words have the same written forms but different meanings, they are called homograms. In order to resolve the ambiguity of homograms, we created a lexicon of homograms that is comprised of all Croatian nouns of different gender, which have the same written forms (if accents are not marked) but different meanings, MSDs, and possibly different accents. This lexicon consists of 19,366 entries and 3,460 unique homograms. Each entry in the lexicon comprises the homogram (unaccented word), the accented word, the corresponding MSD, and the accented lemma. The obtained lexicon enables us to identify and disambiguate homograms within the corpus efficiently and accurately. We also evaluated and analyzed the performance of machine translation (MT) systems for the Croatian-English language pair with a special emphasis on homogram translation. We confirmed that the disambiguation of homograms can improve the performance of MT systems in avoiding major translation mistakes related to assigning the wrong meaning to homograms.Index Terms-Disambiguation of homograms, lexicon of homograms, pitch accent language, word sense disambiguation.

show abstract

{bs,hr,sr}WaC - Web Corpora of Bosnian, Croatian and Serbian

Cited by 79 publications

References 6 publications

Quantitative fine-grained human evaluation of machine translation systems: a case study on English to Croatian

Quantitative fine-grained human evaluation of machine translation systems: a case study on English to Croatian

Psycholinguistic estimates of 3000 words of Croatian: concreteness and imageability

The Role of Homograms in Machine Translation

Contact Info

Product

Resources

About