Comparing example-based and statistical machine translation

Abstract

Link to this article: http://journals.cambridge.org/abstract_S1351324905003888How to cite this article: ANDY WAY and NANO GOUGH (2005). Comparing example-based and statistical machine translation. Natural Language Engineering, 11, pp 295-309 AbstractIn previous work (Gough and Way 2004), we showed that our Example-Based Machine Translation (EBMT) system improved with respect to both coverage and quality when seeded with increasing amounts of training data, so that it significantly outperformed the on-line MT system Logomedia according to a wide variety of automatic evaluation metrics. While it is perhaps unsurprising that system performance is correlated with the amount of training data, we address in this paper the question of whether a large-scale, robust EBMT system such as ours can outperform a Statistical Machine Translation (SMT) system. We obtained a large English-French translation memory from Sun Microsystems from which we randomly extracted a near 4K test set. The remaining data was split into three training sets, of roughly 50K, 100K and 200K sentence-pairs in order to measure the effect of increasing the size of the training data on the performance of the two systems. Our main observation is that contrary to perceived wisdom in the field, there appears to be little substance to the claim that SMT systems are guaranteed to outperform EBMT systems when confronted with 'enough' training data. Our tests on a 4.8 million word bitext indicate that while SMT appears to outperform our system for French-English on a number of metrics, for English-French, on all but one automatic evaluation metric, the performance of our EBMT system is superior to the baseline SMT model.

Hybrid data-driven models of machine translation

“…It 1s surprismg, therefore, that apart from the work of Way and Gough (2005a)' no substantial research has been performed into comparing these empirical approaches. In this thesis, we wish to compare these approaches both from a theoretical and practical point of mew, for the benefit of both SMT and EBMT research communities and t o determine whether in fact they constitute two different, paradigms, or are, in fact, one and the same.…”

“…When aligning chunks for use in the marker-based EBMT system of Gough and Way (2004b), Gough (2005) and Way and Gough (2005a) employed in our work, the MI scores for <source,target> word pairs within the chunks under cons~deration are examined. Links are then created between source and target words, with the restriction that the MI score for a particular word pair under consideration is higher than the MI score for all alternative word pairs containing the same target word.…”

“…As Way and Gough (2005a) state, one advantage EBMT systems have over their SMT counterparts is their re-use of examples. When translating an input sentence, if an EBMT system has previously encountered the sentence it can simply perform an exact sentence match, retrieving the source language sentence and outputting the corresponding target language sentence.…”

“…In this Chapter we address this question. We continue the work described by Way and Gough (2005a) by comparing the performance of the two main corpus-based approaches of example-based MT (EBMT) and statistical phrasebased MT (PBSMT) when trained and tested on reasonably large amounts of data taken from the Sun Microsystems TM (described in Section 3.1). After describing a range of automatic metrics used in the evaluation of MT in Section 3.2, these comparative experiments, along with their results are presented in Section 3 3 All approaches to MT have their advantages and disadvantages,therefore it is unsurprising that many practitioners have investigated combining various MT approaches into hybrid approaches.…”

“…This data set was used in the previous experiments of Gough and Way (2004b), Gough (2005) and Way and Gough (2005a) As the corpus comes from a particular sublanguage domain, is it particularly suited to MT. In MT, a sublanguage usually refers to a particular type of text wrltten with a particular communicative purpose, written using particular language constructions and specialised vocabulary (e.g.…”