A discriminative approach to filter out noisy sentence pairs from bilingual corpora

State-of-the-art English to Persian Statistical Machine Translation system

“…The main differ of the work of [11] and ours in this component is related to feature set has been employed in the classifier. The feature set proposed in our work are NullEN, NullFA, Full_NullEN, Full_NullFA, p(F|E) and p(E|F) and other features was described in [11].…”

“…Filtering Component: The result of sentence aligning algorithm contains several erroneous sentence pairs which should be detected and eliminated from the parallel corpus. Similar to the works of [11] a MaxEnt classifier has been innovated to classify the sentence pair as correct or incorrect. The details of this component are illustrated in next section.…”

The Effect of Parallel Corpus Quality vs Size in English - Toturkish SMT

“…Their classifier is reported to have 0.88 precision and recall rates. Taghipour et al [24] also proposed a classification method for cleaning parallel data. Many features have been tested and used to build the models such as translation probabilities based on IBM translations models [3], the number of the null aligned words, length based features and features based on language model.…”

“…Building a corpus that includes vast amount of parallel sentences is one of the most time-consuming and important works for a high-performance SMT system [10]. Training the translation model component in SMT requires large parallel corpora for the parameters to be estimated [24]. Therefore, higher translation accuracy can be achieved when machine translation systems are trained on increasing amounts of training data [12].…”

“…The manual compilation of a parallel corpus is too expensive, so most of available parallel corpora are generated automatically. Automatic methods for compiling parallel sentence pairs are imprecise and using such a low-quality training corpus that has many non-parallel sentence pairs would cause low quality translations [24]. The noise in an automatically generated corpus might be due to any difference between the contents of source and target documents, non-literal translations or sentence alignment mistakes.…”