An Empirical Evaluation of Noise Contrastive Estimation for the Neural Network Joint Model of Translation

Abstract

The neural network joint model of translation or NNJM (Devlin et al., 2014) combines source and target context to produce a powerful translation feature. However, its softmax layer necessitates a sum over the entire output vocabulary, which results in very slow maximum likelihood (MLE) training. This has led some groups to train using Noise Contrastive Estimation (NCE), which side-steps this sum. We carry out the first direct comparison of MLE and NCE training objectives for the NNJM, showing that NCE is significantly outperformed by MLE on large-scale ArabicEnglish and Chinese-English translation tasks. We also show that this drop can be avoided by using a recently proposed translation noise distribution.

Translation Prediction with Source Dependency-Based Context Representation

“…U i = M i e P A, SI, M i f , CH . Observing that the alignment A and source-dependency tree T f are fixed once the bilingual corpus is given, we specify the NB(U i ) as the set of DBiCUs: each DBiCU is with form of M e P A, SI, M i f , CH , satisfying |M e |=|M i e |; and it is generated by the IBM Model 1 distribution (Brown et al 1993), inspired by (Cherry 2016).…”