On Cross-Lingual Text Similarity Using Neural Translation Models

Abstract

Accurately computing the similarity between two texts written in different languages has tremendous value in many applications, such as cross-lingual information retrieval and cross-lingual text mining/analytics. This paper studies the important problem based on neural networks. Specifically, our focus is on the neural machine translation models. While translation models are utilized, we pay special attention not to the translation itself but to the intermediate states of given texts stored in the translation models. Our assumption is that the intermediate states capture the syntactic and semantic meaning of input texts and are a good representation of the texts, avoiding inevitable translation errors. To study the validity of the assumption, we investigate the utility of the intermediates states and their effectiveness in computing cross-lingual text similarity in comparison with other neural network-based distributed representations of texts, including word and paragraph embedding-based approaches. We demonstrate that an approach using the intermediate states outperforms not only these approaches but also a strong machine translation-based one. Furthermore, it is revealed that intermediate states and translated texts work complementarily each other despite the fact that they are generated from the same NMT models.

Cross-lingual text similarity exploiting neural machine translation models

“…More recently, Seki [3,11] explored the use of NMT models [15–17]. He used the encoder output as the document representation of an input text for each language and computed cross-lingual text similarity.…”

“…Seki [3] argued that the output of the encoder can be seen as a concise, good representation of the input and that, since the intermediate vector is not yet translated to a particular word sequence, it could potentially alleviate the issues caused by polysemy and synonymy inevitable for current MT systems. Based on the intuition, he demonstrated that they could be effectively used for measuring cross-lingual text similarity.…”

“…To learn NMT models, we follow the same procedure as the related work [3]. Specifically, given translations pairs written in languages L 1 and L 2 , we learn two independent NMT models [28], M 12 and M 21 , with gated recurrent unit (GRU) layers and an attention mechanism for both directions of translations, where M 12 denotes the model to translate L 1 to L 2 and M 21 to translate L 2 to L 1 .…”

“…However, the MT-based approach has its own issues. Seki [3] pointed out that a word in one language may be polysemous (e.g. ‘crane’ and ‘bank’) and may not be correctly translated to the right word in the other language depending on the performance of a particular MT system utilised.…”

“…This study is partly in line with this direction with an emphasis on neural machine translation (NMT) models for semantic document representation. Specifically, we build upon Seki’s [3] approach exploiting the intermediate states of NMT models and extend the work by introducing more effective heuristics and techniques, including orthogonal matrix transform and a learning-to-rank approach to combine intermediate states with MTs.…”