Topical Relevance Model

Abstract

Cross-lingual relevance modelling (CLRLM) is a state-of-the-art technique for cross-lingual information retrieval (CLIR) which integrates query term disambiguation and expansion in a unified framework, to directly estimate a model of relevant documents in the target language starting with a query in the source language. However, CLRLM involves integrating a translation model either on the document side if a parallel corpus is available, or on the query side if a bilingual dictionary is available. For low resourced language pairs, large parallel corpora do not exist and the vocabulary coverage of dictionaries is small, as a result of which RLM-based CLIR fails to obtain satisfactory results. Despite the lack of parallel resources for a majority of language pairs, the availability of comparable corpora for many languages has grown considerably in the recent years. Existing CLIR techniques such as cross-lingual relevance models, cannot effectively utilize these comparable corpora, since they do not use information from documents in the source language. We overcome this limitation by using information from retrieved documents in the source language to improve the retrieval quality of the target language documents. More precisely speaking, our model involves a two step approach of first retrieving documents both in the source language and the target language (using query translation), and then improving on the retrieval quality of target language documents by expanding the query with translations of words extracted from the top ranked documents retrieved in the source language which are thematically related (i.e. share the same concept) to the words in the top ranked target language documents. Our key hypothesis is that the query in the source language and its equivalent target language translation retrieve documents which share topics. The ovelapping topics of these top ranked documents in both languages are then used to improve the ranking of the target language documents. Since the model relies on the alignment of topics between language pairs, we call it the cross-lingual topical relevance model (CLTRLM). Experimental results show that the CLTRLM significantly outperforms the standard CLRLM by upto 37% on English-Bengali CLIR, achieving mean average precision (MAP) of up to 60.27% of the Bengali monolingual IR MAP.

Dimension Projection Among Languages Based on Pseudo-Relevant Documents for Query Translation

“…Stopword removal is also performed. 1 The Lemur toolkit 2 is employed as the retrieval engine in our experiments.…”

“…Pseudo-relevance feedback has long been employed as a powerful method for estimating query language models in a large number of studies [6,5,12]. Cross-lingual relevance model (CLRLM) and CLTRLM are state-of-the-art methods in cross-lingual environments [1,11,3]. Unlike CLRLM that depends on parallel corpora and bilingual lexicons, CLTRLM aims at finding a number of bilingual topical variables from a comparable corpus in order to transfer relevance score of a term from one language to another.…”

SS4MCT: A Statistical Stemmer for Morphologically Complex Texts

“…On the other hand, we run another set of experiments without applying any morphological processing method similar to the Persian state-of-the-art CLIR methods. Iterative translation disambiguation (ITD) [11], joint cross-lingual topical relevance model (JCLTRLM) [7], top-ranked translation (TOP-1), and the bi-gram coherence translation method (BiCTM), introduced in [5] (assume |c i | = 0), are the baselines without any morphological processing units. As shown in Table 3 BiCTM outperforms all the baselines when there is no morphological processing unit.…”

C-BiLDA extracting cross-lingual topics from non-parallel texts by distinguishing shared from unshared content

“…Cross-lingual text mining aims to induce and transfer knowledge across different languages to help applications such as cross-lingual information retrieval (Levow et al, 2005;Ganguly et al, 2012;Vulić et al, 2013), document classification (Prettenhofer and Stein, 2010;Ni et al, 2011;Guo and Xiao, 2012a), or cross-lingual annotation projection (Zhao et al, 2009;Das and Petrov, 2011;van der Plas et al, 2011;Kim et al, 2012;Täckström et al, 2013;Ganchev and Das, 2013) in cases where translation and class-labeled resources are scarce or missing. In this article, we utilize probabilistic topic models to perform cross-lingual text mining.…”