How to Compare Bilingual to Monolingual Cross-Language Information Retrieval

Abstract

The study of cross-lingual \acp{IRS} and a deep analysis of system performances should provide guidelines, hints, and directions to drive the design and development of the next generation \ac{MLIA} systems. In addition, effective tools for interpreting and comparing the experimental results should be made easily available to the research community. To this end, we propose a twofold methodology for the evaluation of \ac{CLIR} systems: statistical analyses to provide \ac{MLIA} researchers with quantitative and more sophisticated analysis techniques; graphical tools to allow for a more qualitative comparison and an easier presentation of the results. We provide concrete examples about how the proposed methodology can be applied by studying the monolingual and bilingual tasks of the \ac{CLEF} 2005 and 2006 campaigns