Two levels of evaluation in a complex NL system

Abstract

The QALC question-answering system, developed at LIMSI, has been a participant for two years in the QA track of the TREC conference. In this paper, we present a quantitative evaluation of various modules in our system, based on two criteria: first, the numbers of documents containing the correct answer and selected by the system; secondly, the number of answers found. The first criterion is used for evaluating locally the modules in the system, which contribute in selecting documents that are likely to contain the answer. The second one provides a global evaluation of the system. As such, it also serves for an indirect evaluation of various modules.