Overview of the CLEF 2004 Multilingual Question Answering Track

Abstract

Abstract. The general aim of the third CLEF Multilingual Question Answering Track was to set up a common and replicable evaluation framework to test both monolingual and cross-language Question Answering (QA) systems that process queries and documents in several European languages. Nine target languages and ten source languages were exploited to enact 8 monolingual and 73 cross-language tasks. Twenty-four groups participated in the exercise. Overall results showed a general increase in performance in comparison to last year. The best performing monolingual system irrespective of target language answered 64.5% of the questions correctly (in the monolingual Portuguese task), while the average of the best performances for each target language was 42.6%. The cross-language step instead entailed a considerable drop in performance. In addition to accuracy, the organisers also measured the relation between the correctness of an answer and a system's stated confidence in it, showing that the best systems did not always provide the most reliable confidence score. We provide an overview of the 2005 QA track, detail the procedure followed to build the test sets and present a general analysis of the results. 308A. Vallin et al.

The Role of Lexical Features in Question Answering for Spanish

“…Question Answering (QA) systems have been proposed as a feasible option for the creation of such mechanisms. Moreover, the research in this field shows a constant growth both in interest as well as in complexity [3]. This paper presents the prototype developed in the Language Technologies Laboratory at INAOE 1 for the Spanish monolingual QA evaluation task at CLEF 2005.…”

Question Answering Pilot Task at CLEF 2004

“…Section 3 presents some criteria to design the evaluation measure and presents the K and K1 measures. The results for the Main QA Track at CLEF [3] are taken as a case of study to discuss and compare these measures with the previous ones used at TREC, NTCIR and CLEF. Section 4 presents the results obtained by the participants in the Pilot Task and, finally, Section 5 points out some conclusions and future work.…”

Overview of the CLEF 2007 Multilingual Question Answering Track

“…This process was as challenging as any translation job can be, since many cultural discrepancies and misunderstanding easily creep in. Anyway, as was already pointed out in 2004 "he fact that manual translation captured some of the cross-cultural as well as cross-language problems is good since QA systems are designed to work in the real world" [3].…”