Overview of the CLEF 2007 Multilingual Question Answering Track

Abstract. Esfinge is a general domain Portuguese question answering system that participated in the last four editions of CLEF. This system uses the Web as a fundamental resource in its architecture, using information redundancy rather than sophisticated annotations of the document collections to retrieve answers. In this paper we describe experiments that took as starting point the version of Esfinge that participated at the evaluation contest CLEF 2007. These experiments consisted in using different types of search patterns to retrieve relevant documents for questions, as this issue (document retrieval) was responsible for most of the errors occurred at CLEF 2007. Keywords: Question answering, Portuguese, question reformulation Architecture of EsfingeIn this paper we will give a short description of the Portuguese question answering system Esfinge [1], as well as of a set of experiments performed with this system using different types of search patterns to retrieve relevant documents to answer questions.The architecture of Esfinge is composed by a pipeline of modules that handles each question in order to provide one answer.The questions are initially fed to an Anaphor Resolution module which caters for the resolution of anaphors. This module adds, to the original question, a list of alternative questions where the anaphors are (hopefully) resolved. Then, Esfinge iterates over the set of alternative questions created in the previous module:• The Question Reformulation module transforms the question into patterns of plausible answers. This is done using two different approaches: a) using a set of pre-defined pattern pairs that associate patterns of questions with patterns of plausible answers, producing a set of pairs (answer pattern, score) or b) using PALAVRAS [2] analysis to identify the main verb, its arguments and adjuncts and some entities from previous topic questions which are used to create search patterns.• The Search Document Collections module then uses these patterns to search in document collections. If no documents are retrieved, execution stops and NIL is returned meaning that the system is not able to answer the question.

Section: Evaluation and Discussion Of The Resultsmentioning

confidence: 99%

Answering Portuguese Questions

Costa

Cabral

2008

“…In this edition, questions were grouped by topic [4]. The first question of a topic was self contained in the sense that there is no need of information outside the question to answer it.…”

Section: Test Collectionsmentioning

confidence: 99%

Overview of the Answer Validation Exercise 2008

Rodrigo

Peñas

Verdejo

2009

Self Cite

The Answer Validation Exercise at the Cross Language Evaluation Forum is aimed at developing systems able to decide whether the answer of a Question Answering system is correct or not. We present here the exercise description, the changes in the evaluation methodology with respect to the first edition, and the results of this second edition (AVE 2007). The changes in the evaluation methodology had two objectives: the first one was to quantify the gain in performance when more sophisticated validation modules are introduced in QA systems. The second objective was to bring systems based on Textual Entailment to the Automatic Hypothesis Generation problem which is not part itself of the Recognising Textual Entailment (RTE) task but a need of the Answer Validation setting. 9 groups have participated with 16 runs in 4 different languages. Compared with the QA systems, the results show an evidence of the potential gain that more sophisticated AV modules introduce in the task of QA.

“…The run we submitted for the Romanian to English cross-lingual QA task achieved an overall accuracy of 14%, the best score achieved among systems with English as target language [6]. An in-depth analysis of the results at different stages in the QA process has revealed a number of future system improvement directions.…”

Section: Discussionmentioning

confidence: 99%

“…This year, the QA@CLEF main task distinguishes among four question types: factoid, definition, list and temporally restricted questions [6]. As temporal restrictions can constrain any question type, we first detect whether the question has the type factoid, definition or list, and then search for temporal restrictions.…”

Section: D) Inferring the Question Typementioning

confidence: 99%

“…Last year, a new Romanian-to-English (RO-EN) cross-lingual QA task was organised for the first time within the context of the CLEF campaign [10], and it consisted of retrieving answers to Romanian questions from a collection of English documents. This year's task [6] was similarly organised, with the exception that all questions were clustered in classes related to the same topic, some of which even contain anaphoric references to other questions from the same topic class, or to their answers. Besides the usual news collections employed in the search for answers, this year's novelty was the fact that Wikipedia articles could also be used as answer source.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

University of Wolverhampton at CLEF 2007

Puşcaşu

Orăsan

Abstract. This paper reports on the participation of the University of Wolverhampton in the Multiple Language Question Answering (QA@CLEF) track of the CLEF 2007 campaign. We approached the Romanian to English cross-lingual task with a Question Answering (QA) system that processes a question in the source language (i.e. Romanian), translates the identified keywords into the target language (i.e. English), and finally searches for answers in the English document collection. We submitted one run of our system that has achieved an overall accuracy of 14%, and a precision over non-NIL answers of 33.73%. Error analysis revealed that this low performance is mainly due to the lack of a reliable translation methodology from the source in the target language.