QA4MRE 2011-2013: Overview of Question Answering for Machine Reading Evaluation

Abstract

This paper describes the methodology for testing the performance of Machine Reading systems through Question Answering and Reading Comprehension Tests. This was the attempt of the QA4MRE challenge which was run as a Lab at CLEF 2011-2013. The traditional QA task was replaced by a new Machine Reading task, whose intention was to ask questions that required a deep knowledge of individual short texts and in which systems were required to choose one answer, by analysing the corresponding test document in conjunction with background text collections provided by the organization. Four different tasks have been organized during these years: Main Task, Processing Modality and Negation for Machine Reading, Machine Reading of Biomedical Texts about Alzheimer's disease, and Entrance Exams. This paper describes their motivation, their goals, their methodology for preparing the data sets, their background collections, their metrics used for the evaluation, and the lessons learned along these three years.

Evaluating Textual Entailment Recognition for University Entrance Examinations

“…Inherently, answers cannot be derived from external resources for this type of question. Although such questions may be reduced into textual entailment between a given text and choices Kasahara et al 2010;Peñas et al 2011], these cases were excluded from the present study.…”

“…Third, the performance of entailment recognition can be mapped to the test scores, which are comparable with human performance for the same task. Dagan et al [2006], Kasahara et al [2010], and Peñas et al [2011] investigated the concept of using reading comprehension tests for the evaluation of textual entailment recognition. They considered reading comprehension tests to be intended to test human ability to compute entailment relations, and they considered such tests to be applicable to the evaluation of automatic systems.…”

Which is the Effective Way for Gaokao: Information Retrieval or Neural Networks?

“…These clues may include Tang poem and Song iambic verse, domainspecific expressions, even some mixture of mod- ern Chinese and excerpt from ancient books and etc. The dependence of background knowledge makes the models that are designed for reading comprehension such as (Peñas et al, 2013;Richardson et al, 2013) fail. Thirdly, the diversity of candidates' granularity, i.e.…”

Формирование Вопросно-Ответной Системы В Условиях Ограниченного Объема Семантически Размеченного Корпуса

“…Оценка качества работы таких систем осуществляется на тестовых данных, разрабо-танных обществом экспертов и представленных в виде фрагмента текста, вопроса по этому тексту и предполагаемого ответа. В случае, если система по запросу выдает результат, не соответствующий пред-полагаемому ответу, результат не засчитывается.В 2012 году проводились соревнования по качеству работы вопросно-ответных систем в области биомедицины, наилучший результат (с точностью 55 %) показала система [2]. Отечественных систем, способных составить конкуренцию зарубежным системам по качеству работы, на данный момент не су-ществует.…”

“…В 2012 году проводились соревнования по качеству работы вопросно-ответных систем в области биомедицины, наилучший результат (с точностью 55 %) показала система [2]. Отечественных систем, способных составить конкуренцию зарубежным системам по качеству работы, на данный момент не су-ществует.…”