Open- vs. Restricted-Domain QA Systems in the Biomedical Field

Abstract

Question answering systems (QA systems) stand as a new alternative for information retrieval systems. We conducted a study to evaluate the efficiency of QA systems as terminological sources for physicians, specialized translators and users in general. To this end we analysed the performance of two open-domain and two restricted-domain QA systems. The research entailed a collection of 150 definitional questions from WebMed. We studied the sources that QA systems used to retrieve the answers, and later applied a range of evaluation measures to mark the quality of answers. Through analysing the results obtained by asking the 150 questions in the QA systems MedQA, START, QuALiM and HONqa, it was possible to evaluate the systems' operation through applying specific metrics (MRR, FHS, TRR, Precision, Recall). Despite the limitations demonstrated by these systems, it has been confirmed that these four QA systems are valid and useful for obtaining definitional medical information in that they offer coherent and precise answers.

Evaluación del rendimiento de los sistemas de búsqueda de respuestas de dominio general

“…Por último, y en relación al tipo de respuestas que aquí se ha considerado como inexactas, la presen-cia de las mismas en general no ha sido demasiado elevada, si bien en algún sistema ha superado el 20%. Estos datos mejoran sutilmente los resultados obtenidos en evaluaciones previas de los SBR (Olvera-Lobo y Gutiérrez-Artacho, 2010, 2011a, en donde QuALiM y START fueron evaluados y comparados con SBR de dominio especializado. Aunque las preguntas en las evaluaciones previas se restringían exclusivamente a las de definición y en un ámbito de especialización, se comprueba que los sistemas siguen siendo eficaces en la recuperación de información.…”

Contextual question answering for the health domain

“…In other words, if the first answer for any question is incorrect, then enquireMe has failed to respond correctly to that question. This is in contrast to the way the survey authors (Olvera‐Lobo & Gutierrez‐Artacho, ) evaluated HONqa, QuALiM, and START, where the top five answers produced were considered. Not all questions, however, have five or more answers.…”

“…We prepared three sets of data for testing the four systems. The first set (S1) is benchmark data in the health domain obtained from the website WebMD, similar to the one used by Olvera‐Lobo and Gutierrez‐Artacho (), which comprises 150 definitional questions in the form of “What is X?” The second set (S2) extends the “one‐off” questions in the first set to include follow‐up questions to simulate an interactive environment for contextual question answering. This second set contains a total of 274 questions about the first 99 medical concepts (those that start with A through N) from the complete set of 150 concepts.…”

“…In the table, rows 1–3 show the results for the three systems HONqa, QuALiM, and START using questions from set S1. These results were extracted from the survey paper by Olvera‐Lobo and Gutierrez‐Artacho (). We were unable to repeat the experiment because QuALiM is no longer available at its original web address (http://demos.inf.ed.ac.uk:8080/qualim).…”

“…Row 5 shows the results from evaluating enquireMe using the 150 “What is X?” questions from set S1. The answers produced by enquireMe were assessed in the same way as the survey authors (Olvera‐Lobo & Gutierrez‐Artacho, ) would for rows 1–3 with two caveats. First, unlike the survey authors who recruited health professionals, we assessed the answers generated by enquireMe based strictly on the definitions provided on WebMD.…”