WoZ simulation of interactive question answering

Abstract

QACIAD (Question Answering Challenge for Information Access Dialogue) is an evaluation framework for measuring interactive question answering (QA) technologies. It assumes that users interactively collect information using a QA system for writing a report on a given topic and evaluates, among other things, the capabilities needed under such circumstances. This paper reports an experiment for examining the assumptions made by QACIAD. In this experiment, dialogues under the situation that QACIAD assumes are collected using WoZ (Wizard of Oz) simulating, which is frequently used for collecting dialogue data for designing speech dialogue systems, and then analyzed. The results indicate that the setting of QACIAD is real and appropriate and that one of the important capabilities for future interactive QA systems is providing cooperative and helpful responses.

Follow-up question handling in the IMIX and Ritel systems: A comparative study

“…The experimental setup appears quite succesful in constraining the user questions to factoid questions. In contrast, (Kato et al, 2006), who expressly instructed users to pose "factoid" questions, found that their users posed some 34% non-factoid questions, mostly why, how, and definition questions.…”

“…We found that 25% of the FQs in the unimodal FQ corpus, and 18% of the FQs in the multimodal FQ corpus were in-between, so it makes sense to account for these. (Kato et al, 2006) also reports 26% self-contained FQs in their dialogue corpus. In the new Ritel corpus, we even found that 63% of obviously on-topic FQ were self-contained (for some reason, users tended to remain unusually self-contained in this setting).…”

“…These are French-language. We will also include some results from FQ dialogue corpora from the literature, in particular (Bertomeu et al, 2006) (English) and (Kato et al, 2006) (Japanese).…”

Designing an interactive open-domain question answering system

“…Indeed, Interactive QA systems are often reported at an early stage, such as Wizard-of-Oz studies, or applied to closed domains (Bertomeu et al 2006;Jönsson and Merkel 2003;Kato et al 2006).…”

Personalized, interactive question answering on the Web

“…Some rights reserved. been proposed, which involves the integration of QA systems with dialogue interfaces in order to encourage and accommodate the submission of multiple related questions and handle the user's requests for clarification in a less artificial setting (Maybury, 2002); however, Interactive QA (IQA) systems are still at an early stage or applied to closed domains (Small et al, 2003;Kato et al, 2006). Also, the "complex, interactive QA" TREC track (www.umiacs.umd.edu/ jimmylin/ciqa/) has been organized, but here the interactive aspect refers to the evaluators being enabled to interact with the systems rather than to dialogue per se.…”