Sibyl, a factoid question-answering system for spoken documents

Abstract

In this article, we present a factoid question-answering system, Sibyl, specifically tailored for question answering (QA) on spoken-word documents. This work explores, for the first time, which techniques can be robustly adapted from the usual QA on written documents to the more difficult spoken document scenario. More specifically, we study new information retrieval (IR) techniques designed or speech, and utilize several levels of linguistic information for the speech-based QA task. These include named-entity detection with phonetic information, syntactic parsing applied to speech transcripts, and the use of coreference resolution. Sibyl is largely based on supervised machine-learning techniques, with special focus on the answer extraction step, and makes little use of handcrafted knowledge. Consequently, it should be easily adaptable to other domains and languages. Sibyl and all its modules are extensively evaluated on the European Parliament Plenary Sessions English corpus, comparing manual with automatic transcripts obtained by three different automatic speech recognition (ASR) systems that exhibit significantly different word error rates. This data belongs to the CLEF 2009 track for QA on speech transcripts. The main results confirm that syntactic information is very useful for learning to rank question candidates, improving results on both manual and automatic transcripts, unless the ASR quality is very low. At the same time, our experiments on coreference resolution reveal that the state-of-the-art technology is not mature enough to be effectively exploited for QA with spoken documents. Overall, the performance of Sibyl is comparable or better than the state-of-the-art on this corpus, confirming the validity of our approach.

Entities' Sentiment Relevance

“…Other related works belong to the Passage Retrieval field, since the relevance detection problem can be construed as a specific form of passage retrieval problem (Liu and Croft, 2002;Tiedemann and Mur, 2008). Different approaches were suggested for passage retrieval (Buscaldi et al, 2010;Comas et al, 2012;Hearst, 1997;Lafferty et al, 2001;Lin et al, 2012;Liu and Croft, 2002;Lloret et al, 2012;O'Connor et al, 2013;Otterbacher et al, 2009;Salton et al, 1993;Wachsmuth, 2013), some are more sophisticated than others.…”

Automatic Summarization

“…However, there is also interest in developing QA for spoken data. For example QA for lectures and meetings has been reported in [55,233], while [310] describes research on video news QA. QA for spoken data can utilize many of the methods developed for text QA.…”

Towards Machine Comprehension of Spoken Content: Initial TOEFL Listening Comprehension Test by Machine

“…SQA usually worked with ASR transcripts of the spoken content, and used information retrieval (IR) techniques [3] or relied on knowledge bases [4] to find the proper answer. Sibyl [5], a The story is given in audio format, and its manual transcription is shown. The question and choices are provided in text format.…”