Textual resource acquisition and engineering

Chu-Carroll, Jennifer; Fan, Jialu; Schlaefer, Nico; Zadrozny, Wlodek

doi:10.1147/jrd.2012.2185901

Cited by 24 publications

(16 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Hypothesis Generation phase takes as input results from question analysis, summarized in the previous section. The first four primary search components in the diagram show Watson's Document and Passage search strategies, which target unstructured knowledge resources such as encyclopedia documents and newswire articles [10]. On the other hand, the last two search components, namely, Answer Lookup and PRISMATIC search, use different types of structured resources.…”

Section: Search and Candidate Generation Overviewmentioning

confidence: 99%

Finding needles in the haystack: Search and candidate generation

et al. 2012

Self Cite

View full text Add to dashboard Cite

A key phase in the DeepQA architecture is Hypothesis Generation, in which candidate system responses are generated for downstream scoring and ranking. In the IBM Watsoni system, these hypotheses are potential answers to Jeopardy!i questions and are generated by two components: search and candidate generation. The search component retrieves content relevant to a given question from Watson's knowledge resources. The candidate generation component identifies potential answers to the question from the retrieved content. In this paper, we present strategies developed to use characteristics of Watson's different knowledge sources and to formulate effective search queries against those sources. We further discuss a suite of candidate generation strategies that use various kinds of metadata, such as document titles or anchor texts in hyperlinked documents. We demonstrate that a combination of these strategies brings the correct answer into the candidate answer pool for 87.17% of all the questions in a blind test set, facilitating high end-to-end question-answering performance.

show abstract

Section: Search and Candidate Generation Overviewmentioning

confidence: 99%

Finding needles in the haystack: Search and candidate generation

et al. 2012

Self Cite

View full text Add to dashboard Cite

show abstract

“…and TREC in an iterative error analysis performed by the Watson development team. The collection (subsequently referred to as All Sources) comprises 25.6 GB of text, including Wikipedia and the other encyclopedias in Section 6.2, dictionaries such as Wiktionary, thesauri, newswire sources such as a New York Times archive, literature and other sources of trivia knowledge [Chu-Carroll et al, 2012b]. It also includes the AQUAINT newswire corpus, which was the reference source in TREC 11-15 and contains the answers to all questions in these datasets (except NIL questions, which were not used in our experiments).…”

Section: Experimental Setup Using Watsonmentioning

confidence: 99%

Statistical source expansion for question answering

Schlaefer

Chu-Carroll

Nyberg

et al. 2011

Proceedings of the 20th ACM International Conference on Information and Knowledge Management

View full text Add to dashboard Cite

A source expansion algorithm automatically extends a given text corpus with related information from large, unstructured sources. While the expanded corpus is not intended for human consumption, it can be leveraged in question answering (QA) and other information retrieval or extraction tasks to find more relevant knowledge and to gather additional evidence for evaluating hypotheses. In this thesis, we propose a novel algorithm that expands a collection of seed documents by (1) retrieving related content from the Web or other large external sources, (2) extracting self-contained text nuggets from the related content, (3) estimating the relevance of the text nuggets with regard to the topics of the seed documents using a statistical model, and (4) compiling new pseudo-documents from nuggets that are relevant and complement existing information.In an intrinsic evaluation on a dataset comprising 1,500 hand-labeled web pages, the most effective statistical relevance model ranked text nuggets by relevance with 81% MAP, compared to 43% when relying on rankings generated by a web search engine, and 75% when using a multi-document summarization algorithm. These differences are statistically significant and result in noticeable gains in search performance in a task-based evaluation on QA datasets. The statistical models use a comprehensive set of features to predict the topicality and quality of text nuggets based on topic models built from seed content, search engine rankings and surface characteristics of the retrieved text. Linear models that evaluate text nuggets individually are compared to a sequential model that estimates their relevance given the surrounding nuggets. The sequential model leverages features derived from text segmentation algorithms to dynamically predict transitions between relevant and irrelevant passages. It slightly outperforms the best linear model while using fewer parameters and requiring less training time. In addition, we demonstrate that active learning reduces the amount of labeled data required to fit a relevance model by two orders of magnitude with little loss in ranking performance. This facilitates the adaptation of the source expansion algorithm to new knowledge domains and applications. Applied to the QA task, the proposed method yields consistent and statistically significant performance gains across different datasets, seed corpora and retrieval strategies. We evaluated the impact of source expansion on search performance and end-to-end accuracy using Watson and the OpenEphyra QA system, and datasets comprising over 6,500 questions from the Jeopardy! quiz show and TREC evaluations. By expanding various seed corpora with web search results, we were able to improve the QA accuracy of Watson from 66% to 71% on regular Jeopardy! questions, from 45% to 51% on Final Jeopardy! questions and from 59% to 64% on TREC factoid questions. We also show that the source expansion approach can be adapted to extract relevant content from locally stored sources without requiring a search e...

show abstract

“…challenge used a broad variety of content sources, primary among which were Wikipedia and Wiktionary; the motivation for source selection is presented in Chu-Carroll et al (2012c). challenge used a broad variety of content sources, primary among which were Wikipedia and Wiktionary; the motivation for source selection is presented in Chu-Carroll et al (2012c).…”

Section: Hypothesis Generationmentioning

confidence: 99%

Parallel and nested decomposition for factoid questions

Boguraev¹,

Patwardhan²,

Kalyanpur³

et al. 2013

Nat. Lang. Eng.

View full text Add to dashboard Cite

Typically, automatic Question Answering (QA) approaches use the question in its entirety in the search for potential answers. We argue that decomposing complex factoid questions into separate facts about their answers is beneficial to QA, since an answer candidate with support coming from multiple independent facts is more likely to be the correct one. We broadly categorize decomposable questions as parallel or nested, and we present a novel question decomposition framework for enhancing the ability of single-shot QA systems to answer complex factoid questions. Essential to the framework are components for decomposition recognition, question rewriting, and candidate answer synthesis and re-ranking. We discuss the interplay among these, with particular emphasis on decomposition recognition, a process which, we argue, can be sufficiently informed by lexico-syntactic features alone. We validate our approach to decomposition by implementing the framework on top of IBM Watson TM , a state-of-the-art QA system, and showing a statistically significant improvement over its accuracy.Questions like these are found in domains such as medical, legal, financial, etc. Independent of domain and type, however, they share a common characteristic: If a search query is constructed from all the facts collectively describing the answer, very few (if any) relevant documents are likely to be found, with undesired consequences for the identification of potential answer-bearing passages. The notion of decomposition thus goes hand in hand with that of recursively applying a QA system to the individual facts (sub-questions), followed by suitable re-composition of the candidate answer lists for the sub-questions.Some of our earlier decomposition works were motivated by such considerations: Kalyanpur et al. (2011) offer a brief overview of the decomposition framework, within which we then discuss particular heuristics for recognizing decomposable questions .This paper presents, in depth, our evolved decomposition approach. We describe the framework, and how it relates to a class of generic QA architectures. We follow

show abstract

Textual resource acquisition and engineering

Cited by 24 publications

References 17 publications

Finding needles in the haystack: Search and candidate generation

Finding needles in the haystack: Search and candidate generation

Statistical source expansion for question answering

Parallel and nested decomposition for factoid questions

Contact Info

Product

Resources

About