Overview of ResPubliQA 2009: Question Answering Evaluation over European Legislation

Peñas, Anselmo; Forner, Pamela; Sutcliffe, Richard F. E.; Rodrigo, Álvaro; Forascu, Corina; Alegria, Iñaki; Giampiccolo, Danilo; Moreau, Nicolas; Osenova, Petya

doi:10.1007/978-3-642-15754-7_21

Cited by 44 publications

(20 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The results of LogAnswer in ResPubliQA 2009 and the two official baseline results [2] are shown in Table 1. The loga091dede run was obtained from the standard configuration of LogAnswer with full logic-based processing, while loga092dede was generated with the prover switched off.…”

Section: Results On the Respubliqa 2009 Test Set For Germanmentioning

confidence: 99%

See 1 more Smart Citation

Extending a Logic-Based Question Answering System for Administrative Texts

Glöckner

Pelzer

2010

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. LogAnswer is a question answering (QA) system for German that uses machine learning for integrating logic-based and shallow (lexical) validation features. For ResPubliQA 2009, LogAnswer was adjusted to specifics of administrative texts, as found in the JRC Acquis corpus. Moreover, support for a broader class of questions relevant to the domain was added, including questions that ask for a purpose, reason, or procedure. Results confirm the success of these measures to prepare LogAnswer for ResPubliQA, and of the general consolidation of the system. According to the C@1/Best IR baseline metric that tries to abstract from the language factor, LogAnswer was the third best of eleven systems participating in ResPubliQA. The system was especially successful at detecting wrong answers, with 73% correct rejections.

show abstract

Section: Results On the Respubliqa 2009 Test Set For Germanmentioning

confidence: 99%

“…3 It was first evaluated in QA@CLEF 2008 [1]. The ResPubliQA task [2] posed some new challenges for LogAnswer:…”

Section: Introductionmentioning

confidence: 99%

Extending a Logic-Based Question Answering System for Administrative Texts

Glöckner

Pelzer

2010

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Many attempts have been made in the field of Web-based question answering [Dumais et al 2002;Kwok et al 2001] and cross-language question answering [Isozaki et al 2005;Peñas et al 2009] to overcome this language barrier in information retrieval. However, cross-language information retrieval and question answering systems rely heavily on machine translation, which might produce poor results because of the noise in Web text and the lack of resources such as parallel corpora or translation dictionaries for some language pairs [Ferrandez et al 2009].…”

Section: Introductionmentioning

confidence: 99%

Cross-Language Latent Relational Search between Japanese and English Languages Using a Web Corpus

Duc

Bollegala

Ishizuka

2012

ACM Transactions on Asian Language Information Processing

View full text Add to dashboard Cite

Latent relational search is a novel entity retrieval paradigm based on the proportional analogy between two entity pairs. Given a latent relational search query {(Japan, Tokyo), (France, ?)}, a latent relational search engine is expected to retrieve and rank the entity "Paris" as the first answer in the result list. A latent relational search engine extracts entities and relations between those entities from a corpus, such as the Web. Moreover, from some supporting sentences in the corpus, (e.g., "Tokyo is the capital of Japan" and "Paris is the capital and biggest city of France"), the search engine must recognize the relational similarity between the two entity pairs. In cross-language latent relational search, the entity pairs as well as the supporting sentences of the first entity pair and of the second entity pair are in different languages. Therefore, the search engine must recognize similar semantic relations across languages. In this article, we study the problem of cross-language latent relational search between Japanese and English using Web data. To perform cross-language latent relational search in high speed, we propose a multi-lingual indexing method for storing entities and lexical patterns that represent the semantic relations extracted from Web corpora. We then propose a hybrid lexical pattern clustering algorithm to capture the semantic similarity between lexical patterns across languages. Using this algorithm, we can precisely measure the relational similarity between entity pairs across languages, thereby achieving high precision in the task of cross-language latent relational search. Experiments show that the proposed method achieves an MRR of 0.605 on JapaneseEnglish cross-language latent relational search query sets and it also achieves a reasonable performance on the INEX Entity Ranking task.

show abstract

“…The development of the DLT system spawned the investigation in White and Sutcliffe (2004), where we considered the possibility of locating supporting sentences by identifying terms that are related to those in the query. We enumerated occurrences of various morphological relationship types, including direct matches, different inflections, different Parts‐of‐Speech (POS), and various semantic‐relationship types such as synonyms, hypernyms, word chains, and holonyms, that exist between terms in 50 TREC factoid queries from 2003 and their supporting sentences.…”

Section: Introductionmentioning

confidence: 99%

“… a a This table lists different morphological and semantic relationship types that exist between query terms and those in supporting sentences. An example of each type of relationship is provided (White & Sutcliffe, 2004). …”

Section: Introductionmentioning

confidence: 99%

Butcher, baker, or candlestick maker? Predicting occupations using predicate-argument relations

White

Sutcliffe

2011

J. Am. Soc. Inf. Sci.

View full text Add to dashboard Cite

In a previous question answering study, we identified nine semantic-relationship types, including synonyms, hypernyms, word chains, and holonyms, that exist between terms in Text Retrieval Conference queries and those in their supporting sentences in the Advanced Question Answering for Intelligence (Graff, 2002) corpus. The most frequently occurring relationship type was the hypernym (e.g., Katherine Hepburn is an actress).The aim of the present work, therefore, was to develop a method for determining a person's occupation from syntactic data in a text corpus. First, in the P -System, we compared predicate-argument data involving a proper name for different occupations using Okapi's BM25 weighting algorithm. When classifying actors and using sufficiently frequent names, an accuracy of 0.955 was attained. For evaluation purposes, we also implemented a standard apposition-based classifier (A-System). This performs well, but only if a particular name happens to appear in apposition with the corresponding occupation. Last, we created a hybrid (H -System) which combines the strengths of P with those of A. Using data with a minimum of 100 predicate-argument pairs, H performed best with an overall lenient accuracy of 0.750 while A and P scored 0.615 and 0.656, respectively. We therefore conclude that a hybrid approach combining information from different sources is the best way to predict occupations. IntroductionA question answering (QA) system takes as input a short query and returns an exact answer extracted from a document collection. As part of our participation in the annual Text Retrieval Conference (TREC) and Cross Language Evaluation Forum (CLEF) QA evaluations, we developed the Documents and Linguistic Technology (DLT) system Sutcliffe, White, Slattery, Gabbay, & Mulcahy, 2006). When presented with a query, it applied the method established by many other firstgeneration QA models of identifying the appropriate Named Entity (NE) type needed to answer the question, and then with a scoring function selecting an NE of this type from a set of topical documents as determined by an information retrieval (IR) system. For example, the query "How long is a quarter in an NBA game?" would be answered with an instance of the length_of_time NE type. A Boolean IR system first returned a collection of documents deemed relevant to a modified form of the original query. From these, all recognizable length_of_time NEs were scored by a function, and the NE with the highest score was returned as the answer.The development of the DLT system spawned the investigation in White and Sutcliffe (2004), where we considered the possibility of locating supporting sentences by identifying terms that are related to those in the query. We enumerated occurrences of various morphological relationship types, including direct matches, different inflections, different Partsof-Speech (POS), and various semantic-relationship types such as synonyms, hypernyms, word chains, and holonyms, that exist between terms in 50 TREC factoid queries from 2003 and...

show abstract

Overview of ResPubliQA 2009: Question Answering Evaluation over European Legislation

Cited by 44 publications

References 8 publications

Extending a Logic-Based Question Answering System for Administrative Texts

Extending a Logic-Based Question Answering System for Administrative Texts

Cross-Language Latent Relational Search between Japanese and English Languages Using a Web Corpus

Butcher, baker, or candlestick maker? Predicting occupations using predicate-argument relations

Contact Info

Product

Resources

About