Overview of the CLEF 2005 Interactive Track

Gonzalo, Julio; Clough, Paul; Vallin, Alessandro

doi:10.1007/11878773_30

Cited by 17 publications

(6 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As the quality of machine translation improved, the focus of CLIR user studies expanded from merely enabling users to find documents (e.g., for subsequent human translation) to also support information use (e.g., by translating the full text). The question answering task in the interactive track of the Cross-Language Evaluation Forum (iCLEF) is an example of that more comprehensive perspective [8]. The studies reported in this paper continue to broaden the perspective by adding a focus on complex tasks with live multimedia content.…”

Section: User-centered Evaluation Of Clirmentioning

confidence: 89%

Task-based interaction with an integrated multilingual, multimedia information system

Zhang

Plettenberg

Klavans

et al. 2007

Proceedings of the 7th ACM/IEEE-CS Joint Conference on Digital Libraries

View full text Add to dashboard Cite

This paper describes a formative evaluation of an integrated multilingual, multimedia information system, a series of user studies designed to guide system development. The system includes automatic speech recognition for English, Chinese, and Arabic, automatic translation from Chinese and Arabic into English, and query-based and profile-based search options. The study design emphasizes repeated evaluation with the same (increasingly experienced) participants, exploration of alternative task designs, rich qualitative and quantitative data collection, and rapid analysis to provide the timely feedback needed to support iterative and responsive development. Results indicate that users presented with materials in a language that they do not know can generate remarkably useful work products, but that integration of transcription, translation, search and profile management poses challenges that would be less evident were each technology to be evaluated in isolation.

show abstract

Section: User-centered Evaluation Of Clirmentioning

confidence: 89%

Task-based interaction with an integrated multilingual, multimedia information system

Zhang

Plettenberg

Klavans

et al. 2007

Proceedings of the 7th ACM/IEEE-CS Joint Conference on Digital Libraries

View full text Add to dashboard Cite

show abstract

“…For example, interactive CLEF (Gonzalo and Oard, 2004;Gonzalo et al, 2006) used a minimum of eight subjects while the Interactive Track at TREC-9 and TREC-6 used 16 and 20 searchers respectively (Hersh and Over, 1999;Over, 1997). The number of subjects directly influences the amount of resources required in terms of cost, time and effort.…”

Section: Recruitment Of Subjectsmentioning

confidence: 99%

“…Researchers used one or more of these types of tasks depending on the research goal/questions. Search tasks/topics were used at TREC and CLEF interactive tracks (Over, 1997;Gonzalo et al, 2006); while Jose et al (1998);Borlund (2000); White et al (2007) and Petrelli (2008) adopted Borlund's simulated work task. In the same context, another approach to achieve realism and motivate and engage the recruited subjects in the evaluation is to let them choose the search tasks from a pool of available tasks, for instance, in different domains (Spink, 2002;Su, 2003;White et al, 2007;Joho et al, 2008).…”

Section: Tasks and Topicsmentioning

confidence: 99%

An overview of semantic search evaluation initiatives

Elbedweihy

Wrigley

Clough

et al. 2015

Journal of Web Semantics

Self Cite

View full text Add to dashboard Cite

“…In respect of classical measures, we have used the "accuracy" measure (Gonzalo et al, 2006), the fraction of questions in which the user obtained the information searched within a time limit of three minutes (when an user did not find the correct question within the three minutes allowed per question, the question was considered as not resolved similar to the criteria used in iCLEF 2004 and 2005). Here it has only been taken into account if the query results are correct (anaphor and ellipsis resolution has not been evaluated with this measure).…”

Section: Series Of Context Questionsmentioning

confidence: 99%

An authoring tool for decision support systems in context questions of ecological knowledge

Ferrández¹,

Peral²,

Gregorio³

et al. 2015

Ecological Informatics

View full text Add to dashboard Cite

Decision support systems (DSS) support business or organizational decision-making activities, which require the access to information that is internally stored in databases or data warehouses, and externally in the Web accessed by Information Retrieval (IR) or Question Answering (QA) systems. Graphical interfaces to query these sources of information ease to constrain dynamically query formulation based on user selections, but they present a lack of flexibility in query formulation, since the expressivity power is reduced to the user interface design. Natural language interfaces (NLI) are expected as the optimal solution. However, especially for non-expert users, a real natural communication is the most difficult to realize effectively. In this paper, we propose a NLI that improves the interaction between the user and the DSS by means of referencing previous questions or their answers (i.e. anaphora such as the pronoun reference in -What traits are affected by them?‖), or by eliding parts of the question (i.e. ellipsis such as -And to glume colour?‖ after the question -Tell me the QTLs related to awn colour in wheat‖). Moreover, in order to overcome one of the main problems of NLIs about the difficulty to adapt a NLI to a new domain, our proposal is based on ontologies that are obtained semi-automatically from a framework that allows the integration of internal and external, structured and unstructured information. Therefore, our proposal can interface with databases, data warehouses, QA and IR systems. Because of the high NL ambiguity of the resolution process, our proposal is presented as an authoring tool that helps the user to query efficiently in natural language. Finally, our proposal is tested on a DSS case scenario about Biotechnology and Agriculture, whose knowledge base is the CEREALAB database as internal structured data, and the Web (e.g. PubMed) as external unstructured information.

show abstract

Overview of the CLEF 2005 Interactive Track

Cited by 17 publications

References 7 publications

Task-based interaction with an integrated multilingual, multimedia information system

Task-based interaction with an integrated multilingual, multimedia information system

An overview of semantic search evaluation initiatives

An authoring tool for decision support systems in context questions of ecological knowledge

Contact Info

Product

Resources

About