Task Intelligence Workshop @ WSDM 2019

Awadallah, Ahmed Hassan; Gurrin, Cathal; Sanderson, Mark; White, Ryen W.

doi:10.1145/3289600.3291374

Cited by 1 publication

(1 citation statement)

References 4 publications

(2 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Clearly, to tackle complex tasks by means of search systems, users would require support for task achievement that extends well beyond the list of 'ten blue links'. While research investigation in the area of task-based search gained maturity since the early works rooted in the interactive Information Retrieval (IR) community [39], the topic still attracts an increasing interest by researchers, as acknowledged by the several recent related workshops among which: Supporting Complex Search Tasks (2015-2017) [1], Learning from User Interactions (2018) [28], Task Intelligence (2019) [15] and Investigating Learning During Web Search (2020) 2 .…”

Section: Introductionmentioning

confidence: 99%

CoST: An annotated Data Collection for Complex Search

Dosso

Moreno

Chevalier

et al. 2021

Proceedings of the 30th ACM International Conference on Information &Amp; Knowledge Management

View full text Add to dashboard Cite

While great progress is made in the area of information access, there are still open issues that involve designing intelligent systems supporting task-based search. Despite the importance of task-based search, the information retrieval and information science communities still feel the lack of open-ended and annotated datasets that enable the evaluation of a number of related facets of search tasks in downstream applications. Existing datasets are either sampled from large-scale logs but provide poor annotations, or sampled from lower-scale user studies but focus on ranked list evaluation. In this work, we present CoST 1 : a novel richly annotated dataset for evaluating complex search tasks, collaboratively designed by researchers from the computer science and cognitive psychology domains, and intended to answer a wide range of research questions dealing with task-based search. CoST includes 5667 queries recorded in 630 task-based sessions that result from a user study involving 70 french native participants who are expert in one among 3 different domains of expertise (computer science, medicine, psychology). Each participant completed 15 tasks with 5 different types of cognitive complexity (fact-finding, exploratory learning, decisionmaking, problem-solving, multicriteria-inferential). In addition to search data (e.g., queries and clicks), CoST provides task and sessionrelated data, task annotations and query annotations. We illustrate possible usages of CoST through the evaluation of query classification models and the understanding of the effect of task complexity and domain on user's search behavior.

show abstract

Section: Introductionmentioning

confidence: 99%