Does Document Relevance Affect the Searcher's Perception of Time?

Abstract

Time plays an essential role in multiple areas of Information Retrieval (IR) studies such as search evaluation, user behavior analysis, temporal search result ranking and query understanding. Especially, in search evaluation studies, time is usually adopted as a measure to quantify users' efforts in search processes. Psychological studies have reported that the time perception of human beings can be affected by many stimuli, such as attention and motivation, which are closely related to many cognitive factors in search. Considering the fact that users' search experiences are affected by their subjective feelings of time, rather than the objective time measured by timing devices, it is necessary to look into the different factors that have impacts on search users' perception of time. In this work, we make a first step towards revealing the time perception mechanism of search users with the following contributions: (1) We establish an experimental research framework to measure the subjective perception of time while reading documents in search scenario, which originates from but is also different from traditional time perception measurements in psychological studies. (2) With the framework, we show that while users are reading result documents, document relevance has small yet visible effect on search users' perception of time. By further examining the impact of other factors, we demonstrate that the effect on relevant documents can be also influenced by individuals and tasks. (3) We conduct a preliminary experiment in which the difference between perceived time and dwell time is taken into consideration in a search evaluation task. We found that the revised framework achieved a better correlation with users' satisfaction feedbacks. This work may help us better understand the time perception mechanism of search users and provide insights in how to better incorporate time factor in search evaluation studies.

A Systematic Review of Cost, Effort, and Load Research in Information Search and Retrieval, 1972–2020

“…Maxwell and Azzopardi [96] observed a similar finding, with their results indicating that as the relative cost of querying increased, users issued fewer queries and examined more documents per query. As the information search task becomes more complex, users perceive higher levels of effort [91] and workload [120], and effort measures have been found to correlate with all information task characteristics associated with task difficulty (i.e., higher number of steps to achieve the task goal; intellectual task product; amorphous task goal). In tasks with the highest difficulty level, users are found to perform fewer actions, clicks, and bookmarks per query Capra et al [18].…”

“…Cost: The term "cost" was used in all sixteen articles which examined the construct. Two articles measuring cost also used the term effort interchangeably [91,139]. For the eight articles which defined cost, two main conceptual categories were identified:…”

“…• Time Orientated: Cost was most frequently characterised in relation to the time spent during the user-system interaction (𝑁 = 7) [8-10, 22, 29, 91, 96]. For example, "cost is often measured by the time of a series of actions, such as formulating queries, examining snippets, clicks on results etc" [91]; and "cost is often considered as the amount of time spent" [9]. • Interaction Orientated/Count Based: Cost was also defined in relation to the interactions/number of actions occurring between the user and the system (𝑁 = 4) [9,22,96,144].…”

“…• Meta-cognition/Conscious Awareness: The third category (𝑁 = 3) [77,91,115] to emerge refers to effort as a volitional and intentional process in which the individual is consciously aware. For example, "effort reflects a voluntary allocation of effort that can be reported by the individual" [115].…”

“…Rather, these internal and external resources are limited in capacity and the user must therefore make decisions during the ISR process based on these limitations. The demands imposed during the search process are well recognised, and numerous studies have demonstrated that the cost, effort, and load (CEL) experienced during the search process is affected by not only the search task [27,91,120], but also the system [21,23,37] and the individual characteristics [24,57] of the user. Despite this recognition, our recent perspectives paper [97] highlighted two prominent challenges facing the ISR community in regard to CEL research.…”