On correlation of absence time and search effectiveness

Abstract

Online search evaluation metrics are typically derived based on implicit feedback from the users. For instance, computing the number of page clicks, number of queries, or dwell time on a search result. In a recent paper, Dupret and Lalmas introduced a new metric called absence time, which uses the time interval between successive sessions of users to measure their satisfaction with the system. They evaluated this metric on a version of Yahoo! Answers. In this paper, we investigate the effectiveness of absence time in evaluating new features in a web search engine, such as new ranking algorithm or a new user interface. We measured the variation of absence time to the effects of 21 experiments performed on a search engine. Our findings show that the outcomes of absence time agreed with the judgement of human experts performing a thorough analysis of a wide range of online and offline metrics in 14 out of these 21 cases.We also investigated the relationship between absence time and a set of commonly-used covariates (features) such as the number of queries and clicks in the session. Our results suggest that users are likely to return to the search engine sooner when their previous session has more queries and more clicks.

Future User Engagement Prediction and Its Application to Improve the Sensitivity of Online Experiments

“…The absence time (the time between two user visits) on a par with other engagement metrics was applied to compare different ranking algorithms used at Yahoo! Answers [9] and to evaluate a web search engine changes in ranking algorithm and user interface [3]. The novel periodicity engagement metrics of user behavior (resulted from the Discrete Fourier transform of 4 state-of-the-art engagement measures) were applied in [8] to evaluate, by means of A/B experiments, different changes of the search engine ranking algorithm, changes of the user interface, and changes of the engine's efficiency.…”

“…Following common practice [12,14,9,27,3,8], we define a session as a sequence of user actions whose dwell times are less than 30 minutes. The presence time PT is measured as the sum of all session lengths (in seconds) observed during the considered time period, while the total absence time is measured as the length of the considered time period minus the presence time.…”

“…The basic examples of such approaches are discussed in [18]: the use of a key metric with lower variance and the elimination of users who were not affected by the service change in the treatment group 3 . The applicability of these approaches is limited and strongly depends on the case of a conducted experiment.…”

Engagement Periodicity in Search Engine Usage

“…Answers by means of survival analysis. This methodology was applied in the recent study [2] to evaluate a web search engine changes in its ranking algorithm and its user interface. In our study, we utilize 56 new features resulted from the DFT of 4 engagement measures in order to evaluate, by means of A/B tests, different changes of the search engine ranking algorithm, changes of the user interface, and changes of the engine efficiency (see Section 7).…”

“…yandex.com10 A session is commonly defined as a sequence of actions whose dwell times are less than 30 minutes[2,5,12,21] 11. This length allows to catch periodicity between weeks and, meanwhile, study its long-term changes as we further describe 12.…”