User Interaction Sequences for Search Satisfaction Prediction

Mehrotra, Rishabh; Zitouni, Imed; Awadallah, Ahmed Hassan; Kholy, Ahmed El; Khabsa, Madian

doi:10.1145/3077136.3080833

Cited by 21 publications

(14 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many teams have tried to build machine learning models to create surrogate metrics and use them for decision making. For example, [22] uses a sequences of user interaction to predict search satisfaction. These metrics have greatly expanded the pool of candidate surrogate metrics and sparked research and discussion in this eld.…”

Section: Guidelines For Choosing the Right Surrogate Metricsmentioning

confidence: 99%

Online Experimentation with Surrogate Metrics: Guidelines and a Case Study

Duan

Zhang

2021

Proceedings of the 14th ACM International Conference on Web Search and Data Mining

View full text Add to dashboard Cite

A/B tests have been widely adopted across industries as the golden rule that guides decision making. However, the long-term true north metrics we ultimately want to drive through A/B test may take a long time to mature. In these situations, a surrogate metric which predicts the long-term metric is often used instead to conclude whether the treatment is eective. However, because the surrogate rarely predicts the true north perfectly, a regular A/B test based on surrogate metrics tends to have high false positive rate and the treatment variant deemed favorable from the test may not be the winning one. In this paper, we discuss how to adjust the A/B testing comparison to ensure experiment results are trustworthy. We also provide practical guidelines on the choice of good surrogate metrics. To provide a concrete example of how to leverage surrogate metrics for fast decision making, we present a case study on developing and evaluating the predicted conrmed hire surrogate metric in LinkedIn job marketplace.

show abstract

Section: Guidelines For Choosing the Right Surrogate Metricsmentioning

confidence: 99%

Online Experimentation with Surrogate Metrics: Guidelines and a Case Study

Duan

Zhang

2021

Proceedings of the 14th ACM International Conference on Web Search and Data Mining

View full text Add to dashboard Cite

show abstract

“…Using the novelty sub-scale of UES, [35] investigate differences between sequences of interactions and find that a user clicking on the next search result page followed by a click on a result can effectively discriminate between high and low novelty. Similarly, in exploring whether interaction sequences can be predictive of satisfaction, [22] found that a click followed by a long dwell time is correlated with satisfaction; in contrast, moving around the search engine result page is associated with dissatisfaction. In [34], UES sub-scales were used as separate prediction targets to investigate behaviour during a search task.…”

Section: 32mentioning

confidence: 99%

“…Interaction data have been used successfully in other domains, namely online news [9,12,15,18] and search [10,14,34]. It has also been used to extract user behaviours as proxies for engagement [3,5] and satisfaction [8,22]. As techniques and measures for engagement are untested in this domain, we study a production-quality and nationally released interactive TV show where the audience are given control over the presentation and their path through the story.…”

Section: Introductionmentioning

confidence: 99%

Using Interaction Data to Predict Engagement with Interactive Media

Carlton

Brown²,

Jay

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Media is evolving from traditional linear narratives to personalised experiences, where control over information (or how it is presented) is given to individual audience members. Measuring and understanding audience engagement with this media is important in at least two ways: (1) a post-hoc understanding of how engaged audiences are with the content will help production teams learn from experience and improve future productions; (2), this type of media has potential for real-time measures of engagement to be used to enhance the user experience by adapting content on-the-fly. Engagement is typically measured by asking samples of users to self-report, which is time consuming and expensive. In some domains, however, interaction data have been used to infer engagement.Fortuitously, the nature of interactive media facilitates a much richer set of interaction data than traditional media; our research aims to understand if these data can be used to infer audience engagement. In this paper, we report a study using data captured from audience interactions with an interactive TV show to model and predict engagement. We find that temporal metrics, including overall time spent on the experience and the interval between events, are predictive of engagement. The results demonstrate that interaction data can be used to infer users' engagement during and after an experience, and the proposed techniques are relevant to better understand audience preference and responses.CCS Concepts: • Human-centered computing → User models; User studies; • Information systems → Web log analysis.

show abstract

“…images, pausing to read and absorb content among others. Following Mehrotra et al [34], we extract interaction sequence from user interaction with the SERP. To do so, we construct a universal action sequence timeline from the following three di erent timelines:…”

Section: Example Sequencementioning

confidence: 99%

“…Table 1 lists the major actions considered. For details on the actions considered, the interested is referred to Mehrotra et al [34].…”

Section: Example Sequencementioning

confidence: 99%