Evaluating and predicting user engagement change with degraded search relevance

Song, Yang; Shi, Xiaolin; Fu, Xin

doi:10.1145/2488388.2488494

Cited by 40 publications

(40 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Click models [18,37,42,8,6] making usages of query logs and click logs [23,45,19] can be used in many of the semantic search subjects -usually click models are understood as statistical tools using query click log data to adjust the ranking to offer for users better web search results so that users can search more accurately and more efficiently. Starting from 2015 that click models have been researched for mobiles (cell phones) by PocketTrend system using search logs [30].…”

Section: Related Workmentioning

confidence: 99%

Semantic Search on Applicant Tracking System

Ha¹,

Rahman²

2017

International Journal of Advanced Research in Computer and Comm

View full text Add to dashboard Cite

Web search engines often federate many user queries to relevant structured databases. For example, a recruitment-related query might be federated to a jobseekers-and-employers database containing their resumes and skills. The relevant structured data items are then returned to the user along with web search results. Though each structured database is searched in isolation, the search often produces empty / incomplete results as the database may not contain the required information to answer the query. Starting from our Applicant Tracking System (ATS), we have 16 development databases of over 650,000 profile documents of resumes / cover letters / skills. There are on average 238 keywords per document. In fact, per minute there can be up to 200,000 transactions within all these databases. Our existing traditional database search technique (by full-text keyword PostgreSQL search) can be frozen or taking very long to respond unless if we cut off / only search from top profiles, we will not have the search results ready by thirty seconds, but this cut-off limitation returned incorrect results; for example for a query "Jet fuel Thermal Oxidation" to request information about job seekers whose resumes contain skills in Oil and Gas industry, in the top ten results there was a conflict in relevance ranking. In order to research a more suitable full-text keyword search technique better than the existing database search, we considered employment of semantic search models. Our semantic search technique has 88% -91.22% accuracy with very much quicker queries that can help users to make a search of 4 keywords of skills completed from 1 second to 28 seconds. Furthermore, the semantic search engine becomes very strong that users can search by entering a whole text paragraph. We designed a combination of semantic search that look for web pages per search and database search.

show abstract

Section: Related Workmentioning

confidence: 99%

Semantic Search on Applicant Tracking System

Ha¹,

Rahman²

2017

International Journal of Advanced Research in Computer and Comm

View full text Add to dashboard Cite

show abstract

“…While loyalty and popularity essentially make sense for relative comparison of websites, activity enables measuring engagement for a particular website independently of other websites. The most commonly used activity metrics include number of queries per session, number of clicks, number of clicks per query, dwell (presence) time (see e.g., [10,32]). …”

Section: Related Workmentioning

confidence: 99%

Benchmarking Exploratory OLAP

Djedaini

Furtado

Labroche

et al. 2017

Performance Evaluation and Benchmarking. Traditional - Big Data - Interest of Things

View full text Add to dashboard Cite

Abstract. Supporting interactive database exploration (IDE) is a problem that attracts lots of attention these days. Exploratory OLAP (OnLine Analytical Processing) is an important use case where tools support navigation and analysis of the most interesting data, using the best possible perspectives. While many approaches were proposed (like query recommendation, reuse, steering, personalization or unexpected data recommendation), a recurrent problem is how to assess the effectiveness of an exploratory OLAP approach. In this paper we propose a benchmark framework to do so, that relies on an extensible set of user-centric metrics that relate to the main dimensions of exploratory analysis. Namely, we describe how to model and simulate user activity, how to formalize our metrics and how to build exploratory tasks to properly evaluate an IDE system under test (SUT). To the best of our knowledge, this is the first proposal of such a benchmark. Experiments are two-fold: first we evaluate the benchmark protocol and metrics based on synthetic SUTs whose behavior is well known. Second, we concentrate on two different recent SUTs from IDE literature that are evaluated and compared with our benchmark. Finally, potential extensions to produce an industrystrength benchmark are listed in the conclusion.

show abstract

“…A rich practical experience of using this methodology for different evaluation cases in many web companies was studied recently [13,19,28,15]. These studies concern, inter alia, experiments with different components of a web service (the user interface [13,8], ranking algorithms [27,8], etc. ), largescale experimental infrastructure [28,15], different aspects of user behavior and interaction with a web service (clicks [14,16], speed [19,16], abandonment [16], periodicity [8], etc.…”

Section: Related Workmentioning

confidence: 99%

“…Basic sensitivity improvement techniques for an online controlled experiment were concerned in [18]: increasing of the experiment duration or of the user population, participated in the experiment [5]; the use of another evaluation metric with lower variance [8]; and the elimination of users who were not affected by the service change in the treatment group [27]. More sophisticated variation reduction techniques were proposed in [7]: the stratification and the usage of control variates based on pre-experiment data.…”

Section: Related Workmentioning

confidence: 99%

“…That is why engagement metrics are often considered to be most appropriate for online evaluation. In this study, we pay special attention to the metrics that reflect the loyalty aspect of engagement: the state-of-the-art number of sessions per user metric [14,27] (which is accepted as the "North-star" for A/B testing evaluation in major search engine companies like Bing [15,16]) and the recently proposed absence time metric [9]. Our approach raises the problem of prediction of the future individual user engagement.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Future User Engagement Prediction and Its Application to Improve the Sensitivity of Online Experiments

Drutsa

Gusev

Serdyukov

2015

Proceedings of the 24th International Conference on World Wide Web

View full text Add to dashboard Cite

Modern Internet companies improve their services by means of data-driven decisions that are based on online controlled experiments (also known as A/B tests). To run more online controlled experiments and to get statistically significant results faster are the emerging needs for these companies. The main way to achieve these goals is to improve the sensitivity of A/B experiments. We propose a novel approach to improve the sensitivity of user engagement metrics (that are widely used in A/B tests) by utilizing prediction of the future behavior of an individual user. This problem of prediction of the exact value of a user engagement metric is also novel and is studied in our work. We demonstrate the effectiveness of our sensitivity improvement approach on several real online experiments run at Yandex. Especially, we show how it can be used to detect the treatment effect of an A/B test faster with the same level of statistical significance.

show abstract

Evaluating and predicting user engagement change with degraded search relevance

Cited by 40 publications

References 20 publications

Semantic Search on Applicant Tracking System

Semantic Search on Applicant Tracking System

Benchmarking Exploratory OLAP

Future User Engagement Prediction and Its Application to Improve the Sensitivity of Online Experiments

Contact Info

Product

Resources

About