A Tool for Conducting User Studies on Mobile Devices

Costa, Luca; Aliannejadi, Mohammad; Crestani, Fábio

doi:10.1145/3343413.3377985

Cited by 5 publications

(1 citation statement)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We use the UniMobile 6 dataset released by Aliannejadi et al [4,8]. The dataset contains 5,812 cross-app mobile search queries for 206 search tasks spread across multiple task categories.…”

Section: Datamentioning

confidence: 99%

The Impact of User Demographics and Task Types on Cross-App Mobile Search

Aliannejadi¹,

Crestani²,

Huibers³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Recent developments in the mobile app industry have resulted in various types of mobile apps, each targeting a different need and a specific audience. Consequently, users access distinct apps to complete their information need tasks. This leads to the use of various apps not only separately, but also collaboratively in the same session to achieve a single goal. Recent work has argued the need for a unified mobile search system that would act as metasearch on users' mobile devices. The system would identify the target apps for the user's query, submit the query to the apps, and present the results to the user in a unified way. In this work, we aim to deepen our understanding of user behavior while accessing information on their mobile phones by conducting an extensive analysis of various aspects related to the search process. In particular, we study the effect of task type and user demographics on their behavior in interacting with mobile apps. Our findings reveal trends and patterns that can inform the design of a more effective mobile information access environment.

show abstract

“…We use the UniMobile 6 dataset released by Aliannejadi et al [4,8]. The dataset contains 5,812 cross-app mobile search queries for 206 search tasks spread across multiple task categories.…”

Section: Datamentioning

confidence: 99%

The Impact of User Demographics and Task Types on Cross-App Mobile Search

Aliannejadi¹,

Crestani²,

Huibers³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

The Impact of User Demographics and Task Types on Cross-App Mobile Search

Aliannejadi

Crestani

Huibers

et al. 2021

Flexible Query Answering Systems

Self Cite

View full text Add to dashboard Cite

User interaction data is an important source of supervision in counterfactual learning to rank (CLTR). Such data suffers from presentation bias. Much work in unbiased learning to rank (ULTR) focuses on position bias, i.e., items at higher ranks are more likely to be examined and clicked. Inter-item dependencies also influence examination probabilities, with outlier items in a ranking as an important example. Outliers are defined as items that observably deviate from the rest and therefore stand out in the ranking. In this paper, we identify and introduce the bias brought about by outlier items: users tend to click more on outlier items and their close neighbors.To this end, we first conduct a controlled experiment to study the effect of outliers on user clicks. Next, to examine whether the findings from our controlled experiment generalize to naturalistic situations, we explore real-world click logs from an e-commerce platform. We show that, in both scenarios, users tend to click significantly more on outlier items than on non-outlier items in the same rankings. We show that this tendency holds for all positions, i.e., for any specific position, an item receives more interactions when presented as an outlier as opposed to a non-outlier item. We conclude from our analysis that the effect of outliers on clicks is a type of bias that should be addressed in ULTR. We therefore propose an outlieraware click model that accounts for both outlier and position bias, called outlier-aware position-based model (OPBM). We estimate click propensities based on OPBM; through extensive experiments performed on both real-world e-commerce data and semi-synthetic data, we verify the effectiveness of our outlier-aware click model. Our results show the superiority of OPBM against baselines in terms of ranking performance and true relevance estimation. CCS CONCEPTS• Information systems → Learning to rank.

show abstract