Differences in Crowdsourced vs. Lab-based Mobile and Desktop Input Performance Data

Findlater, Leah; Zhang, Joan; Froehlich, Jon E.; Moffatt, Karyn

doi:10.1145/3025453.3025820

Cited by 35 publications

(31 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In all, despite some statistically significant differences between the effects of novel and repeat primes, the effects are very small. This may, in part, reflect a general tendency for less-controlled participants to perform tasks faster with less accuracy than lab participants [27], although our visiblity tasks results indicate that some pre-existing response behaviours (e.g. to press the right-hand button) may have also influenced the experiment.…”

Section: Table 14 Results Summarymentioning

confidence: 74%

Subliminal semantic number processing on smartphones

Pinder

Vermeulen

Cowan

2018

Proceedings of the 20th International Conference on Human-Computer Interaction With Mobile Devices and Services

View full text Add to dashboard Cite

One potential method of improving the efficiency of human-computer interaction is to display information subliminally. Such information cannot be recalled consciously, but has some impact on the perceiver. However, it is not yet clear whether people can extract meaning from subliminal presentation of information in mobile contexts. We therefore explored subliminal semantic priming on smartphones. This builds on mixed evidence for subliminal priming across HCI in general, and mixed evidence for the effect of subliminal affective priming on smartphones. Our semi-controlled experiment (n=103) investigated subliminal processing of numerical information on smartphones. We found evidence that concealed transfer of information is possible to a very limited extent, but little evidence of a semantic effect. Overall, the impact is effectively negligible for practical applications. We discuss the implications of our results for real-world deployments and outline future research themes as HCI moves beyond mobile.

show abstract

Section: Table 14 Results Summarymentioning

confidence: 74%

Subliminal semantic number processing on smartphones

Pinder

Vermeulen

Cowan

2018

Proceedings of the 20th International Conference on Human-Computer Interaction With Mobile Devices and Services

View full text Add to dashboard Cite

show abstract

“…Along the same line, in [28] authors showed that rewarding workers when they quit their participation in a batch of HITs allows to filter out low-quality workers early, thus retaining only highly accurate workers. Recently, Findlater et al showed that results of online HCI experiments are similar to those achieved in the lab for desktop interactions, but this was less so in the case of mobile devices [20].…”

Section: Worker Differences and Participation Biasmentioning

confidence: 94%

Modus Operandi of Crowd Workers

Gadiraju

Checco

Gupta

et al. 2017

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

The ubiquity of the Internet and the widespread proliferation of electronic devices has resulted in flourishing microtask crowdsourcing marketplaces, such as Amazon MTurk. An aspect that has remained largely invisible in microtask crowdsourcing is that of work environments; defined as the hardware and software affordances at the disposal of crowd workers which are used to complete microtasks on crowdsourcing platforms. In this paper, we reveal the significant role of work environments in the shaping of crowd work. First, through a pilot study surveying the good and bad experiences workers had with UI elements in crowd work, we revealed the typical issues workers face. Based on these findings, we then deployed over 100 distinct microtasks on CrowdFlower, addressing workers in India and USA in two identical batches. These tasks emulate the good and bad UI element designs that characterize crowdsourcing microtasks. We recorded hardware specifics such as CPU speed and device type, apart from software specifics including the browsers used to complete tasks, operating systems on the device, and other properties that define the work environments of crowd workers. Our findings indicate that crowd workers are embedded in a variety of work environments which influence the quality of work produced. To confirm and validate our data-driven findings we then carried out semi-structured interviews with a sample of Indian and American crowd workers from this platform. Depending on the design of UI elements in microtasks, we found that some work environments support crowd workers more than others. Based on our overall findings resulting from all the three studies, we introduce ModOp, a tool that helps to design crowdsourcing microtasks that are suitable for diverse crowd work environments. We empirically show that the use of ModOp results in reducing the cognitive load of workers, thereby improving their user experience without effecting the accuracy or task completion time.

show abstract

“…Studies show that the mobile Web users perform different tasks compared to desktop Web users [9,27]. Findlater et al show that user interactions change between lab versus crowdsourced studies, and depend on whether the study is done on mobile or desktop devices [15]. XDBrowser [33], provides a new cross-device Web browser which can automatically translate a page design from one device type to another.…”

Section: Mobile Versus Non-mobile Quantificationmentioning

confidence: 99%

Modeling User-Centered Page Load Time for Smartphones

Kelton

Ryoo

Balasubramanian

et al. 2020

22nd International Conference on Human-Computer Interaction With Mobile Devices and Services

View full text Add to dashboard Cite

Page Load Time (PLT) is critical in measuring web page load performance. However, the existing PLT metrics are designed to measure the Web page load performance on desktops/laptops and do not consider user interactions on mobile browsers. As a result, they are ill-suited to measure mobile page load performance from the perspective of the user. In this work, we present the Mobile User-Centered Page Load Time Estimator (muPLT est), a model that estimates the PLT of users on Web pages for mobile browsers. We show that traditional methods to measure user PLT for desktops are unsuited to mobiles because they only consider the initial viewport, which is the part of the screen that is in the user's view when they first begin to load the page. However, mobile users view multiple viewports during the page load process since they start to scroll even before the page is loaded. We thus construct the muPLT est to account for page load activities across viewports. We train our model with crowdsourced scrolling behavior from live users. We show that muPLT est predicts ground truth user-centered PLT, or the muPLT, obtained from live users with an error of 10-15% across 50 Web pages. Comparatively, traditional PLT metrics perform within 44-90% of the muPLT. Finally, we show how developers can use the muPLT est to scalably estimate changes in user experience when applying different Web optimizations. CCS CONCEPTS • Human-centered computing → User models.

show abstract

Differences in Crowdsourced vs. Lab-based Mobile and Desktop Input Performance Data

Cited by 35 publications

References 24 publications

Subliminal semantic number processing on smartphones

Subliminal semantic number processing on smartphones

Modus Operandi of Crowd Workers

Modeling User-Centered Page Load Time for Smartphones

Contact Info

Product

Resources

About