Repeated measures design in crowdsourcing-based experiments for visualization

Abstract

Crowdsourcing platforms, such as Amazon's Mechanical Turk (MTurk), are providing visualization researchers with a new avenue for conducting empirical studies. While such platforms offer several advantages over lab-based studies, they also feature some "unknown" or "uncontrolled" variables, which could potentially introduce serious confounding effects in the resultant measurement data. In this paper, we present our experience of using repeated measures in three empirical studies using MTurk. Each study presented participants with a set of stimuli, each featuring a condition of an independent variable. Participants were exposed to stimuli repeatedly in a pseudo-random order through four trials and their responses were measured digitally. Only a small portion of the participants were able to perform with absolute consistency for all stimuli throughout each experiment. This suggests that a repeated measures design is highly desirable (if not essential) when designing empirical studies for crowdsourcing platforms. Additionally, the majority of participants performed their tasks with reasonable consistency when all stimuli in an experiment are considered collectively. In other words, to most participants, inconsistency occurred occasionally. This suggests that crowdsourcing remains a valid experimental environment, provided that one can integrate the means to observe and alleviate the potential confounding effects of "unknown" or "uncontrolled" variables in the design of the experiment.

Du Bois Wrapped Bar Chart: Visualizing categorical data with disproportionate values

“…To calculate, we first averaged performance on a participant basis per factor (e.g., Entropy bin) and then bootstrapped on user level performance. This enabled us to control for heterogeneity between participant performance, which is commonly found in crowdsourced visualization experiments [2]. Since Study 2 is a within-subjects study, we report paired Cohen's d for within-subject paired samples [23].…”

“…• We design and develop the Du Bois wrapped bar chart 2 inspired by Du Bois' work for web-based visualizations using D3.js.…”

“…In the following sections, we review related work, design of wrapped bar charts, experiment design, discussion of results, and conclude with limitations and future works. 2 The similar term "wrapped bar graph" has been used for a different visualization technique. Stephen Few [17] introduced this term for a chart that splits the sorted bars in a horizontal bar chart into multiple columns to eliminate the need for scrolling [24].…”

CommViz: Visualization of semantic patterns in large social communication networks

“…However, very few techniques or tools are built for identical purposes in the same domains, and on similar or the exact same types of data. 54 While most traditional user-centered studies were carried out in collaboration with known ''expert'' users in the domain, recently crowdsourcing has been explored for such evaluations, 55,56 resulting in a greatly increased and diversified set of users. 57 We also chose to make use of a crowdsourcing methodology, utilizing Amazon's Mechanical Turk Service.…”

One Shot Crowdtesting: Approaching the Extremes of Crowdsourced Subjective Quality Testing

“…In case of a factorial test design, separate groups of subjects are required for each combination of the different values of the independent variables. Among others, [9] found that inconsistency occurred occasionally to most online participants, and they concluded that a within-subject design is essential for empirical crowdsourcing studies. Nevertheless, monitoring of task execution allows to immediately detect inconsistencies and filter out such participants (e.g., [10]).…”