Local Standards for Sample Size at CHI

Abstract

We describe the primary ways researchers can determine the size of a sample of research participants, present the benefits and drawbacks of each of those methods, and focus on improving one method that could be useful to the CHI community: local standards. To determine local standards for sample size within the CHI community, we conducted an analysis of all manuscripts published at CHI2014. We find that sample size for manuscripts published at CHI ranges from 1-916,000 and the most common sample size is 12. We also find that sample size differs based on factors such as study setting and type of methodology employed. The outcome of this paper is an overview of the various ways sample size may be determined and an analysis of local standards for sample size within the CHI community. These contributions may be useful to researchers planning studies and reviewers evaluating the validity of results.

How Relevant are Incidental Power Poses for HCI?

“…In practice, the effect remains difficult to study as the small effect size requires large participant pools to reliably detect the effect. Such large participant pools are rather uncommon in HCI [12] with the exception of crowdsourced online experiments where the reduced experimental control might negatively effect the signal to noise ratio of an already small effect. Besides such practical considerations, the very large effect on (dis)comfort severely limits the range of acceptable expansive interfaces.…”

“…The analysis that brought forward this finding was exploratory, and our experiment included only 80 participants -more than usual inperson experiments in HCI [12] but less than the failed replications of explicitly elicited power poses. We suggest that replications could focus on specific, promising or important application areas where effects in different directions might have an either desirable or detrimental impact on people's lives, and participants should be screened for relevant personality traits, such as impulsiveness or the "the big-five" [33], to examine interaction effects with these covariates.…”

Supporting Self-Assembly

“…The user experience questionnaire 2 (UEQ) is used in our experiment. Users' burden was reported as a high negative factor leading to users' quitting.…”

“…We conduct the experiment with 20 participants [2]. Before the experiment, we introduce the experiment to the candidates by saying it is to promote daily exercise to promote healthy behavior.…”

Moving Transparent Statistics Forward at CHI

“…These could be patterned after (or even use) badges from the Open Science Framework (OSF) [1]  Voluntary pre-registration of analyses for papers  MOOCs or other courses for authors and reviewers  Suggested graduate curricula for HCI PhDs  Adding new metadata in PCS, such as experimental meta-data similar to that collected manually by Caine [2] to facilitate tracking changes in statistical practice in the field over time.…”