Statistical Significance Testing at CHI PLAY: Challenges and Opportunities for More Transparency

Vornhagen, Jan Benjamin; Tyack, April; Mekler, Elisa D.

doi:10.1145/3410404.3414229

Cited by 39 publications

(27 citation statements)

References 134 publications

(183 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In line with Vornhagen et al [53] we discuss why precisely defining a construct before its measurement is necessary for interpretable and generalizable findings. Referencing DeVellis [18] conceptualization of content validity, researchers cannot be certain that the content, or items, of a scale is appropriate for their chosen construct if they do not have a clear definition of their construct.…”

Section: Defining Constructssupporting

confidence: 58%

“…As explained above, the constructs we measure and the measurement we chose for these constructs are a fundamental challenge of every empirical study. In their paper Vornhagen et al [53] gave a broad idea as to how theoretical variables should be reported in studies. They explain these constructs must be precisely defined, the measurement procedure must be elaborated on and the selection of measurements must be justified.…”

Section: General Concerns When Reporting Measurementsmentioning

confidence: 99%

See 1 more Smart Citation

Transparency in Measurement Reporting: A Systematic Literature Review of CHI PLAY

Aeschbach¹,

Perrig²,

Weder³

et al. 2021

Preprint

View full text Add to dashboard Cite

Measuring theoretical concepts, so-called constructs, is a central challenge of Player Experience research. Building on recent work in HCI and psychology, we conducted a systematic literature review to study the transparency of measurement reporting. We accessed the ACM Digital Library to analyze all 48 full papers published at CHI PLAY 2020, of those, 24 papers used self-report measurements and were included in the full review. We assessed specifically, whether researchers reported What, How and Why they measured. We found that researchers matched their measures to the construct under study and that administrative details, such as number of points on a Likert-type scale, were frequently reported. However, definitions of the constructs to be measured and justifications for selecting a particular scale were sparse. Lack of transparency in these areas threaten the validity of singular studies, but further compromise the building of theories and accumulation of research knowledge in meta-analytic work. This work is limited to only assessing the current transparency of measurement reporting at CHI PLAY 2020, however we argue this constitutes a fair foundation to asses potential pitfalls. To address these pitfalls, we propose a prescriptive model of a measurement selection process, which aids researchers to systematically define their constructs, specify operationalizations, and justify why these measures were chosen. Future research employing this model should contribute to more transparency in measurement reporting. The research was funded through internal resources. All materials are available on https://osf.io/4xz2v/.

show abstract

Section: Defining Constructssupporting

confidence: 58%

Section: General Concerns When Reporting Measurementsmentioning

confidence: 99%

Transparency in Measurement Reporting: A Systematic Literature Review of CHI PLAY

Aeschbach¹,

Perrig²,

Weder³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…We surveyed 237 boardgame players about their attitudes towards and experience with HDBs. This was explicitly exploratory research [121,132]; we wanted to see what people would tell us about HDBs, given the opportunity. Respondents were recruited via the project's and researchers' social media accounts, as well as at international gaming events, trade fairs and conventions in Europe, the US, and Australia, as well as through posts on hobbyist gaming websites.…”

Section: Surveymentioning

confidence: 99%

Unpacking “Boardgames with Apps”: The Hybrid Digital Boardgame Model

Rogerson

Sparrow

Gibbs

2021

Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

Increasingly, modern boardgames incorporate digital apps and tools to deliver content in novel ways. Despite disparate approaches to incorporating digital tools in otherwise non-digital boardgames, there has to date been no detailed classification of the different roles that these tools play in supporting gameplay. In this paper, we present a model for understanding hybrid boardgame play as performing a set of discrete functions. Through a mixed-methods approach incorporating critical play sessions, a survey of 237 boardgame players, and interviews with 18 boardgame designers and publishers, we identified the key functions performed by the digital tools in these games. Using affinity mapping, we grouped these functions into eight core categories, which were tested and refined using a survey of 44 boardgame players and designers. This model defines and classifies the diverse functions that digital tools perform in hybrid digital boardgames, demonstrating their range and potential application for researchers and game designers. CCS CONCEPTS •Human-centered computing~Interaction design~Empirical studies in interaction design •Applied computing~Computers in other domains~Personal computers and PC applications~Computer games

show abstract

“…This finding, however, is not entirely surprising, not only because of the different disciplines involved but also because some of the contributing research areas struggle with similar challenges themselves. For instance, Human-Computer-Interaction research that investigates video games often encounters substantial methodological and statistical challenges (Vornhagen et al, 2020), whereas educational psychology faces related issues, such as the replication crisis (Maxwell et al, 2015) or infrequent improvements to theoretical frameworks (Alexander, 2018;Mayer, 2018). Nonetheless, the results of this review can be used to address these critical issues and improve future empirical research in the field.…”

Section: Overall Conclusionmentioning

confidence: 99%

A Systematic Literature Review of Analytics for Adaptivity Within Educational Video Games

Ninaus

Nebel

2021

Front. Educ.

View full text Add to dashboard Cite

Research has shown that serious games, digital game-based learning, and educational video games can be powerful learning instruments. However, experimental and meta-research have revealed that several moderators and variables influence the resulting learning outcomes. Advances in the areas of learning and game analytics potentially allow for controlling and improving the underlying learning processes of games by adapting their mechanics to the individual needs of the learner, to properties of the learning material, and/or to environmental factors. However, the field is young and no clear-cut guidelines are yet available. To shed more light on this topic and to identify common ground for further research, we conducted a systematic and pre-registered analysis of the literature. Particular attention was paid to different modes of adaptivity, different adaptive mechanisms in various learning domains and populations, differing theoretical frameworks, research methods, and measured concepts, as well as divergent underlying measures and analytics. Only 10 relevant papers were identified through the systematic literature search, which confirms that the field is still in its very early phases. The studies on which these papers were based, however, show promise in terms of the efficacy of adaptive educational games. Moreover, we identified an increased interest in the field of adaptive educational games and in the use of analytics. Nevertheless, we also identified a clear lack of common theoretical foundations as well as the application of rather heterogenous methods for investigating the effects of adaptivity. Most problematic was the lack of sufficient information (e.g., descriptions of used games, adaptive mechanisms), which often made it difficult to draw clear conclusions. Future studies should therefore focus on strong theory building and adhere to reporting standards across disciplines. Researchers from different disciplines must act in concert to advance the current state of the field in order to maximize its potential.

show abstract

Statistical Significance Testing at CHI PLAY: Challenges and Opportunities for More Transparency

Cited by 39 publications

References 134 publications

Transparency in Measurement Reporting: A Systematic Literature Review of CHI PLAY

Transparency in Measurement Reporting: A Systematic Literature Review of CHI PLAY

Unpacking “Boardgames with Apps”: The Hybrid Digital Boardgame Model

A Systematic Literature Review of Analytics for Adaptivity Within Educational Video Games

Contact Info

Product

Resources

About