Predicting Game Difficulty and Churn Without Players

Roohi, Shaghayegh; Relas, Asko; Takatalo, Jari; Heiskanen, Henri; Hämäläinen, Perttu

doi:10.1145/3410404.3414235

Cited by 26 publications

(37 citation statements)

References 88 publications

(102 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is important to acknowledge that the measurement of subjective experience through self-reported survey scales is only one of several research methods used at CHI PLAY. For example, and this is by no means exhaustive, player behavior and experience is also explored using qualitative methods such as grounded theory [9] and ethnomethodologyinformed ethnography [49] or with AI agents in simulations [42]. Indeed, when submitting a paper to the CHI PLAY conference, authors are able to choose from many different contributions beyond research which involves the quantitative measurement of constructs [2].…”

Section: Introductionmentioning

confidence: 99%

Transparency in Measurement Reporting: A Systematic Literature Review of CHI PLAY

Aeschbach¹,

Perrig²,

Weder³

et al. 2021

Preprint

View full text Add to dashboard Cite

Measuring theoretical concepts, so-called constructs, is a central challenge of Player Experience research. Building on recent work in HCI and psychology, we conducted a systematic literature review to study the transparency of measurement reporting. We accessed the ACM Digital Library to analyze all 48 full papers published at CHI PLAY 2020, of those, 24 papers used self-report measurements and were included in the full review. We assessed specifically, whether researchers reported What, How and Why they measured. We found that researchers matched their measures to the construct under study and that administrative details, such as number of points on a Likert-type scale, were frequently reported. However, definitions of the constructs to be measured and justifications for selecting a particular scale were sparse. Lack of transparency in these areas threaten the validity of singular studies, but further compromise the building of theories and accumulation of research knowledge in meta-analytic work. This work is limited to only assessing the current transparency of measurement reporting at CHI PLAY 2020, however we argue this constitutes a fair foundation to asses potential pitfalls. To address these pitfalls, we propose a prescriptive model of a measurement selection process, which aids researchers to systematically define their constructs, specify operationalizations, and justify why these measures were chosen. Future research employing this model should contribute to more transparency in measurement reporting. The research was funded through internal resources. All materials are available on https://osf.io/4xz2v/.

show abstract

Section: Introductionmentioning

confidence: 99%

Transparency in Measurement Reporting: A Systematic Literature Review of CHI PLAY

Aeschbach¹,

Perrig²,

Weder³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…We account for the complexity of engagement as player experience [8] by operationalizing it in terms of player churn, which can be triggered by any of the underlying factors [38]. Existing computational work has considered the churn probability of an individual player [6,47], or the average churn rate measured over a population of players and a part of the game such as a single game level [45,46]. Predicting the churn probability of a particular player allows game developers to offer personalized incentives to continue playing.…”

Section: Operationalizing Engagement and Difficultymentioning

confidence: 99%

“…In their specific study, they found that the number of moves a DRL agent needs for completing a level has the highest correlation with human player pass rates if one uses the maximum found among the top 5% best runs. As part of the second extension contributed in this paper, we test their hypothesis by modifying the AI gameplay features used in our previous approach [46] (see RQ2).…”

Section: Operationalizing Engagement and Difficultymentioning

confidence: 99%

“…In this paper, we introduce two extensions to an automated playtesting algorithm developed in prior work [46], in which we used DRL game-playing agents for player modelling, more specifically to predict player engagement and game difficulty. In particular, this algorithm combined simulated game-playing via deep reinforcement learning (DRL) agents with a meta-level player population simulation.…”

Section: Introductionmentioning

confidence: 99%

“…To answer these questions, we have implemented and compared 2 × 2 × 3 = 12 approaches resulting from combining the two predictive models introduced in our previous work [46] with either DRL or MCTS agents (RQ1) and three feature selection approaches (RQ2). Similar to the original paper, we apply and evaluate our extension in Angry Birds Dream Blast [18].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Predicting Game Difficulty and Engagement Using AI Players

Roohi

Guckelsberger

Relas³

et al. 2021

Proc. ACM Hum.-Comput. Interact.

Self Cite

View full text Add to dashboard Cite

This paper presents a novel approach to automated playtesting for the prediction of human player behavior and experience. We have previously demonstrated that Deep Reinforcement Learning (DRL) game-playing agents can predict both game difficulty and player engagement, operationalized as average pass and churn rates. We improve this approach by enhancing DRL with Monte Carlo Tree Search (MCTS). We also motivate an enhanced selection strategy for predictor features, based on the observation that an AI agent's best-case performance can yield stronger correlations with human data than the agent's average performance. Both additions consistently improve the prediction accuracy, and the DRL-enhanced MCTS outperforms both DRL and vanilla MCTS in the hardest levels. We conclude that player modelling via automated playtesting can benefit from combining DRL and MCTS. Moreover, it can be worthwhile to investigate a subset of repeated best AI agent runs, if AI gameplay does not yield good predictions on average.

show abstract

Automated Game Design Testing Using Machine Learning

Stephens,

Exton

2024

Encyclopedia of Computer Graphics and Games

View full text Add to dashboard Cite

Predicting Game Difficulty and Churn Without Players

Cited by 26 publications

References 88 publications

Transparency in Measurement Reporting: A Systematic Literature Review of CHI PLAY

Transparency in Measurement Reporting: A Systematic Literature Review of CHI PLAY

Predicting Game Difficulty and Engagement Using AI Players

Automated Game Design Testing Using Machine Learning

Contact Info

Product

Resources

About