Estimating Player Completion Rate in Mobile Puzzle Games Using Reinforcement Learning

Abstract

In this work we investigate whether it is plausible to use the performance of a reinforcement learning (RL) agent to estimate the difficulty measured as the player completion rate of different levels in the mobile puzzle game Lily's Garden.For this purpose we train an RL agent and measure the number of moves required to complete a level. This is then compared to the level completion rate of a large sample of real players.We find that the strongest predictor of player completion rate for a level is the number of moves taken to complete a level of the ∼5% best runs of the agent on a given level. A very interesting observation is that, while in absolute terms, the agent is unable to reach human-level performance across all levels, the differences in terms of behaviour between levels are highly correlated to the differences in human behaviour. Thus, despite performing subpar, it is still possible to use the performance of the agent to estimate, and perhaps further model, player metrics.

Predicting Game Difficulty and Churn Without Players

“…However, the average moves left and average cleared goals features correlate most with human pass rates if calculated from the best AI runs. This supports Kristensen et al 's [30] hypothesis for at least some, although not all, investigated gameplay features. Based on the results in Figure 3, we used the top 15% runs for computing the average moves left, and top 5% for computing the average cleared goals percentage features.…”

“…F3P: Similar to F3, but with averages computed from a subset of the best rather than all runs, characterized by a high number of moves left after passing a level. The last selection strategy, F3P, is inspired by Kristensen et al [30], who have observed a stronger correlation between human ground truth data and features extracted from best runs (see Section 3). To back this up and to identify the best sample size, we computed the Spearman correlation between the ground truth human pass rate and the average of the DRL agent's performance over different percentages of the agent's best runs (Figure 3).…”

Predicting Game Difficulty and Engagement Using AI Players

“…We hypothesize that employing MCTS in addition to DRL can improve the predictions, as the combination has been highly successful in playing very challenging games like Go [54]. We moreover adopt Kristensen et al 's [30] hypothesis that the difficulty of the hardest levels could be better predicted by the agent's best case performance rather than average performance. To implement this, we employed multiple attempts per level and only utilized the data from the most successful attempts as gameplay features.…”

“…Kristensen et al [30] predict difficulty in a match-3 game based on metrics extracted from the performance of RL game-playing agents. Importantly for us, they hypothesize that performance metrics derived from an AI agent's best attempts in playing a game level, assessed over multiple runs, can better predict the perceived difficulty of human players than metrics based on average agent performance.…”

Characterising Players of a Cube Puzzle Game with a Two-level Bag of Words

“…Sequential models of in-game player behaviours are an alternative to aggregated players' actions, for predicting personality and expertise [6], assistance in serious games [36], churn prediction [17,38], or player categorisation from past and predicted behaviours [8]. Deep and reinforcement learning may help predict completion rate [18], or excessive gaming [34].…”