The Evaluation of Rating Systems in Online Free-for-All Games

Dehpanah, Arman; Ghori, Muheeb Faizan; Gemmell, Jonathan; Mobasher, Bamshad

doi:10.48550/arxiv.2008.06787

Cited by 3 publications

(8 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Battle royale games consist of many teams and players, and the result of a match is often an ordered list of ranks. Therefore, we used NDCG to evaluate predictions [3,4] for our battle royale dataset; PUBG.…”

Section: Discussionmentioning

confidence: 99%

Evaluating Team Skill Aggregation in Online Competitive Games

Dehpanah¹,

Ghori²,

Gemmell³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

One of the main goals of online competitive games is increasing player engagement by ensuring fair matches. These games use rating systems for creating balanced match-ups. Rating systems leverage statistical estimation to rate players' skills and use skill ratings to predict rank before matching players. Skill ratings of individual players can be aggregated to compute the skill level of a team. While research often aims to improve the accuracy of skill estimation and fairness of match-ups, less attention has been given to how the skill level of a team is calculated from the skill level of its members. In this paper, we propose two new aggregation methods and compare them with a standard approach extensively used in the research literature. We present an exhaustive analysis of the impact of these methods on the predictive performance of rating systems. We perform our experiments using three popular rating systems, Elo, Glicko, and TrueSkill, on three real-world datasets including over 100,000 battle royale and head-to-head matches. Our evaluations show the superiority of the MAX method over the other two methods in the majority of the tested cases, implying that the overall performance of a team is best determined by the performance of its most skilled member. The results of this study highlight the necessity of devising more elaborated methods for calculating a team's performance-methods covering different aspects of players' behavior such as skills, strategy, or goals.

show abstract

Section: Discussionmentioning

confidence: 99%

Evaluating Team Skill Aggregation in Online Competitive Games

Dehpanah¹,

Ghori²,

Gemmell³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Rank correlation coefficients and their derivatives [4,[17][18][19][20] are among the metrics commonly used for this purpose. A recent study compared several metrics based on how they explain the predictive power of rating systems [21]. This study focused on solo battle royale matches where there are no teams, and players compete against each other as singletons.…”

Section: Related Workmentioning

confidence: 99%

“…Elo was originally designed for head-to-head matches between two players. It was later extended to solo battle royale games where many players compete against one another as singletons at the same time in the same match [21].…”

Section: A Elomentioning

confidence: 99%

“…Similar to Elo, Glicko was originally designed for head-tohead matches between two players and was later extended to solo battle royale matches [21]. We extend Glicko to teambased battle royale games as we did with Elo.…”

Section: B Glickomentioning

confidence: 99%

“…Mean reciprocal rank (MRR) is a common metric in the field of information retrieval for evaluating the performance of a query-response system that returns a ranked list of responses for a query. To use MRR for evaluating the performance of rating systems in team battle royale matches, we utilize an extended version of MRR introduced in [21]. In a battle royale match with N teams, MRR can be calculated as:…”

Section: Mean Reciprocal Rankmentioning

confidence: 99%

See 2 more Smart Citations

The Evaluation of Rating Systems in Team-based Battle Royale Games

Dehpanah,

Ghori,

Gemmell

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Online competitive games have become a mainstream entertainment platform. To create a fair and exciting experience, these games use rating systems to match players with similar skills. While there has been an increasing amount of research on improving the performance of these systems, less attention has been paid to how their performance is evaluated. In this paper, we explore the utility of several metrics for evaluating three popular rating systems on a real-world dataset of over 25,000 team battle royale matches. Our results suggest considerable differences in their evaluation patterns. Some metrics were highly impacted by the inclusion of new players. Many could not capture the real differences between certain groups of players. Among all metrics studied, normalized discounted cumulative gain (NDCG) demonstrated more reliable performance and more flexibility. It alleviated most of the challenges faced by the other metrics while adding the freedom to adjust the focus of the evaluations on different groups of players.

show abstract

Behavioral Player Rating in Competitive Online Shooter Games

Dehpanah¹,

Ghori²,

Gemmell³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Competitive online games use rating systems for matchmaking; progression-based algorithms that estimate the skill level of players with interpretable ratings in terms of the outcome of the games they played. However, the overall experience of players is shaped by factors beyond the sole outcome of their games. In this paper, we engineer several features from in-game statistics to model players and create ratings that accurately represent their behavior and true performance level. We then compare the estimating power of our behavioral ratings against ratings created with three mainstream rating systems by predicting rank of players in four popular game modes from the competitive shooter genre. Our results show that the behavioral ratings present more accurate performance estimations while maintaining the interpretability of the created representations. Considering different aspects of the playing behavior of players and using behavioral ratings for matchmaking can lead to matchups that are more aligned with players' goals and interests, consequently resulting in a more enjoyable gaming experience.

show abstract

The Evaluation of Rating Systems in Online Free-for-All Games

Cited by 3 publications

References 0 publications

Evaluating Team Skill Aggregation in Online Competitive Games

Evaluating Team Skill Aggregation in Online Competitive Games

The Evaluation of Rating Systems in Team-based Battle Royale Games

Behavioral Player Rating in Competitive Online Shooter Games

Contact Info

Product

Resources

About