Skilloscopy: Bayesian Modeling of Decision Makers' Skill

Abstract

This paper proposes and demonstrates an approach, Skilloscopy, to the assessment of decision makers. In an increasingly sophisticated, connected and information-rich world, decision making is becoming both more important and more difficult. At the same time, modelling decision-making on computers is becoming more feasible and of interest, partly because the information-input to those decisions is increasingly on record. The aims of Skilloscopy are to rate and rank decision makers in a domain relative to each other: the aims do not include an analysis of why a decision is wrong or suboptimal, nor the modelling of the underlying cognitive process of making the decisions. In the proposed method a decision-maker is characterised by a probability distribution of their competence in choosing among quantifiable alternatives. This probability distribution is derived by classic Bayesian inference from a combination of prior belief and the evidence of the decisions. Thus, decision-makers' skills may be better compared, rated and ranked.The proposed method is applied and evaluated in the gamedomain of Chess. A large set of games by players across a broad range of the World Chess Federation (FIDE) Elo ratings has been used to infer the distribution of players' rating directly from the moves they play rather than from game outcomes.Demonstration applications address questions frequently asked by the Chess community regarding the stability of the Elo rating scale, the comparison of players of different eras and/or leagues, and controversial incidents possibly involving fraud.The method of Skilloscopy may be applied in any decision domain where the value of the decision-options can be quantified.

How Much Does Users' Psychology Matter in Engineering Mean-Field-Type Games

“…Thousands of theoretical papers have been published about the prisoner's dilemma game and more than 30 experiments about have been conducted in the literature, only few of them are dedicated to the emergence of cooperative behaviors in one-shot games [37]. The works in [38,39,40] consider decision-making problems. However, the effect of users' psychology on its decision is not examined in these previous works.…”

A Comparative Review of Skill Assessment: Performance, Prediction and Profiling

“…An important advantage is that his simple formula can be used iteratively as each new observation arrives. Skilloscopy is the name given to the assessment of skill by Bayesian Inference [2][3][4][5][6][7][8][9]. It proceeds from initial inherited or presumed probabilities p i that AP 'is' BP(c i ): AP's initial presumed apparent competence ac is therefore  i p i .…”

“…The Reference ELO Player RP e , a Reference Player with ELO e may be defined by analyzing the moves of a set of players with ELO e, e.g., [2690,2710]. This was done [7][8][9], in fact restricting chosen games to those between two such players. 1 The players' ratings in MM/EV, AD/ASD and SR/SK terms may be used to calibrate their respective scales.…”

“…Bayesian inference as used by Haworth et al [2][3][4][5][6][7][8][9] is the natural vehicle for this as it combines an expression of prior belief with the modification of that belief in the light of subsequent evidence. Players may also be profiled from a 'cold start' if 'know nothing' priors are adopted.…”