Bayesian Execution Skill Estimation

Abstract

The performance of agents in many domains with continuous action spaces depends not only on their ability to select good actions to execute, but also on their ability to execute planned actions precisely. This ability, which has been called an agent’s execution skill, is an important characteristic of an agent which can have a significant impact on their success. In this paper, we address the problem of estimating the execution skill of an agent given observations of that agent acting in a domain. Each observation includes the executed action and a description of the state in which the action was executed and the reward received, but notably excludes the action that the agent intended to execute. We previously introduced this problem and demonstrated that estimating an agent’s execution skill is possible under certain conditions. Our previous method focused entirely on the reward that the agent received from executed actions and assumed that the agent was able to select the optimal action for each state. This paper addresses the execution skill estimation problem from an entirely different perspective, focusing instead on the action that was executed. We present a Bayesian framework for reasoning about action observations and show that it is able to outperform previous methods under the same conditions. We also show that the flexibility of this framework allows it to be applied in settings where the previous limiting assumptions are not met. The success of the proposed method is demonstrated experimentally in a toy domain as well as the domain of computational billiards.

Estimating Agent Skill in Continuous Action Domains

“…When the agent is perfectly rational, we are able to more easily predict the actions that they will take, and thus AXE works very well with the correct β value. This confirms findings from previous work on the effectiveness of the AXE method (Archibald & Nieves-Rivera, 2019). When agents are rational, each of the methods can be used to estimate execution skill, given enough observations.…”

“…AXE can work well in settings where a set of focal actions can be generated for each state, and when agents are behaving with nearly perfect rationality. It has been demonstrated to work well for estimating the execution skill of a top computational billiards agent (Archibald & Nieves-Rivera, 2019;Archibald, Altman, & Shoham, 2009), and has also been used for estimating the execution skill of professional baseball pitchers (Melville et al, 2023), which will be explored in more detail in Section 10. However, the applicability of AXE is somewhat limited by the fact that it requires two things to be specified: the set of focal actions and the β value.…”

A Survey of Opponent Modeling in Adversarial Domains

“…Much like many basketball and soccer teams are content to let the opposing team shoot from long range since the probability of scoring is low, we would like our opponent modeling systems to reason similarly not only about what the opponent plans to do, but also about their probability of success should they choose a given action. The problem of skill estimation (Archibald & Nieves-Rivera, 2018) is closely related, and Bayesian techniques have been proposed for simulated, real-valued games including darts and billiards (Archibald & Nieves-Rivera, 2019), but this is still an under-explored area of research and has yet to be integrated in opponent modeling systems at any notable scale.…”