A New AI Evaluation Cosmos: Ready to Play the Game?

Abstract

Hernández-Orallo, J.; Baroni, M.; Bieger, J.; Chmait, N.; Dowe, DL.; Hofmann, K.; Martínez-Plumed, F.... (2017)

A Holistic Framework for Forecasting Transformative AI

“…In contrast to these studies, a measure of intelligence has been proposed by Riedl that is based on creativity [95]. The topic has also garnered mainstream attention with a workshop being dedicated to it in 2015 [96], and work on it being featured in Nature in 2016 [97]. Although there is no consensus on how to measure AI progress or intelligence, it is clear that simple measures which can be represented in a small number of dimensions are elusive.…”

Hope, exploration, and equilibrated action schemes

“…Thus, as the reward pattern becomes closer to (uniformly) random, learning seems of less importance/relevance to the agent's score. Nevertheless, further controlled experimentation is required to assess the above premise, across different food-density distributions (under 50%), time periods (long-term behaviour) and types of foraging agents (among other factors) (Chmait et al 2016a;Hernández-Orallo et al 2017). Moreover, in multiagent foraging, the trade-off can involve balancing the allocation of agents to exploration and exploitation.…”

Navigating the landscape of multiplayer games

“…Overall, AI research has primarily placed emphasis on training of strong agents; we refer to this as the Policy Problem, which entails the search for super human-level AI performance. Despite this progress, the need for a task theory, a framework for taxonomizing, characterizing, and decomposing AI tasks has become increasingly important in recent years 11 , 12 . Naturally, techniques for understanding the space of games are likely beneficial for the algorithmic development of future AI entities 12 , 13 .…”

“…A core challenge associated with designing such a task theory has been recently coined the Problem Problem, defined as "the engineering problem of generating large numbers of interesting adaptive environments to support research" 15 . Research associated with the Problem Problem has a rich history spanning over 30 years, including the aforementioned work on task theory 11,12,16 , procedurally-generated videogame features [17][18][19] , generation of games and rule-sets for General Game Playing [20][21][22][23][24][25][26] , and procedural content generation techniques [27][28][29][30][31][32][33][34][35][36] ; we refer readers to Supplementary Note 1 for detailed discussion of these and related works. An important question that underlies several of these interlinked fields is: what makes a game interesting enough for an AI agent to learn to play?…”