"I Want To See How Smart This AI Really Is": Player Mental Model Development of an Adversarial AI Player

Abstract

Understanding players' mental models are crucial for game designers who wish to successfully integrate player-AI interactions into their game. However, game designers face the difficult challenge of anticipating how players model these AI agents during gameplay and how they may change their mental models with experience. In this work, we conduct a qualitative study to examine how a pair of players develop mental models of an adversarial AI player during gameplay in the multiplayer drawing game iNNk. We conducted ten gameplay sessions in which two players (n = 20, 10 pairs) worked together to defeat an AI player. As a result of our analysis, we uncovered two dominant dimensions that describe players' mental model development (i.e., focus and style). The first dimension describes the focus of development which refers to what players pay attention to for the development of their mental model (i.e., top-down vs. bottom-up focus). The second dimension describes the differences in the style of development, which refers to how players integrate new information into their mental model (i.e., systematic vs. reactive style). In our preliminary framework, we further note how players process a change when a discrepancy occurs, which we observed occur through comparisons (i.e., compare to other systems, compare to gameplay, compare to self). We offer these results as a preliminary framework for player mental model development to help game designers anticipate how different players may model adversarial AI players during gameplay.

Thought Bubbles: A Proxy into Players' Mental Model Development

“…For example, researchers can investigate the use of thought bubbles for diegetic elicitation through non-verbal queries such as card sorting [63], or diagrammatic representations [77]. In addition, recent advances in AI have inspired many researchers to study human-AI interaction [1,86] and investigate mental models of AI [40,92]. Thought bubbles can be utilized to advance our understanding of users' mental model development of AI over time.…”

Machine Learning Processes As Sources of Ambiguity: Insights from AI Art

“…For instance, previous ethnographic studies in HCI have employed conversation analysis to reveal how errors of virtual assistants such as Alexa become a source of humor when they "fail" during family dinner time conversations [72]. Game designers have engaged human players to detect ML errors as a form of playful experience [89,98]. More broadly, researchers have found that exposing ML errors to users is helpful for them to develop accurate mental models of the system [38,89].…”

“…Game designers have engaged human players to detect ML errors as a form of playful experience [89,98]. More broadly, researchers have found that exposing ML errors to users is helpful for them to develop accurate mental models of the system [38,89]. Therefore, designers should use these AI artworks as a starting point to broaden the design space of how ML errors are exposed to users and how the ML system can recover from them.…”