A Feel for the Game: AI, Computer Games and Perceiving Perception

Abstract

I walk into the room and the smell of burning wood hits me immediately. The warmth from the fireplace grows as I step nearer to it. The fire needs to heat the little cottage through night so I add a log to the fire. There are a few sparks and embers. I throw a bigger log onto the fire and it drops with a thud. Again, there are barely any sparks or embers. The heat and the smell stay the same. They don’t change and I do not become habituated to it. Rather, they are just a steady stream, so I take off my VR headset and give my recommendations to the team programming the gameified world of the virtual museum of the future (one depicting an ancient Turkish settlement, being built now at the institution where one of us works). As much as this technological world seems almost too futuristic, it actually retrieves obsolete items from the past—a heater, a piece of wood, and a spray bottle—in keeping with McLuhan’s (1973) insights regarding media that provide strong participation goals and the rubric for achieving them. Moreover, the VR world extends the progression of game AI that occasioned the love-hate relationship with the “walking sim.” The stronger the AI, the more clearly defined the rubric for participation. In the VR interactive museum the designers want people to be able to “play” with haptic devices—like the smell, smoke, and heat generators—in order to heighten not only the immersion but also the perception of being there, or what Bolter and Grusin (1999) call “immediacy.” Indeed, Bolter and Grusin argue that the need for immediacy overwhelmingly takes over, regardless of the media’s intrusion. However, in the example above, the system fell short because the designers had not figured for someone laying down the “log” on the virtual fire and having it send a representative—that is, a perceptual, based on experience, intuition, etc.—amount of sparks and heat. Someone else could throw the log as hard as they want. The machine only senses log in or log out. This corresponds precisely with how we feel about phenomena, for machines and AI are based upon a model of intelligence which prioritises mental representation and symbolic manipulation. For Laird and van Lent (2001), in their field defining presentation, the “killer app” of human-level AI was going to be computer games. Writing a decade later in the same conference proceedings, Weber, Mateas, and Jhala (2011), are still responding to this original position, by way of AI in strategy games. Writing for this year’s, IEEE meeting Petrović (2018) also makes the case for human-level AI in games. What becomes clear, then, is that as much as we have wanted games to offer human behaviours, perception has taken a backseat in the extant models.As phenomenology makes clear, the emphasis on behaviour over perception leaves out the crucial, indeed foundational mode of intelligence: affective intentionality. Simply put, how we feel about phenomena impacts how we perceive phenomena as significant, inconsequential, interesting, etc. Thus, we should be asking if machines can understand significance? Can they feel any particular way about a game, a move, or the phenomenon of play? This becomes important when mental representation provides the mode of symbolic manipulation and vice versa. This occurs in and through a given game or gameified world’s ability to instill, simulate, or otherwise produce affective intentionality. We would argue that herein lies the crux of the mixed reactions to Red Dead Redemption 2. Similarly, the example from the virtual museum highlights the ongoing omission. Human-level AI should not just reproduce a human’s response to inputs, but should produce responses that a human would perceive. In short human-level AI needs to perceive perception itself. Indeed, this is the primary cognitive and affective response. Phenomenology does not tell us that; first principles semiotics tells us that. However, phenomenology gives us the means and methods to understand the response to affective intentionality and, more importantly, to develop the contingent hermeneutic (Merleau-Ponty, 2013). Moreover, semiotics will never encompass the materiality required of such a system, let alone the simulated materiality that exists through the interaction with the AI device and its interface, a device that bears the mark of the maker, just as surely as a bespoke shirt does. Thus, our paper will consider the production of affective intentionality and the ways VR games and gameified systems, like the virtual museum and Red Dead Redemption 2, facilitate, impede, and especially teach the perception of perception. As a corollary, then, our paper necessarily considers meta-cognitive processes—that is, the strategies for learning about learning—that occur in and through interaction with AI in games and devices (cf Hacker, 1998, 2016). Indeed, meta-cognition becomes a contingent component for instilling affective intentionality.

Twitch, Fish, Pokémon and Plumbers: Game live streaming by nonhuman actors

“…Fizek (2018:208) similarly suggests that ‘human players are driven, immersed, or frustrated. AI and machine just are’, and in the same vein Ouellette and Conway (Ouellette and Conway, 2020:11) point towards the importance of affective intentionality in consideration of agency, and that ‘how we feel about phenomena impacts how we perceive phenomena as significant, inconsequential, interesting, etc’. In both cases, these authors ask whether these nonhuman agents can ‘feel any particular way’ about their play, in this case in the way that (presumably) human streamers and spectators do.…”

“…This is not to say that nonhumans don’t ‘see the results of [their] decisions and choices’ (Murray, 2016:159); they do. However, nonhuman – in particular, technological – perception differs vastly from human perception (Ouellette and Conway 2020) and so this actor-consequence connection varies. Considering ‘Lucky Draw’, the game ‘sees’ the result of each algorithmically chosen transformed block and responds accordingly, ultimately either triggering the P switch or not, although there is no deliberate link between these individual choices and the successful completion of the level.…”

“…Johnson, 2018). One of the challenges facing game design is the gulf that persists between programmed games and human perception (Petrovic 2018; Ouellete and Conway 2020). Like a livestream spectator who watches a streamer play their favourite game to see what they do differently, here spectators gather to witness – or participate in – a different approach to play in the hopes of evaluating its legitimacy through eventual success or failure.…”

“…Her analysis presents us with frameworks and examples for thinking about how to move beyond humans as the ‘sole meaningful agents’ in games (Ibid:206), and we will draw upon this work several times throughout our paper as we extend such an understanding into the realm of live streaming. We additionally note Ouellette & Conway’s paper in which they question whether ‘the phenomenon of play [is] even an ontological possibility’ for a computer (Ouellette and Conway, 2020:11), something we will address in considering the difference between an AI actively playing a game, and an algorithmic system incidentally enabling play to occur. Yannakakis & Togelius’s work (Yannakakis and Togelius, 2018) identifies limits in what games AIs are able to play, particularly when it comes to developing and implementing strategy in strategy games, which they link with intention (2018:135), a topic we will return to in this paper.…”