Nash equilibria in human sensorimotor interactions explained by Q-learning with intrinsic costs

Lindig-León, Cecilia; Schmid, Gerrit; Braun, Daniel A.

doi:10.1038/s41598-021-99428-0

Cited by 9 publications

(8 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Here a question arises of whether learning the action(s)-value and predicting the partner actions occur independently, or are actually part of one single process. In a recent study focusing on sensorimotor versions of classical discrete games, Lindig-León et al (2021) observed that convergence to a Nash equilibrium is consistent with a model-free form of reinforcement learning, in which actions are generated as a trade-off between their value and the requirement of minimizing their change with respect to the previous trial. This learning mechanism does not explicitly account for partner actions, but it is unclear if it would extend to more complex forms of coordination that involve more than just discrete decisions.…”

Section: Learning In Joint Actionmentioning

confidence: 69%

Game theory and partner representation in joint action: toward a computational theory of joint agency

2022

View full text Add to dashboard Cite

The sense of agency – the subjective feeling of being in control of our own actions – is one central aspect of the phenomenology of action. Computational models provided important contributions toward unveiling the mechanisms underlying the sense of agency in individual action. In particular, the sense of agency is believed to be related to the match between the actual and predicted consequences of our own actions (comparator model). In the study of joint action, models are even more necessary to understand the mechanisms underlying the development of coordination strategies and how the subjective experiences of control emerge during the interaction. In a joint action, we not only need to predict the consequences of our own actions; we also need to predict the actions and intentions of our partner, and to integrate these predictions to infer their joint consequences. Understanding our partner and developing mutually satisfactory coordination strategies are key components of joint action and in the development of the sense of joint agency. Here we discuss a computational architecture which addresses the sense of agency during intentional, real-time joint action. We first reformulate previous accounts of the sense of agency in probabilistic terms, as the combination of prior beliefs about the action goals and constraints, and the likelihood of the predicted movement outcomes. To look at the sense of joint agency, we extend classical computational motor control concepts - optimal estimation and optimal control. Regarding estimation, we argue that in joint action the players not only need to predict the consequences of their own actions, but also need to predict partner’s actions and intentions (a ‘partner model’) and to integrate these predictions to infer their joint consequences. As regards action selection, we use differential game theory – in which actions develop in continuous space and time - to formulate the problem of establishing a stable form of coordination and as a natural extension of optimal control to joint action. The resulting model posits two concurrent observer-controller loops, accounting for ‘joint’ and ‘self’ action control. The two observers quantify the likelihoods of being in control alone or jointly. Combined with prior beliefs, they provide weighing signals which are used to modulate the ‘joint’ and ‘self’ motor commands. We argue that these signals can be interpreted as the subjective sense of joint and self agency. We demonstrate the model predictions by simulating a sensorimotor interactive task where two players are mechanically coupled and are instructed to perform planar movements to reach a shared final target by crossing two differently located intermediate targets. In particular, we explore the relation between self and joint agency and the information available to each player about their partner. The proposed model provides a coherent picture of the inter-relation of prediction, control, and the sense of agency in a broader range of joint actions.

show abstract

Section: Learning In Joint Actionmentioning

confidence: 69%

Game theory and partner representation in joint action: toward a computational theory of joint agency

2022

View full text Add to dashboard Cite

show abstract

“…In previous studies, it was found that such haptic couplings between two different players in the Prisoner's Dilemma are compatible with the Nash solution, as most interaction endpoints laid in the same quadrant of the workspace than the Nash equilibrium [31]. Similar analyses have also advocated the adequacy of the Nash solution concept for describing sensorimotor interactions in more general scenarios, including mixed equilibrium games like matching pennies [57], coordination games with multiple Nash equilibria like the battle of sexes, chicken or stag hunt [32] as well as Bayesian games that require sensorimotor communication [34]. Importantly, none of the above studies could distinguish the Nash solution from the quantal response equilibrium, as the two solution concepts are often very close together and perfectly coincide in the absence of computational or precision limits.…”

Section: Discussionmentioning

confidence: 80%

“…The corresponding shifts for the response frequencies of player 1 reproduce the same pattern as observed in the human players. This suggests that reinforcement learning models based on Q-learning cannot only explain convergence to Nash equilibrium solutions [57], but more generally convergence to quantal response equilibria.…”

Section: Resultsmentioning

confidence: 99%

Bounded rational response equilibria in human sensorimotor interactions

2021

Self Cite

View full text Add to dashboard Cite

The Nash equilibrium is one of the most central solution concepts to study strategic interactions between multiple players and has recently also been shown to capture sensorimotor interactions between players that are haptically coupled. While previous studies in behavioural economics have shown that systematic deviations from Nash equilibria in economic decision-making can be explained by the more general quantal response equilibria, such deviations have not been reported for the sensorimotor domain. Here we investigate haptically coupled dyads across three different sensorimotor games corresponding to the classic symmetric and asymmetric Prisoner's Dilemma, where the quantal response equilibrium predicts characteristic shifts across the three games, although the Nash equilibrium stays the same. We find that subjects exhibit the predicted deviations from the Nash solution. Furthermore, we show that taking into account subjects' priors for the games, we arrive at a more accurate description of bounded rational response equilibria that can be regarded as a quantal response equilibrium with non-uniform prior. Our results suggest that bounded rational response equilibria provide a general tool to explain sensorimotor interactions that include the Nash equilibrium as a special case in the absence of information processing limitations.

show abstract

“…However, the vast majority of previous studies addressing joint action within a game theoretic framework only focus on equilibrium situations [4,17]. Very few studies [6,19,20] have addressed the way joint coordination is negotiated and learned in scenarios that involve movements. One simple learning strategy when the players play repeatedly the game is that at every round each player determines their best response based on their beliefs about how their opponents will play (fictitious play, FP)see [21,22].…”

Section: Introductionmentioning

confidence: 99%

Computational joint action: dynamical models to understand the development of joint coordination

De Vicariis,

Chackochan,

Bandini

et al. 2024

Preprint

View full text Add to dashboard Cite

Coordinating with others is part of our everyday experience. Previous studies using sensorimotor coordination games suggest that human dyads develop coordination strategies that can be interpreted as Nash equilibria. However, if the players are uncertain about what their partner is doing, they develop coordination strategies which are robust to the actual partner’s actions. This has suggested that humans select their actions based on an explicit prediction of what the partner will be doing – a partner model – which is probabilistic by nature. However, the mechanisms underlying the development of a joint coordination over repeated trials remain unknown. Very much like sensorimotor adaptation of individuals to external perturbations (eg force fields or visual rotations), dynamical models may help to understand how joint coordination develops over repeated trials.Here we present a general computational model – based on game theory and Bayesian estimation – designed to understand the mechanisms underlying the development of a joint coordination over repeated trials. Joint tasks are modeled as quadratic games, where each participant’s task is expressed as a quadratic cost function. Each participant predicts their partner’s next move (partner model) by optimally combining predictions and sensory observations, and selects their actions through a stochastic optimization of its expected cost, given the partner model. The model parameters include perceptual uncertainty (sensory noise), partner representation (retention rate and process noise), uncertainty in action selection and its rate of decay (which can be interpreted as the action’s learning rate). The model can be used in two ways: (i) to simulate interactive behaviors, thus helping to make specific predictions in the context of a given joint action scenario; and (ii) to analyze the action time series in actual experiments, thus providing quantitative metrics that describe individual behaviors during an actual joint action.We demonstrate the model in a variety of joint action scenarios. In a sensorimotor version of the Stag Hunt game, the model predicts that different representations of the partner lead to different Nash equilibria. In a joint two via-point (2-VP) reaching task, in which the actions consist of complex trajectories, the model captures well the observed temporal evolution of performance. For this task we also estimated the model parameters from experimental observations, which provided a comprehensive characterization of individual dyad participants.Computational models of joint action may help identifying the factors preventing or facilitating the development of coordination. They can be used in clinical settings, to interpret the observed behaviors in individuals with impaired interaction capabilities. They may also provide a theoretical basis to devise artificial agents that establish forms of coordination that facilitate neuromotor recovery.Author summaryActing together (joint action) is part of everyday experience. But, how do we learn to coordinate with others and collaborate? Using a combination of experiments and computational models we show that through multiple repetitions of the same joint task we select the action which represents the ‘best response’ to what we believe our opponent will do. Such a belief about our partner (partner model) is developed gradually, by optimally combining prior assumptions (how repeatable or how erratic our opponent behaves) with sensory information about our opponent’s past actions. Rooted in game theory and Bayesian estimation, the model accounts for the development of the mutual ‘trust’ among partners which is essential for establishing a mutually advantageous collaboration, and explains how we combine decisions and movements in complex coordination scenarios. The model can be used as a generative tool, to simulate the development of coordination in a specific joint action scenario, and as an analytic tool to characterize the individual traits or defects in the ability to establish collaborations.

show abstract

Nash equilibria in human sensorimotor interactions explained by Q-learning with intrinsic costs

Cited by 9 publications

References 61 publications

Game theory and partner representation in joint action: toward a computational theory of joint agency

Game theory and partner representation in joint action: toward a computational theory of joint agency

Bounded rational response equilibria in human sensorimotor interactions

Computational joint action: dynamical models to understand the development of joint coordination

Contact Info

Product

Resources

About