Coping with the variability in humans reward during simulated human-robot interactions through the coordination of multiple learning strategies

Dromnelle, Rémi; Girard, Benoît; Renaudo, Erwan; Chatila, Raja; Khamassi, Mehdi

doi:10.1109/ro-man47096.2020.9223451

Cited by 6 publications

(6 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Dromnelle et al [2020a] we studied the evolution of the average performance of the different robots when the human does not interact with them. As in the navigation experiments, the MF-only robot was the one with the worst performance.…”

Section: Results Of the Experiments Without Human Interventionmentioning

confidence: 99%

“…In the second part, we present the results obtained and show how our coordination system allows the robot, in a task with more states, and without major change in our architecture, to maintain again a high level of performance while decreasing greatly its computational cost, but also to deal with the volatility of human behavior. The work presented in this section is an extended version of the publication Dromnelle et al [2020a], to which we will refer when mentioning previously published results. The pdf of the publication can be accessed from: https://hal.archives-ouvertes.fr/hal-02899767v2/document.…”

Section: Spatial Pattern Of Expert Selectionmentioning

confidence: 99%

“…Part of the results in the navigation scenario (Experiment 1), those with change in reward location, but not those with change in the wall configuration, have been published in a conference paper [Dromnelle et al, 2020b]. Part of the results in the HRI scenario (Experiment 2) have been published in a second conference paper [Dromnelle et al, 2020a]. We present new unpublished results in both experiments, new extended analyses of the properties of the robotic architecture which explain these results, and a thorougher description of the methods.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Reducing Computational Cost During Robot Navigation and Human–Robot Interaction with a Human-Inspired Reinforcement Learning Architecture

Dromnelle

Renaudo

Chétouani

et al. 2022

Int J of Soc Robotics

Self Cite

View full text Add to dashboard Cite

We present a new neuro-inspired reinforcement learning architecture for robot online learning and decision-making during both social and non-social scenarios. The goal is to take inspiration from the way humans dynamically and autonomously adapt their behavior according to variations in their own performance while minimizing cognitive effort. Following computational neuroscience principles, the architecture combines model-based (MB) and model-free (MF) reinforcement learning (RL). The main novelty here consists in arbitrating with a meta-controller which selects the current learning strategy according to a trade-off between efficiency and computational cost. The MB strategy, which builds a model of the long-term effects of actions and uses this model to decide through dynamic programming, enables flexible adaptation to task changes at the expense of high computation costs.The MF strategy is less flexible but also 1000 times less costly, and learns by observation of MB decisions. We test the architecture in three experiments: a navigation task in a real environment with task changes (wall configuration changes, goal location changes); a simulated object manipulation task under human teaching signals; and a simulated human-robot cooperation task to tidy up objects on a table. We show that our human-inspired strategy coordination method enables the robot to maintain an optimal performance in terms of reward and computational cost compared to an MB expert alone, which achieves the best performance but has the highest computational cost. We also show that the method makes it possible to cope with sudden changes in the environment, goal changes or changes in the behavior of the human partner during interaction tasks. The robots that performed these experiments, whether real or virtual, all used the same set of parameters, thus showing the generality of the method.

show abstract

Section: Results Of the Experiments Without Human Interventionmentioning

confidence: 99%

Section: Spatial Pattern Of Expert Selectionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Reducing Computational Cost During Robot Navigation and Human–Robot Interaction with a Human-Inspired Reinforcement Learning Architecture

Dromnelle

Renaudo

Chétouani

et al. 2022

Int J of Soc Robotics

Self Cite

View full text Add to dashboard Cite

show abstract

“…The robot control architecture proposed in Dromnelle et al (2020b) and also successfully applied to a simulated human-robot interaction task in Dromnelle et al (2020a) takes inspiration from the mammalian brain's ability to coordinate multiple neural learning systems. Such ability is indeed considered to be key to making animals able to show flexible behavior in a variety of situations, to adapt to changes in the environment, while at the same time minimizing computational cost and physical energy (Renaudo et al, 2014).…”

Section: Methodsmentioning

confidence: 99%

Model-Based and Model-Free Replay Mechanisms for Reinforcement Learning in Neurorobotics

et al. 2022

Self Cite

View full text Add to dashboard Cite

Experience replay is widely used in AI to bootstrap reinforcement learning (RL) by enabling an agent to remember and reuse past experiences. Classical techniques include shuffled-, reversed-ordered- and prioritized-memory buffers, which have different properties and advantages depending on the nature of the data and problem. Interestingly, recent computational neuroscience work has shown that these techniques are relevant to model hippocampal reactivations recorded during rodent navigation. Nevertheless, the brain mechanisms for orchestrating hippocampal replay are still unclear. In this paper, we present recent neurorobotics research aiming to endow a navigating robot with a neuro-inspired RL architecture (including different learning strategies, such as model-based (MB) and model-free (MF), and different replay techniques). We illustrate through a series of numerical simulations how the specificities of robotic experimentation (e.g., autonomous state decomposition by the robot, noisy perception, state transition uncertainty, non-stationarity) can shed new lights on which replay techniques turn out to be more efficient in different situations. Finally, we close the loop by raising new hypotheses for neuroscience from such robotic models of hippocampal replay.

show abstract

“…In the łmachine perspective", the agent represents the system and the user is part of the environment providing some reward, i.e. teaching to the system how to react to users actions [33]. In the łuserž perspective, the agent represents the user, the environment includes the system/interface.…”

Section: Reinforcement Learning and Hcimentioning

confidence: 99%

Computational Model of the Transition from Novice to Expert Interaction Techniques

Bailly¹,

Khamassi²,

Girard³

2023

ACM Trans. Comput.-Hum. Interact.

Self Cite

View full text Add to dashboard Cite

Despite the benefits of expert interaction techniques, many users do not learn them and continue to use novice ones. This article aims at better understanding if, when and how users decide to learn and ultimately adopt expert interaction techniques. This dynamic learning process is a complex skill-acquisition and decision-making problem. We first present and compare three generic benchmark models, inspired by the neuroscience literature, to explain and predict the learning process for shortcut adoption. Results show that they do not account for the complexity of users’ behavior. We then introduce a dedicated model, Transition, combining five cognitive mechanisms: implicit and explicit learning, decay, planning and perseveration. Results show that our model outperforms the three benchmark models both in terms of model fitting and model simulation. Finally, a post-analysis shows that each of the five mechanisms contribute to goodness-of-fit, but the role of perseveration is unclear regarding model simulation.

show abstract

Coping with the variability in humans reward during simulated human-robot interactions through the coordination of multiple learning strategies

Cited by 6 publications

References 12 publications

Reducing Computational Cost During Robot Navigation and Human–Robot Interaction with a Human-Inspired Reinforcement Learning Architecture

Reducing Computational Cost During Robot Navigation and Human–Robot Interaction with a Human-Inspired Reinforcement Learning Architecture

Model-Based and Model-Free Replay Mechanisms for Reinforcement Learning in Neurorobotics

Computational Model of the Transition from Novice to Expert Interaction Techniques

Contact Info

Product

Resources

About