Offline Learning of Counterfactual Predictions for Real-World Robotic Reinforcement Learning

Abstract

Autonomous excavation is a challenging task. The unknown contact dynamics between the excavator bucket and the terrain could easily result in large contact forces and jamming problems during excavation. Traditional model-based methods struggle to handle such problems due to complex dynamic modeling. In this paper, we formulate the excavation skills with three novel manipulation primitives. We propose to learn the manipulation primitives with offline reinforcement learning (RL) to avoid large amounts of online robot interactions. The proposed method can learn efficient penetration skills from sub-optimal demonstrations, which contain subtrajectories that can be "stitched" together to formulate an optimal trajectory without causing jamming. We evaluate the proposed method with extensive experiments on excavating a variety of rigid objects and demonstrate that the learned policy outperforms the demonstrations. We also show that the learned policy can quickly adapt to unseen and challenging fragmented rocks with online fine-tuning.

GVFs in the real world: making predictions online for water treatment

“…The advantage of this approach is that the control-rules are easy to explain to human operators, but since control is triggered by predictions that are continually updated in deployment the resultant controller adapts to changing conditions. An extension of this idea is to use GVF predictions-like the ones we learned in this work-as input to a neural-network based RL agent, similarly to how it was done for autonomous driving (Graves et al, 2020;Jin et al, 2022). This work provides the foundations for these next steps in industrial control with RL.…”

Counterfactual learning in enhancing resilience in autonomous agent systems

“…Counterfactuals can be used to analyse the current situation and envisage the different outcomes by changing the antecedents to add to the knowledge. Counterfactual predictions are used by Jin et al (2022) to guide the exploration of a reinforcement learner for robotic manipulation tasks. The active counterfactual predictions generated add to the existing body of knowledge about the task for the learning model to improve its performance.…”

A Twin Agent Reinforcement Learning Framework by Integrating Deterministic and Stochastic Policies

“…At this juncture, it is exciting to look for the integration of artificial intelligence in process control as it can bring in a paradigm shift by enabling more intelligent, adaptive, and efficient control of industrial processes, ultimately contributing to increased productivity and sustainability . As such, a great deal of research is being done on model-free controlling techniques, particularly on model-free reinforcement learning (RL) algorithms. − In essence, a trained RL has the capabilities of generating a control policy by optimizing the cumulative reward signal over successive interactions with an environment until a desired goal is achieved. − …”