Towards life-long adaptive agents: using metareasoning for combining knowledge-based planning with situated learning

Abstract

We consider task planning for long-living intelligent agents situated in dynamic environments. Specifically, we address the problem of incomplete knowledge of the world due to the addition of new objects with unknown action models. We propose a multilayered agent architecture that uses meta-reasoning to control hierarchical task planning and situated learning, monitor expectations generated by a plan against world observations, forms goals and rewards for the situated reinforcement learner, and learns the missing planning knowledge relevant to the new objects. We use occupancy grids as a low-level representation for the high-level expectations to capture changes in the physical world due to the additional objects, and provide a similarity method for detecting discrepancies between the expectations and the observations at run-time; the meta-reasoner uses these discrepancies to formulate goals and rewards for the learner, and the learned policies are added to the hierarchical task network plan library for future re-use. We describe our experiments in the Minecraft and Gazebo microworlds to demonstrate the efficacy of the architecture and the technique for learning. We test our approach against an ablated reinforcement learning (RL) version, and our results indicate this form of expectation enhances the learning curve for RL while being more generic than propositional representations.

A Cognitive Reformation

“…This too is a difficult problem because of the need to specify representations and mechanisms at the various levels as well as the distribution of control of processing among the levels. Parashar et al (2018) describe how a virtual robot uses a metacognitive architecture to detect, explain, and address discrepancies between the expected states and the observed states. The metacognitive level in their architecture uses dual encoding: symbolic representations so that the metacognitive mechanisms can reason over them, and visual encoding so that the robot can detect discrepancies in the world, thus grounding the internal symbols in the external world.…”

“…In order to make the robots work, Fitzgerald et al () and Parashar et al () developed representations and processes that offer hypotheses for cognitive science to explore, and cognitive science likely may offer useful inputs and critiques to them. I expect that there are many examples—dozens, scores, perhaps hundreds of examples—not only from AI but also from anthropology and philosophy that could benefit cognitive science as well as benefit from it.…”

Educational Robotics and Robot Creativity: An Interdisciplinary Dialogue

“…Whereas they are effective when dealing with cases that bear resemblance to the task that has already been experienced by the robot, CBR systems have limited efficiency when they encounter novel problems. Parashar et al (2018) have introduced an architecture enabling an agent to cope with novelty. The work addresses the issue raised by Sarathy and Scheutz (2018) , Konidaris et al (2018) and combines planning and reinforcement learning approaches.…”

“…The work addresses the issue raised by Sarathy and Scheutz (2018) , Konidaris et al (2018) and combines planning and reinforcement learning approaches. This combination of top-down and bottom-up approaches makes the work of Parashar et al (2018) especially relevant for the context of creative problem solving in robotics. The authors proposed a three-layered agent architecture, with 1) object-level reasoning acts based on the information encoded from the environment; 2) deliberative reasoning, responsible for plan construction and action based on object-level information, and 3) a meta-reasoning layer responsible for problem construction and re-construction based on object-level and deliberative-level information and learning history.…”

Special issue on adaptive and learning agents 2017

“…The second paper Towards life-long adaptive agents: a hybrid planning paradigm for combining domain knowledge with reinforcement learning by Parashar et al (2018) considers the problem of task planning for long-living agents situated in dynamic environments. The authors propose a multi-layered agent architecture which uses meta-reasoning to control hierarchical task planning and situated learning, while monitoring expectations generated by a plan against world observations and forming goals and rewards for the reinforcement learner.…”