Intelligent problem-solving as integrated hierarchical reinforcement learning

Abstract

According to cognitive psychology and related disciplines, the development of complex problem-solving behaviour in biological agents depends on hierarchical cognitive mechanisms. Hierarchical reinforcement learning is a promising computational approach that may eventually yield comparable problem-solving behaviour in artificial agents and robots. However, to date the problem-solving abilities of many human and non-human animals are clearly superior to those of artificial systems. Here, we propose steps to integrate biologically inspired hierarchical mechanisms to enable advanced problem-solving skills in artificial agents. Therefore, we first review the literature in cognitive psychology to highlight the importance of compositional abstraction and predictive processing. Then we relate the gained insights with contemporary hierarchical reinforcement learning methods. Interestingly, our results suggest that all identified cognitive mechanisms have been implemented individually in isolated computational architectures, raising the question of why there exists no single unifying architecture that integrates them. As our final contribution, we address this question by providing an integrative perspective on the computational challenges to develop such a unifying architecture. We expect our results to guide the development of more sophisticated cognitively inspired hierarchical machine learning architectures.

Studying Complex Evolution of Hyperelastic Materials under External Field Stimuli using Artificial Neural Networks with Spatiotemporal Features in a Small‐Scale Dataset

“…Although the MAE, MSE, and Huber of CBLSTM‐1 are lower than using CNN or BLSTM alone, they are higher than those of a CBLSTM model that extracts features separately (Figure S9, Supporting Information), which is attributed to the hierarchical learning mechanism helps to reduce the difficulty of the model's understanding of the task. [ 35 ]…”

“…This is because the hierarchical learning mechanism was beneficial to solving the problem by reducing the difficult of the tasks. [ 35 ] The CBLSTM model was designed for common stimulus–response tasks in materials science experiments. Its basic structure included four convolutional layers and two BLSTM units.…”

“…Although the MAE, MSE, and Huber of CBLSTM-1 are lower than using CNN or BLSTM alone, they are higher than those of a CBLSTM model that extracts features separately (Figure S9, Supporting Information), which is attributed to the hierarchical learning mechanism helps to reduce the difficulty of the model's understanding of the task. [35] To select the most suitable loss function for different models, we plotted the prediction error CDFs of models with the MAE, MSE, and Huber as the loss functions (Figure 4D,H,L). The curve of MAE is closer to the y-axis, meaning that the MAE predicts the stress-strain relation with higher precision than the MSE and the Huber in three models.…”

Inference of Affordances and Active Motor Control in Simulated Agents

“…As a result, planning via active inference becomes more effective and enables, for example, the avoidance of uncertainty while moving towards a given goal location. We furthermore show that the architecture exhibits zero-shot learning abilities [Eppe et al, 2022], directly solving related environments and tasks within.…”

“…Without further modifications, our agent would not make such a detour deliberately. In the future, we want to investigate how this can be mitigated through hierarchical planning on events [Eppe et al, 2022].…”