Lifelong Augmentation of Multimodal Streaming Autobiographical Memories

Abstract

Abstract-Robot systems that interact with humans over extended periods of time will benefit from storing and recalling large amounts of accumulated sensorimotor and interaction data. We provide a principled framework for the cumulative organisation of streaming autobiographical data so that data can be continuously processed and augmented as the processing and reasoning abilities of the agent develop and further interactions with humans take place. As an example, we show how a kinematic structure learning algorithm reasons a-posteriori about the skeleton of a human hand. A partner can be asked to provide feedback about the augmented memories, which can in turn be supplied to the reasoning processes in order to adapt their parameters. We employ active, multi-modal remembering, so the robot as well as humans can gain insights of both the original and augmented memories. Our framework is capable of storing discrete and continuous data in real-time. The data can cover multiple modalities and several layers of abstraction (e.g. from raw sound signals over sentences to extracted meanings). We show a typical interaction with a human partner using an iCub humanoid robot. The framework is implemented in a platform-independent manner. In particular, we validate its multi platform capabilities using the iCub, Baxter and NAO robots. We also provide an interface to cloud based services, which allow automatic annotation of episodes. Our framework is geared towards the developmental robotics community, as it 1) provides a variety of interfaces for other modules, 2) unifies previous works on autobiographical memory, and 3) is licensed as open source software.

DAC-h3: A Proactive Robot Cognitive Architecture to Acquire and Express Knowledge About the World and the Self

“…3) Autobiographical Memory: The autobiographical memory (ABM [101], [102], [103]) collects long term information (days, months, years) about interactions motivated by the human declarative long term memory situated in the medial temporal lobe, and the distinction between facts and events [101]. It stores data (e.g.…”

“…In addition, the ABM captures continuous information during an episode (e.g. images from the camera, joints values) [102], which can be used by reasoning modules focusing on the action itself, leading to the production of a procedural memory (e.g. through learning from motor babbling or imitation) [103].…”

Developmental Bayesian Optimization of Black-Box with Visual Similarity-Based Transfer Learning

“…The memories are stored in a PostgreSQL 2 database similarly to other implementation of long-term memory system [16], [26]. The episodic memory stores for each iteration i of each run r the label of the object, the set of n hyperparameters {p 1 (i), p 2 (i), ..., p n (i)} tested and the score s i of the performance.…”

Hierarchical action learning by instruction through interactive grounding of body parts and proto-actions

“…The Microsoft Speech Recognition software detects the spoken utterances, following a predefined grammar that allows semantic roles extraction for words in spoken utterances. An autobiographical-like memory system under the form of a PostgreSQL database [22] is used to store discrete events (e.g. recognized sentences, motor babbling actions) along continuous data (e.g.…”