A self-organizing neural network architecture for learning human-object interactions

Mici, Luiza; Parisi, German I.; Wermter, Stefan

doi:10.1016/j.neucom.2018.04.015

Cited by 33 publications

(22 citation statements)

References 58 publications

(94 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similar GWR-based approaches have been proposed for the incremental learning of body motion patterns (Mici et al 2017, Elfaramawy et al 2017, Parisi et al 2016) and human-object interaction (Mici et al 2018). However, these unsupervised learning approaches do not take into account top-down task-relevant signals that can regulate the stability-plasticity balance, potentially leading to scalability issues for large-scale datasets.…”

Section: Dynamic Architecturesmentioning

confidence: 99%

Continual lifelong learning with neural networks: A review

et al. 2019

Self Cite

View full text Add to dashboard Cite

Humans and animals have the ability to continually acquire, fine-tune, and transfer knowledge and skills throughout their lifespan. This ability, referred to as lifelong learning, is mediated by a rich set of neurocognitive mechanisms that together contribute to the development and specialization of our sensorimotor skills as well as to long-term memory consolidation and retrieval. Consequently, lifelong learning capabilities are crucial for computational systems and autonomous agents interacting in the real world and processing continuous streams of information. However, lifelong learning remains a long-standing challenge for machine learning and neural network models since the continual acquisition of incrementally available information from non-stationary data distributions generally leads to catastrophic forgetting or interference. This limitation represents a major drawback for state-of-the-art deep neural network models that typically learn representations from stationary batches of training data, thus without accounting for situations in which information becomes incrementally available over time. In this review, we critically summarize the main challenges linked to lifelong learning for artificial learning systems and compare existing neural network approaches that alleviate, to different extents, catastrophic forgetting. Although significant advances have been made in domain-specific learning with neural networks, extensive research efforts are required for the development of robust lifelong learning on autonomous agents and robots. We discuss well-established and emerging research motivated by lifelong learning factors in biological systems such as structural plasticity, memory replay, curriculum and transfer learning, intrinsic motivation, and multisensory integration.

show abstract

Section: Dynamic Architecturesmentioning

confidence: 99%

Continual lifelong learning with neural networks: A review

et al. 2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…The neural update rate decreases as the neurons become more habituated, which has the effect of preventing that noisy input interferes with consolidated neural representations. Similar GWR-based approaches have been proposed for the incremental learning of body motion patterns [54,8,61] and human-object interaction [55].…”

Section: Growing Self-organizing Networkmentioning

confidence: 99%

Online Continual Learning on Sequences

Parisi¹,

Lomonaco

2020

Recent Trends in Learning From Data

Self Cite

View full text Add to dashboard Cite

Online continual learning (OCL) refers to the ability of a system to learn over time from a continuous stream of data without having to revisit previously encountered training samples. Learning continually in a single data pass is crucial for agents and robots operating in changing environments and required to acquire, fine-tune, and transfer increasingly complex representations from non-i.i.d. input distributions. Machine learning models that address OCL must alleviate catastrophic forgetting in which hidden representations are disrupted or completely overwritten when learning from streams of novel input. In this chapter, we summarize and discuss recent deep learning models that address OCL on sequential input through the use (and combination) of synaptic regularization, structural plasticity, and experience replay. Different implementations of replay have been proposed that alleviate catastrophic forgetting in connectionists architectures via the re-occurrence of (latent representations of) input sequences and that functionally resemble mechanisms of hippocampal replay in the mammalian brain. Empirical evidence shows that architectures endowed with experience replay typically outperform architectures without in (online) incremental learning tasks.

show abstract

“…3D features when objects are small and partially occluded. Alternatively, other methods model human-object interactions considering the skeletal features combined with the object's identity (Rybok et al, 2014;Mici et al, 2018c).…”

Section: Single-layer Approachesmentioning

confidence: 99%

“…More formally, we propose a hierarchical arrangement of Growing When Required (GWR) networks (Marsland et al, 2002 ) which integrate multiple visual cues regarding the body pose, the manipulated object, and their spatial relation during human-object interaction, accumulated over a short and a longer period of time in order to jointly learn actions and human activities respectively. In our previous research, we have successfully applied and evaluated hierarchical architectures of the GWR network for clustering human body pose and motion patterns as well as for learning prototypical representations of human-object interactions in an unsupervised fashion (Parisi et al, 2015 ; Mici et al, 2018a , c ). The generative properties of the GWR networks have been shown to be particularly suitable for the human-object interaction recognition due to generalizing well to unseen action-object pairs.…”

Section: Introductionmentioning

confidence: 99%

Compositional Learning of Human Activities With a Self-Organizing Neural Architecture

2019

Self Cite

View full text Add to dashboard Cite

An important step for assistive systems and robot companions operating in human environments is to learn the compositionality of human activities, i.e., recognize both activities and their comprising actions. Most existing approaches address action and activity recognition as separate tasks, i.e., actions need to be inferred before the activity labels, and are thus highly sensitive to the correct temporal segmentation of the activity sequences. In this paper, we present a novel learning approach that jointly learns human activities on two levels of semantic and temporal complexity: (1) transitive actions such as reaching and opening, e.g., a cereal box, and (2) high-level activities such as having breakfast. Our model consists of a hierarchy of GWR networks which process and learn inherent spatiotemporal dependencies of multiple visual cues extracted from the human body skeletal representation and the interaction with objects. The neural architecture learns and semantically segments input RGB-D sequences of high-level activities into their composing actions, without supervision. We investigate the performance of our architecture with a set of experiments on a publicly available benchmark dataset. The experimental results show that our approach outperforms the state of the art with respect to the classification of the high-level activities. Additionally, we introduce a novel top-down modulation mechanism to the architecture which uses the actions and activity labels as constraints during the learning phase. In our experiments, we show how this mechanism can be used to control the network's neural growth without decreasing the overall performance.

show abstract

A self-organizing neural network architecture for learning human-object interactions

Cited by 33 publications

References 58 publications

Continual lifelong learning with neural networks: A review

Continual lifelong learning with neural networks: A review

Online Continual Learning on Sequences

Compositional Learning of Human Activities With a Self-Organizing Neural Architecture

Contact Info

Product

Resources

About