PROPRE: PROjection and PREdiction for multimodal correlations learning. An application to pedestrians visual data discrimination

Abstract

PROPRE is a generic and modular unsupervised neural learning paradigm that extracts meaningful concepts of multimodal data flows based on predictability across modalities. It consists on the combination of three modules. First, a topological projection of each data flow on a self-organizing map. Second, a decentralized prediction of each projection activity from each others map activities. Third, a predictability measure that compares predicted and real activities. This measure is used to modulate the projection learning so that to favor the mapping of predictable stimuli across modalities. In this article, we use Kohonen map for the projection module, linear regression for the prediction one and we propose multiple generic predictability measures. We illustrate the properties and performances of PROPRE paradigm on a challenging supervised classification task of visual pedestrian data. The modulation of the projection learning by the predictability measure improves significantly classification performances of the system independently of the measure used. Moreover, PROPRE provides a combination of interesting functional properties, such as a dynamical adaptation to input statistic variations, that is rarely available in other machine learning algorithms.

Active learning of local predictable representations with artificial curiosity

“…Second, it influences the choice of the next input of the model so that to favor areas of the input space leading to learning progress, thus providing an active learning of the input/output relationship (see section II-D2 for details). Thus, PROPRE is an hybrid architecture that provides online, adaptive, active learning [12] that can also be unsupervised when using PROPRE in a multimodal context where The predictability module, that monitors the quality of the prediction (P ), modulates the generative learning so that to favor representations that are better to predict of the input/output relationship than a local average. It also chooses the next input received by the model, depending on the learning progress, closing the perception/action loop.…”

“…In our previous articles [1], [12], the projection step consisted of the classical self-organizing map model proposed by Kohonen [15]. This model is related to the minimization of the quantization error -plus a topological term -so that the distribution of the prototypes tends to be similar to the one of the input data [15], [16].…”

“…Thus, Q(t) represents the percentage of prediction of the right position of the hand. This measure was also used for classification learning [12] or multimodal regression [13].…”

“…This interestingness measure modulates the generative learning so that to favor the learning of representations that predict the target better than a local average. This predictability modulation mechanism provides a better representations learning, for instance for visual pedestrian pose classification [12]. Moreover, it leads to the gathering of the representations where a relationship with the output is defined, thus avoiding unlearnable areas of the input space [1], so that PROPRE can be well suited for the developmental robotics field.…”

Multimodal space representation driven by self-evaluation of predictability

“…We also show that the self-evaluation module of predictability leads to learning of representations that improve classification of visual pedestrian data in a supervised context. Moreover, these representations can be incrementally updated to take into account various changes in the data flows [2]. Following our target to use PROPRE for multimodal online and incremental learning on real developmental robotic platforms, in this article, we apply it to the unsupervised learning of hand gestures observed by two time-of-flight (ToF) cameras and also propose a new improved self-evaluation module.…”