Achieving "synergy" in cognitive behavior of humanoids via deep learning of dynamic visuo-motor-attentional coordination

Hwang, Jungsik; Jung, Minsoo; Madapana, Naveen; Kim, Jinhyung; Choi, Minkyu; Tani, Jun

doi:10.1109/humanoids.2015.7363448

Cited by 11 publications

(18 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The VMDNN model was composed of 7 layers: the V I , V F , V S layers in the MSTNN subnetwork, the PFC layer, and the M S , M F , M O layers in the MTRNN subnetwork. The structure of the VMDNN model used in this study was found empirically in our preliminary experiments [46]. Note that the structure of the VMDNN model including the number of layers in each subnetwork can be extended depending on the complexity of the task since the 'deeper' structure can enhance learning of complex functions in visuomotor patterns [11].…”

Section: B Network Configurationmentioning

confidence: 99%

“…Each MSTNN layer consisted of a set of feature maps retaining the spatial information of the visual input. The for the time constant at each level of the model were found heuristically in our preliminary study [46].…”

Section: B Network Configurationmentioning

confidence: 99%

“…Throughout the experiments, the time constants of the M S , M F and M O layers were fixed to 70, 2 and 1 respectively. The proper values for the time constant at each level of the model were found heuristically in our preliminary study [46].…”

Section: B Network Configurationmentioning

confidence: 99%

See 2 more Smart Citations

Seamless Integration and Coordination of Cognitive Skills in Humanoid Robots: A Deep Learning Approach

Hwang

Tani

2018

IEEE Trans. Cogn. Dev. Syst.

Self Cite

View full text Add to dashboard Cite

AbstractThis study investigates how adequate coordination among the different cognitive processes of a humanoid robot can be developed through end-to-end learning of direct perception of visuomotor stream. We propose a deep dynamic neural network model built on a dynamic vision network, a motor generation network, and a higher-level network. The proposed model was designed to process and to integrate direct perception of dynamic visuomotor patterns in a hierarchical model characterized by different spatial and temporal constraints imposed on each level. We conducted synthetic robotic experiments in which a robot learned to read human's intention through observing the gestures and then to generate the corresponding goal-directed actions. Results verify that the proposed model is able to learn the tutored skills and to generalize them to novel situations. The model showed synergic coordination of perception, action and decision making, and it integrated and coordinated a set of cognitive skills including visual perception, intention reading, attention switching, working memory, action preparation and execution in a seamless manner. Analysis reveals that coherent internal representations emerged at each level of the hierarchy. Higher-level representation reflecting actional intention developed by means of continuous integration of the lower-level visuo-proprioceptive stream.

show abstract

Section: B Network Configurationmentioning

confidence: 99%

Section: B Network Configurationmentioning

confidence: 99%

See 1 more Smart Citation

Seamless Integration and Coordination of Cognitive Skills in Humanoid Robots: A Deep Learning Approach

Hwang

Tani

2018

IEEE Trans. Cogn. Dev. Syst.

Self Cite

View full text Add to dashboard Cite

show abstract

“…The proposed model consists of two pathways (visual and proprioceptive pathway for perceiving and predicting the dynamic visual images and the perceptual outcome of the robot's intended actions respectively) and those two pathways are tightly coupled by means of the lateral connection at the highest layers in each pathway and end-to-end training of the dynamic visuo-proprioceptive patterns. The proposed model is an extension of our previous model [1,7,8] which was able to abstract and associate visual perception with proprioceptive information through a spatio-temporal hierarchical structure. In the current study, we extended the previous model under the predictive coding framework [2][3][4][5] to endow the model with several key features.…”

Section: Introductionmentioning

confidence: 99%

Predictive coding-based deep dynamic neural network for visuomotor learning

Hwang

Kim

Ahmadi

et al. 2017

2017 Joint IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob)

Self Cite

View full text Add to dashboard Cite

This study presents a dynamic neural network model based on the predictive coding framework for perceiving and predicting the dynamic visuo-proprioceptive patterns. In our previous study [1], we have shown that the deep dynamic neural network model was able to coordinate visual perception and action generation in a seamless manner. In the current study, we extended the previous model under the predictive coding framework to endow the model with a capability of perceiving and predicting dynamic visuo-proprioceptive patterns as well as a capability of inferring intention behind the perceived visuomotor information through minimizing prediction error. A set of synthetic experiments were conducted in which a robot learned to imitate the gestures of another robot in a simulation environment. The experimental results showed that with given intention states, the model was able to mentally simulate the possible incoming dynamic visuo-proprioceptive patterns in a top-down process without the inputs from the external environment. Moreover, the results highlighted the role of minimizing prediction error in inferring underlying intention of the perceived visuo-proprioceptive patterns, supporting the predictive coding account of the mirror neuron systems. The results also revealed that minimizing prediction error in one modality induced the recall of the corresponding representation of another modality acquired during the consolidative learning of raw-level visuo-proprioceptive patterns.

show abstract

“…Hwang et al [123] demonstrated gesture recognition with a recurrent model, and coordinated it with attention switching, object perception, and grasping. The robot focused on a human collaborator, who gestured to one of two objects.…”

Section: Examples In Recent Researchmentioning

confidence: 99%

Deep learning in robotics: a review of recent research

2017

View full text Add to dashboard Cite

Advances in deep learning over the last decade have led to a flurry of research in the application of deep artificial neural networks to robotic systems, with at least thirty papers published on the subject between 2014 and the present. This review discusses the applications, benefits, and limitations of deep learning vis-à-vis physical robotic systems, using contemporary research as exemplars. It is intended to communicate recent advances to the wider robotics community and inspire additional interest in and application of deep learning in robotics.

show abstract

Achieving "synergy" in cognitive behavior of humanoids via deep learning of dynamic visuo-motor-attentional coordination

Cited by 11 publications

References 30 publications

Seamless Integration and Coordination of Cognitive Skills in Humanoid Robots: A Deep Learning Approach

Seamless Integration and Coordination of Cognitive Skills in Humanoid Robots: A Deep Learning Approach

Predictive coding-based deep dynamic neural network for visuomotor learning

Deep learning in robotics: a review of recent research

Contact Info

Product

Resources

About