Intention Understanding in Human–Robot Interaction Based on Visual-NLP Semantics

Li, Zhihao; Mu, Yishan; Sun, Zhenglong; Song, Sifan; Su, Jionglong; Zhang, Jiaming

doi:10.3389/fnbot.2020.610139

Cited by 18 publications

(11 citation statements)

References 25 publications

(26 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…21 tactile sensor arrays are assembled on the inner surface of the ABS shell of the robot mentioned in [106] Realsense D435 Camera NVIDIA Jetson TX2 Figure 4: A new task-based framework that enables robots to understand human intentions using visual-NLP semantic information [109]: it includes a language semantics module to extract keywords no matter if the command directive is explicit or not, a visual object recognition module to identify multiple objects located to the front of the robot, and a similarity computation algorithm for inferring the intention based on a given task (i.e., selecting some desired item out of multiple objects on a table and giving it to a particular user among several human participants). Result of the similarity computation is then translated into structured robot control language RCL (grasp object to place) to be comprehended by robots.…”

Section: App (Instances) On the Touch Screenmentioning

confidence: 99%

“…An appropriated facial expression out of 20 choices (shyness, thinking (turning eyes), etc.) will be displayed to the child with body gestures and speech output sometimes Attention-based image captioning with gaze-following [110] A robot takes a video as input, then builds a 3D dense map based on SLAM, and estimates the gaze simultaneously Attention prediction is done by attention heat map adding occlusion and salience detection Te robot verbally describes the region focused on by the child based on attention heat mapsIntention understanding based on visual-NLP semantics and responding to the child accordingly[109] …”

mentioning

confidence: 99%

See 1 more Smart Citation

AppraisalCloudPCT: A Computational Model of Emotions for Socially Interactive Robots for Autistic Rehabilitation

Yan

Lin

Wang

et al. 2023

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

Computational models of emotions can not only improve the effectiveness and efficiency of human-robot interaction but also coordinate a robot to adapt to its environment better. When designing computational models of emotions for socially interactive robots, especially for robots for people with special needs such as autistic children, one should take into account the social and communicative characteristics of such groups of people. This article presents a novel computational model of emotions called AppraisalCloudPCT that is suitable for socially interactive robots that can be adopted in autistic rehabilitation which, to the best of our knowledge, is the first computational model of emotions built for robots that can satisfy the needs of a special group of people such as autistic children. To begin with, some fundamental and notable computational models of emotions (e.g., OCC, Scherer’s appraisal theory, PAD) that have deep and profound influence on building some significant models (e.g., PRESENCE, iGrace, xEmotion) for socially interactive robots are revisited. Then, a comparative assessment between our AppraisalCloudPCT and other five significant models for socially interactive robots is conducted. Great efforts have been made in building our proposed model to meet all of the six criteria for comparison, by adopting the appraisal theories on emotions, perceptual control theory on emotions, a component model view of appraisal models, and cloud robotics. Details of how to implement our model in a socially interactive robot we developed for autistic rehabilitation are also elaborated in this article. Future studies should examine how our model performs in different robots and also in more interactive scenarios.

show abstract

Section: App (Instances) On the Touch Screenmentioning

confidence: 99%

mentioning

confidence: 99%

AppraisalCloudPCT: A Computational Model of Emotions for Socially Interactive Robots for Autistic Rehabilitation

Yan

Lin

Wang

et al. 2023

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

show abstract

“…Li et al [23] use natural language processing to infer human-given commands for robots, by using keyword extraction, visual object recognition, and similarity computation. Its main intent is to use visual semantic information to allow a robot deduce task intents, avoiding simple keywords that map predefined tasks explicitly.…”

Section: Related Workmentioning

confidence: 99%

HOTSPOT: An Ad Hoc Teamwork Platform for Mixed Human-Robot Teams

Ribeiro¹,

Henriques²,

Colcher³

et al. 2021

Preprint

View full text Add to dashboard Cite

Ad hoc teamwork is a research topic in multi-agent systems whereby an agent (the "ad hoc agent") must successfully collaborate with a set of unknown agents (the "teammates") without any prior coordination or communication protocol. However, research in ad hoc teamwork is predominantly focused on agent-only teams, but not in agent-human teams, which we believe is an exciting research avenue and has enormous application potential in human-robot teams. This paper will tap into this potential by proposing HOTSPOT, the first framework for ad hoc teamwork in human-robot teams. Our framework comprises two main modules, addressing the two key challenges in the interaction between a robot acting as the ad hoc agent and human teammates. First, a <i>decision-theoretic module</i> that is responsible for all task-related decision making (task identification, teammate identification, and planning). Second, a <i>communication module</i> that uses natural language processing in order to parse all communication between the robot and the human. To evaluate our framework, we use a task where a mobile robot and a human cooperatively collect objects in an open space, illustrating the main features of our framework in a real-world task.

show abstract

“…Third, data may be scarce when a customized dataset needs to be created for a specific task. For example, when developing a robot to assist human workers, developers should create new data suitable for the working environment [10], [11]. Therefore, the data scarcity problem is an important topic to be addressed.…”

Section: Introductionmentioning

confidence: 99%

Exploiting All Samples in Low-Resource Sentence Classification: Early Stopping and Initialization Parameters

Choi

Lee

2023

IEEE Access

View full text Add to dashboard Cite

To improve deep-learning performance in low-resource settings, many researchers have redesigned model architectures or applied additional data (e.g., external resources, unlabeled samples). However, there have been relatively few discussions on how to make good use of small amounts of labeled samples, although it is potentially beneficial and should be done before applying additional data or redesigning models. In this study, we assume a low-resource setting in which only a few labeled samples (i.e., 30-100 per class) are available, and we discuss how to exploit them without additional data or model redesigns. We explore possible approaches in the following three aspects: training validation splitting, early stopping, and weight initialization. Extensive experiments are conducted on six public sentence classification datasets. Performance on various evaluation metrics (e.g., accuracy, loss, and calibration error) significantly varied depending on the approaches that were combined in the three aspects. Based on the results, we propose an integrated method, which is to initialize the model with a weight averaging method and use a nonvalidation stop method to train all samples. This simple integrated method consistently outperforms the competitive methods; e.g., the average accuracy of six datasets of this method was 1.8% higher than those of conventional validation-based methods. In addition, the integrated method further improves the performance when adapted to several state-of-the-art models that use additional data or redesign the network architecture (e.g., self-training and enhanced structural models). Our results highlight the importance of the training strategy and suggest that the integrated method can be the first step in the low-resource setting. This study provides empirical knowledge that will be helpful when dealing with low-resource data in future efforts. Our code is publicly available at https://github.com/DMCB-GIST/exploit_all_samples.

show abstract

Intention Understanding in Human–Robot Interaction Based on Visual-NLP Semantics

Cited by 18 publications

References 25 publications

AppraisalCloudPCT: A Computational Model of Emotions for Socially Interactive Robots for Autistic Rehabilitation

AppraisalCloudPCT: A Computational Model of Emotions for Socially Interactive Robots for Autistic Rehabilitation

HOTSPOT: An Ad Hoc Teamwork Platform for Mixed Human-Robot Teams

Exploiting All Samples in Low-Resource Sentence Classification: Early Stopping and Initialization Parameters

Contact Info

Product

Resources

About