Multimodal Signal Processing and Learning Aspects of Human-Robot Interaction for an Assistive Bathing Robot

Zlatintsi, Athanasia; Rodomagoulakis, Isidoros; Koutras, Petros; Dometios, Athanasios; Pitsikalis, Vassilis; Tzafestas, Costas S.; Maragos, Petros

doi:10.1109/icassp.2018.8461568

Cited by 20 publications

(15 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In order to build our models we performed offline classification experiments of pre-segmented commands, based on a task-dependent grammar of 23 German spoken commands, accompishing accuracies of 76% and 68% for the two tasks. For a more detailed analysis and further results regarding the offline experiments of the audio-gestural HRI module we refer the reader to [59].…”

Section: Audio Processingmentioning

confidence: 99%

I-Support: A robotic platform of an assistive bathing robot for the elderly population

Zlatintsi

Dometios

Kardaris

et al. 2020

Robotics and Autonomous Systems

Self Cite

View full text Add to dashboard Cite

In this paper we present a prototype integrated robotic system, the I-Support bathing robot, that aims at supporting new aspects of assisted daily-living activities on a real-life scenario. The paper focuses on describing and evaluating key novel technological features of the system, with the emphasis on cognitive human-robot interaction modules and their evaluation through a series of clinical validation studies. The I-Support project on its whole has envisioned the development of an innovative, modular, ICTsupported service robotic system that assists frail seniors to safely and independently complete an entire sequence of physically and cognitively demanding bathing tasks, such as properly washing their back and their lower limbs. A variety of innovative technologies have been researched and a set of advanced modules of sensing, cognition, actuation and control have been developed and seamlessly integrated to enable the system to adapt to the target population abilities. These technologies include: human activity monitoring and recognition, adaptation of a motorized chair for safe transfer of the elderly in and out the bathing cabin, a context awareness system that provides full environmental awareness, as well as a prototype soft robotic arm and a set of user-adaptive robot motion planning and control algorithms. This paper focuses in particular on the multimodal action recognition system, developed to monitor, analyze and predict user actions with a high level of accuracy and detail in real-time, which are then interpreted as robotic tasks. In the same framework, the analysis of human actions that have become available through the project's multimodal audio-gestural dataset, has led to the successful modelling of Human-Robot Communication, achieving an effective and natural interaction between users and the assistive robotic platform. In order to evaluate the I-Support system, two multinational validation studies were conducted under realistic operating conditions in two clinical pilot sites. Some of the findings of these studies are presented and analysed in the paper, showing good results in terms of: (i) high acceptability regarding the system usability by this particularly challenging target group, the elderly end-users, and (ii) overall task effectiveness of the system in different operating modes.

show abstract

Section: Audio Processingmentioning

confidence: 99%

I-Support: A robotic platform of an assistive bathing robot for the elderly population

Zlatintsi

Dometios

Kardaris

et al. 2020

Robotics and Autonomous Systems

Self Cite

View full text Add to dashboard Cite

show abstract

“…Surpassing human-level performance propelled the research in applications where different modalities amongst language, vision, sensory, text play an essential role inaccurate predictions and identification [ 45 ]. Several state-of-the-art approaches in multimodal fusion employing deep learning models are proposed in the literature, such as approaches presented by F. Ramzan et al [ 24 ], A. Zlatintsi et al [ 25 ], M. Dhouib and S. Masmoudi [ 26 ], Y. D. Zhang et al [ 27 ], C. Devaguptapu et al [ 28 ], and P. Narkhede et al [ 46 ]. The purpose of these approaches is to enhance the multimodal fusion method with which the objects can be efficiently detected from static images or given video sequences with the preferable use of the deep learning library.…”

Section: Theoretical Backgroundmentioning

confidence: 99%

“…In contrast, the previous approaches [ 16 , 17 , 18 , 19 , 20 ] have preferred to use holistic techniques based on the silhouette in visible (RGB) imaging to identify the individual. (2) The proposed approach implements a YOLOv3 model-based method for fusing face and gait, which is better than other related methods used in the CNNs model [ 17 , 21 , 24 , 25 ], SVM, ANN technology [ 26 ], and YOLO model [ 27 , 28 ]. Besides, our approach seeks to increase recognition accuracy without combining visual (RGB) imaging features with thermal imaging features compared to related methods in the same night dataset.…”

Section: Introductionmentioning

confidence: 99%

Real-Time Human Recognition at Night via Integrated Face and Gait Recognition Technologies

Manssor

Sun

Elhassan

2021

Sensors

View full text Add to dashboard Cite

Human recognition technology is a task that determines the people existing in images with the purpose of identifying them. However, automatic human recognition at night is still a challenge because of its need to align requirements with a high accuracy rate and speed. This article aims to design a novel approach that applies integrated face and gait analyses to enhance the performance of real-time human recognition in TIR images at night under various walking conditions. Therefore, a new network is proposed to improve the YOLOv3 model by fusing face and gait classifiers to identify individuals automatically. This network optimizes the TIR images, provides more accurate features (face, gait, and body segment) of the person, and possesses it through the PDM-Net to detect the person class; then, PRM-Net classifies the images for human recognition. The proposed methodology uses accurate features to form the face and gait signatures by applying the YOLO-face algorithm and YOLO algorithm. This approach was pre-trained on three night (DHU Night, FLIR, and KAIST) databases to simulate realistic conditions during the surveillance-protecting areas. The experimental results determined that the proposed method is superior to other results-related methods in the same night databases in accuracy and detection time.

show abstract

“…The respective classifiers' output scores for all possible intentions were fused by a weighted linear combination with tunable weights while the intention with the highest fused score was predicted. Another work operating on classifier outputs for fusion by Zlatintsi et al [11] proposed an intention recognition system for an assistive bathing robot based on speech and gestures. They applied a late fusion scheme meaning that an intention was chosen as the detected one if it was ranked highest by the speech classifier and among the two highest ranked intentions according to observed gestures.…”

Section: Related Workmentioning

confidence: 99%

Multimodal Uncertainty Reduction for Intention Recognition in Human-Robot Interaction

Trick

Koert

Peters

et al. 2019

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

Assistive robots can potentially improve the quality of life and personal independence of elderly people by supporting everyday life activities. To guarantee a safe and intuitive interaction between human and robot, human intentions need to be recognized automatically. As humans communicate their intentions multimodally, the use of multiple modalities for intention recognition may not just increase the robustness against failure of individual modalities but especially reduce the uncertainty about the intention to be predicted. This is desirable as particularly in direct interaction between robots and potentially vulnerable humans a minimal uncertainty about the situation as well as knowledge about this actual uncertainty is necessary. Thus, in contrast to existing methods, in this work a new approach for multimodal intention recognition is introduced that focuses on uncertainty reduction through classifier fusion. For the four considered modalities speech, gestures, gaze directions and scene objects individual intention classifiers are trained, of which all output a probability distribution over all possible intentions. By combining these output distributions using the Bayesian method Independent Opinion Pool [1] the uncertainty about the intention to be predicted can be decreased. The approach is evaluated in a collaborative human-robot interaction task with a 7-DoF robot arm. The results show that fused classifiers which combine multiple modalities outperform the respective individual base classifiers with respect to increased accuracy, robustness, and reduced uncertainty.

show abstract

Multimodal Signal Processing and Learning Aspects of Human-Robot Interaction for an Assistive Bathing Robot

Cited by 20 publications

References 17 publications

I-Support: A robotic platform of an assistive bathing robot for the elderly population

I-Support: A robotic platform of an assistive bathing robot for the elderly population

Real-Time Human Recognition at Night via Integrated Face and Gait Recognition Technologies

Multimodal Uncertainty Reduction for Intention Recognition in Human-Robot Interaction

Contact Info

Product

Resources

About