A Multimodal User Interface for an Assistive Robotic Shopping Cart

Ryumin, Dmitry; Kagirov, Ildar; Axyonov, Alexandr; Pavlyuk, Nikita; Saveliev, Anton; Kipyatkova, Irina; Železný, Miloš; Mporas, Iosif; Karpov, Alexey

doi:10.3390/electronics9122093

Cited by 24 publications

(14 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Within further research, it is considered to perform approbation of the proposed solution on different robotic platforms [26,27] to assess accuracy and operation speed of this solution on low-powered computational devices.…”

Section: Discussionmentioning

confidence: 99%

Approach to Image-Based Recognition of User Face in Setting of Partial Face Occlusion by Personal Protective Equipment

Letenkov

Iakovlev

Karpov

2021

Electromechanics and Robotics

Self Cite

View full text Add to dashboard Cite

Within this research, aimed to solve the problem of face recognition in setting of partial face occlusion by personal protective equipment an approach was proposed, which assumed training of a neural network model FaceNet on a specialized dataset Masked VGGFace2. This paper presents a strategy of training dataset augmentation based on the convolutional neural network model MTCNN and MaskTheFace software. Using this suite, based on the VGGFace2 dataset, an extended dataset Masked VGGFace2 was established, including images of human faces, partially occluded by personal protective equipment. Within this research, a series of experiments was performed, aimed to qualitative assessment of predictions, obtained with this solution, in comparison to the FaceNet model, which was trained on the original VGGFace2 dataset. The performed experiments revealed, that the proposed solution demonstrates a significantly higher recognition quality of partially occluded faces (AP = 0.9488)-recognition accuracy increase as per average precision (AP) metric was over 24%, compared to the original model. On test subset from the original dataset VGGFace2, the value of the AP metric was 0.9635, what outperformed the respective metric of the original model by 2.8% and justified a high genericity of the proposed solution in terms of face recognition problem.

show abstract

Section: Discussionmentioning

confidence: 99%

Approach to Image-Based Recognition of User Face in Setting of Partial Face Occlusion by Personal Protective Equipment

Letenkov

Iakovlev

Karpov

2021

Electromechanics and Robotics

Self Cite

View full text Add to dashboard Cite

show abstract

“…Multimodal interaction is also into the assistive robotic shopping cart for people with hearing impairments, presented in [53]. This work focuses on Russian sign language training through a single-handed gesture recognition and a touch control screen.…”

Section: Navigation and Human-robot Interactionmentioning

confidence: 99%

Waymarking in Social Robots: Environment Signaling Using Human–Robot Interaction

Corrales-Paredes

Malfáz

Egido-García

et al. 2021

Sensors

View full text Add to dashboard Cite

Travellers use the term waymarking to define the action of posting signs, or waymarks, along a route. These marks are intended to be points of reference during navigation for the environment. In this research, we will define waymarking as the skill of a robot to signal the environment or generate information to facilitate localization and navigation, both for its own use and for other robots as well. We present an automated environment signaling system using human–robot interaction and radio frequency identification (RFID) technology. The goal is for the robot, through human–robot interaction, to obtain information from the environment and use this information to carry out the signaling or waymarking process. HRI will play a key role in the signaling process since this type of communication makes it possible to exchange more specific and enriching information. The robot uses common phrases such as “Where am I?” and “Where can I go?”, just as we humans do when we ask other people for information about the environment. It is also possible to guide the robot and “show” it the environment to carry out the task of writing the signs. The robot will use the information received to create, update, or improve the navigation data in the RFID signals. In this paper, the signaling process will be described, how the robot acquires the information for signals, writing and updating process and finally, the implementation and integration in a real social robot in a real indoor environment.

show abstract

“…Наконец, решение проблемы набора данных, в конечном счете, сильно зависит от жестового языка, с которым работает исследователь. Авторы настоящей статьи представили собственный набор данных для русского жестового языка в трехмерном формате [53]; впоследствии этот набор данных использовался для системы автоматического распознавания жестов на русском жестовом языке [54][55][56]. Introduction: Currently, the recognition of gestures and sign languages is one of the most intensively developing areas in computer vision and applied linguistics.…”

Section: выводы по разделуunclassified

Analytical review of models and methods for automatic recognition of gestures and sign languages

Ryumin¹,

Kagirov²,

Axyonov³

et al. 2021

ICS

Self Cite

View full text Add to dashboard Cite

Introduction: Currently, the recognition of gestures and sign languages is one of the most intensively developing areas in computer vision and applied linguistics. The results of current investigations are applied in a wide range of areas, from sign language translation to gesture-based interfaces. In that regard, various systems and methods for the analysis of gestural data are being developed. Purpose: A detailed review of methods and a comparative analysis of current approaches in automatic recognition of gestures and sign languages. Results: The main gesture recognition problems are the following: detection of articulators (mainly hands), pose estimation and segmentation of gestures in the flow of speech. The authors conclude that the use of two-stream convolutional and recurrent neural network architectures is generally promising for efficient extraction and processing of spatial and temporal features, thus solving the problem of dynamic gestures and coarticulations. This solution, however, heavily depends on the quality and availability of data sets. Practical relevance: This review can be considered a contribution to the study of rapidly developing sign language recognition, irrespective to particular natural sign languages. The results of the work can be used in the development of software systems for automatic gesture and sign language recognition.

show abstract

A Multimodal User Interface for an Assistive Robotic Shopping Cart

Cited by 24 publications

References 47 publications

Approach to Image-Based Recognition of User Face in Setting of Partial Face Occlusion by Personal Protective Equipment

Approach to Image-Based Recognition of User Face in Setting of Partial Face Occlusion by Personal Protective Equipment

Waymarking in Social Robots: Environment Signaling Using Human–Robot Interaction

Analytical review of models and methods for automatic recognition of gestures and sign languages

Contact Info

Product

Resources

About