Deep‐Learning‐Assisted Noncontact Gesture‐Recognition System for Touchless Human‐Machine Interfaces

Abstract

Human‐machine interfaces (HMIs) play important role in the communication between humans and robots. Touchless HMIs with high hand dexterity and hygiene hold great promise in medical applications, especially during the pandemic of coronavirus disease 2019 (COVID‐19) to reduce the spread of virus. However, current touchless HMIs are mainly restricted by limited types of gesture recognition, the requirement of wearing accessories, complex sensing platforms, light conditions, and low recognition accuracy, obstructing their practical applications. Here, an intelligent noncontact gesture‐recognition system is presented through the integration of a triboelectric touchless sensor (TTS) and deep learning technology. Combined with a deep‐learning‐based multilayer perceptron neural network, the TTS can recognize 16 different types of gestures with a high average accuracy of 96.5%. The intelligent noncontact gesture‐recognition system is further applied to control a robot for collecting throat swabs in a noncontact mode. Compared with present touchless HMIs, the proposed system can recognize diverse complex gestures by utilizing charges naturally carried on human fingers without the need of wearing accessories, complicated device structures, adequate light conditions, and achieves high recognition accuracy. This system could provide exciting opportunities to develop a new generation of touchless medical equipment, as well as touchless public facilities, smart robots, virtual reality, metaverse, etc.

Triboelectric Nanogenerator Enabled Wearable Sensors and Electronics for Sustainable Internet of Things Integrated Green Earth

“…Starting from the conventional control terminals, such as joystick, keyboard, and touchpad, there are enormous efforts focus on the wearable devices to facilitate the intuitive human machine interactions and the immersive haptic feedback. [188][189][190][191][192][193][194][195] Owing to the advancements of various materials, the conventional piezoresistive and capacitive sensors are able to be designed as flexible patches with the great wearability. Hua et al have developed a stretchable sensory e-skin with the multimodal sensing capabilities, including temperature, strain, humidity, light, magnetic field, pressure, and proximity, as illustrated in Figure 9a.…”

“…Starting from the conventional control terminals, such as joystick, keyboard, and touchpad, there are enormous efforts focus on the wearable devices to facilitate the intuitive human machine interactions and the immersive haptic feedback. [ 188–195 ]…”

Water-Modulated Biomimetic Hyper-Attribute-Gel Electronic Skin for Robotics and Skin-Attachable Wearables

“…and monitor body activities and positions (e.g., proprioception, gesture, etc. ), which is widely used for health monitoring and human-machine interfaces. − Furthermore, symbiosis with human skin at the interface of prosthetics or even replacement of defective skin is the ultimate goal of e-skin. Thus, the ideal e-skin should cover all the property portfolio of human skin, including physical–chemical properties (i.e., stretchability, self-healing, biocompatibility, biodegradability, weak acidity, antibacterial activities, fire retardance, temperature adaptivity), as well as sensory properties (temperature, humidity, tactile, strain, etc.…”

Recent Progress in Self-Powered Wireless Sensors and Systems Based on TENG

“…A nanogenerator is generally considered the ideal real-time power supply for wireless sensors. Since the triboelectric nanogenerator (TENG) was invented by Wang’s group in 2012, many researchers have proved the TENG is a novel, feasible, and effective device to harvest energy from the ambient environment [ 8 , 12 , 13 , 14 ]. As a new self-powered sensing technology, TENGs are aptly used in the self-powered wireless sensing field [ 15 , 16 , 17 , 18 ].…”