Human-machine interfaces (HMIs) experience increasing requirements for intuitive and effective manipulation. Current commercialized solutions of glove-based HMI are limited by either detectable motions or the huge cost on fabrication, energy, and computing power. We propose the haptic-feedback smart glove with triboelectric-based finger bending sensors, palm sliding sensor, and piezoelectric mechanical stimulators. The detection of multidirectional bending and sliding events is demonstrated in virtual space using the self-generated triboelectric signals for various degrees of freedom on human hand. We also perform haptic mechanical stimulation via piezoelectric chips to realize the augmented HMI. The smart glove achieves object recognition using machine learning technique, with an accuracy of 96%. Through the integrated demonstration of multidimensional manipulation, haptic feedback, and AI-based object recognition, our glove reveals its potential as a promising solution for low-cost and advanced human-machine interaction, which can benefit diversified areas, including entertainment, home healthcare, sports training, and medical industry.

Wearable devices rely on hybrid mechanisms that possess the advantages of establishing a smarter system for healthcare, sports monitoring, and smart home applications. Socks with sensing capabilities can reveal more direct sensory information on the body for longer duration in daily life. However, the limitation of suitable materials for smart textile makes the development of multifunctional socks a major challenge. In this paper, we have developed a self-powered and self-functional sock (S2-sock) to realize diversified functions including energy harvesting and sensing various physiological signals, i.e., gait, contact force, sweat level, etc., by hybrid integrating poly­(3,4-ethylenedioxythiophene) polystyrenesulfonate (PEDOT:PSS)-coated fabric triboelectric nanogenerator (TENG) and lead zirconate titanate (PZT) piezoelectric chips. An output power of 1.71 mW is collected from a PEDOT:PSS-coated sock with mild jumping at 2 Hz and load resistance of 59.7 MΩ. The study shows that cotton socks worn daily can potentially be a power source for enabling self-sustained socks comprising wireless transmission modules and integrated circuits in the future. We also investigate the influences of environmental humidity, temperature, and weight variations and verify that our S2-sock can successfully achieve walking pattern recognition and motion tracking for smart home applications. On the basis of the sensor fusion concept, the outputs from TENG and PZT sensors under exercise activities are effectively merged together for quick detection of the sweat level. By leveraging the hybrid S2-sock, we can achieve more functionality in the applications of foot-based energy harvesting and monitoring the diversified physiological signals for healthcare, smart homes, etc.

Designing efficient sensors for soft robotics aiming at human machine interaction remains a challenge. Here, we report a smart soft-robotic gripper system based on triboelectric nanogenerator sensors to capture the continuous motion and tactile information for soft gripper. With the special distributed electrodes, the tactile sensor can perceive the contact position and area of external stimuli. The gear-based length sensor with a stretchable strip allows the continuous detection of elongation via the sequential contact of each tooth. The triboelectric sensory information collected during the operation of soft gripper is further trained by support vector machine algorithm to identify diverse objects with an accuracy of 98.1%. Demonstration of digital twin applications, which show the object identification and duplicate robotic manipulation in virtual environment according to the real-time operation of the soft-robotic gripper system, is successfully created for virtual assembly lines and unmanned warehouse applications.

The rapid progress of Internet of things (IoT) technology raises an imperative demand on human machine interfaces (HMIs) which provide a critical linkage between human and machines. Using a glove as an intuitive and low‐cost HMI can expediently track the motions of human fingers, resulting in a straightforward communication media of human–machine interactions. When combining several triboelectric textile sensors and proper machine learning technique, it has great potential to realize complex gesture recognition with the minimalist‐designed glove for the comprehensive control in both real and virtual space. However, humidity or sweat may negatively affect the triboelectric output as well as the textile itself. Hence, in this work, a facile carbon nanotubes/thermoplastic elastomer (CNTs/TPE) coating approach is investigated in detail to achieve superhydrophobicity of the triboelectric textile for performance improvement. With great energy harvesting and human motion sensing capabilities, the glove using the superhydrophobic textile realizes a low‐cost and self‐powered interface for gesture recognition. By leveraging machine learning technology, various gesture recognition tasks are done in real time by using gestures to achieve highly accurate virtual reality/augmented reality (VR/AR) controls including gun shooting, baseball pitching, and flower arrangement, with minimized effect from sweat during operation.

The era of artificial intelligence and internet of things is rapidly developed by recent advances in wearable electronics. Gait reveals sensory information in daily life containing personal information, regarding identification and healthcare. Current wearable electronics of gait analysis are mainly limited by high fabrication cost, operation energy consumption, or inferior analysis methods, which barely involve machine learning or implement nonoptimal models that require massive datasets for training. Herein, we developed low-cost triboelectric intelligent socks for harvesting waste energy from low-frequency body motions to transmit wireless sensory data. The sock equipped with self-powered functionality also can be used as wearable sensors to deliver information, regarding the identity, health status, and activity of the users. To further address the issue of ineffective analysis methods, an optimized deep learning model with an end-to-end structure on the socks signals for the gait analysis is proposed, which produces a 93.54% identification accuracy of 13 participants and detects five different human activities with 96.67% accuracy. Toward practical application, we map the physical signals collected through the socks in the virtual space to establish a digital human system for sports monitoring, healthcare, identification, and future smart home applications.

Triboelectric nanogenerators and sensors can be applied as human− machine interfaces to the next generation of intelligent and interactive products, where flexible tactile sensors exhibit great advantages for diversified applications such as robotic control. In this paper, we present a self-powered, flexible, triboelectric sensor (SFTS) patch for finger trajectory sensing and further apply the collected information for robotic control. This innovative sensor consists of flexible and environmentally friendly materials, i.e., starch-based hydrogel, polydimethylsiloxane (PDMS), and silicone rubber. The sensor patch can be divided into a two-dimensional (2D) SFTS for in-plane robotic movement control and a one-dimensional (1D) SFTS for out-of-plane robotic movement control. The 2D-SFTS is designed with a grid structure on top of the sensing surface to track the continuous sliding information on the fingertip, e.g., trajectory, velocity, and acceleration, with four circumjacent starchbased hydrogel PDMS elastomer electrodes. Combining the 2D-SFTS with the 1D-SFTS, three-dimensional (3D) spatial information can be generated and applied to control the 3D motion of a robotic manipulator, and the real-time demonstration is successfully realized. With the facile design and very low-cost materials, the proposed SFTS shows great potential for applications in robotics control, touch screens, and electronic skins.

With the prospect of a smart society in the foreseeable future, humans are experiencing an increased link to electronics in the digital world, which can benefit our life and productivity drastically. In recent decades, advances in the Human Machine Interface (HMI) have improved from tactile sensors, such as touchpads and joysticks, to now include the accurate detection of dexterous body movements in more diversified and sophisticated devices. Advancements in highly adaptive machine learning techniques, neural interfaces, and neuromorphic sensing have generated the potential for an economic and feasible solution for next-generation applications such as wearable HMIs with intimate and multi-directional sensing capabilities. This review offers a general knowledge of HMI technologies beginning with tactile sensors and their piezoresistive, capacitive, piezoelectric, and triboelectric sensing mechanisms. A further discussion is given on how machine learning, neural interfaces, and neuromorphic electronics can be used to enhance next-generation HMIs in an upcoming 5 G infrastructure and advancements in the internet of things and artificial intelligence of things in the near future. The efficient interactions with kinetic and physiological signals from human body through the fusion of tactile sensor and neural electronics will bring a revolution to both the advanced manipulation and medical rehabilitation.

Advancements of virtual reality technology pave the way for developing wearable devices to enable somatosensory sensation, which can bring more comprehensive perception and feedback in the metaverse-based virtual society. Here, we propose augmented tactile-perception and haptic-feedback rings with multimodal sensing and feedback capabilities. This highly integrated ring consists of triboelectric and pyroelectric sensors for tactile and temperature perception, and vibrators and nichrome heaters for vibro- and thermo-haptic feedback. All these components integrated on the ring can be directly driven by a custom wireless platform of low power consumption for wearable/portable scenarios. With voltage integration processing, high-resolution continuous finger motion tracking is achieved via the triboelectric tactile sensor, which also contributes to superior performance in gesture/object recognition with artificial intelligence analysis. By fusing the multimodal sensing and feedback functions, an interactive metaverse platform with cross-space perception capability is successfully achieved, giving people a face-to-face like immersive virtual social experience.