Hand–object interaction recognition based on visual attention using multiscopic cyber-physical-social system

Abstract

Computer vision-based cyber-physical-social systems (CPSS) are predicted to be the future of independent hand rehabilitation. However, there is a link between hand function and cognition in the elderly that this technology has not adequately supported. To investigate this issue, this paper proposes a multiscopic CPSS framework by developing hand–object interaction (HOI) based on visual attention. First, we use egocentric vision to extract features from hand posture at the microscopic level. With 94.87% testing accuracy, we use three layers of graph neural network (GNN) based on hand skeletal features to categorize 16 grasp postures. Second, we use a mesoscopic active perception ability to validate the HOI with eye tracking in the task-specific reach-to-grasp cycle. With 90.75% testing accuracy, the distance between the fingertips and the center of an object is used as input to a multi-layer gated recurrent unit based on recurrent neural network architecture. Third, we incorporate visual attention into the cognitive ability for classifying multiple objects at the macroscopic level. In two scenarios with four activities, we use GNN with three convolutional layers to categorize some objects. The outcome demonstrates that the system can successfully separate objects based on related activities. Further research and development are expected to support the CPSS application in independent rehabilitation.

Multiscopic CPSS for Independent Block-Design Test Based on Hand–Object Interaction Recognition With Visual Attention

“…As in our previous work on the chopsticks manipulation test (CMT), we conducted a multiscopic approach to examine the importance of combining finger joint angle estimation and visual attention measurement in hand rehabilitation [34]. This study further supported the use of multiscopic methods to address dynamic locomotion in legged robots [35], cognitive memory systems for continuous gesture learning [36], and the application of CPSS for activity daily living (ADL) [37]. Building upon these insights, we propose a multiscopic approach to develop a CPSS for HOI recognition based on visual attention specifically for the BDT.…”

“…We have successfully evaluated the multiscopic CPSS [37] as a novel framework for BDT applications, demonstrating high accuracy across the microscopic, mesoscopic, and macroscopic models. However, it is essential to note that the system's accuracy may vary when applied to different datasets or under varying conditions.…”