Finger Joint Angle Estimation With Visual Attention for Rehabilitation Support: A Case Study of the Chopsticks Manipulation Test

Abstract

Most East Asian rehabilitation centers offer chopsticks manipulation tests (CMT). In addition to impaired hand function, approximately two-thirds of stroke survivors have visual impairment related to eye movement. This article investigates the significance of combining finger joint angle estimation and a visual attention measurement in CMT. We present a multiscopic framework that consists of microscopic, mesoscopic, and macroscopic levels. We develop a feature extraction technique to extract the kinematic finger model at the microscopic level. At the mesoscopic level, we propose an active perception ability to detect the position and geometry of the finger on the chopsticks. The proposed framework estimates the proximal interphalangeal (PIP) joint angle on the index finger during CMT using fully connected cascade neural networks (FCC-NN). At the macroscopic level, we implement a cognitive ability by measuring visual attention during CMT. We further evaluate the proposed framework with a conventional test that counts the number of peanuts (NP) which are moved from one bowl to another using chopsticks within a particular time frame. For the evaluation, we introduce three parameters, namely joint angle estimation movement (JAEM), chopstick attention movement (CAM), and chopstick tip movement (CTM), by detecting the local minima and maxima of the signal. According to the experiment results, the velocity of these three parameters could indicate improvement in hand and eye function during CMT. We expect this study to benefit therapists and researchers by providing valuable information that is not accessible in the clinic. Code and datasets are available online at https://github.com/anom-tmu/cmt-attention.INDEX TERMS hand-eye interaction, rehabilitation evaluation, first-person vision, eye-gaze, eye-tracking.

Hand–object interaction recognition based on visual attention using multiscopic cyber-physical-social system

“…We used the YOLOv5 [24] model to extract the object's location from picture frames represented by the bounding box and labels. To improve this detection, the Simple Online Real-time Tracking (SORT) [29] technique was used.…”

“…From an egocentric standpoint, it is critical to support the current system. With a case study on the Chopsticks Manipulation Test, we examined the significance of combining finger joint angle estimation and a visual attention measurement in hand rehabilitation [24]. Our previous work used a multiscopic method to address dynamic locomotion in a legged robot [25] and simulation for human-robot interactions [26].…”

An efficient activity recognition for homecare robots from multi-modal communication dataset

“…Many efforts have been made to recognize human activities using images and videos with 2D and 3D visual information [5]- [8]. Semantics and the context of a situation are usually used for classification, which involves typical HOI [9]- [11]. However, most of their frameworks require massive labelled datasets and much training time to achieve a high level of accuracy.…”