Feature-based Egocentric Grasp Pose Classification for Expanding Human-Object Interactions

Besari, Adnan Rachmat Anom; Saputra, Azhar Aulia; Chin, Wei Hong; Kubota, Naoyuki; Kurnianingsih, Kurnianingsih

doi:10.1109/isie45552.2021.9576369

Cited by 6 publications

(3 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The testing accuracy for the 16 grasps pose using the GNN model is 94.87%. This result outperformed the accuracy of the MLP network, which scored 78.75% in our previous work [34]. This discovery demonstrates that the gathered dataset of grasp postures contains essential characteristics.…”

Section: Discussion On Microscopic Levelsupporting

confidence: 48%

“…First, we conducted a single-participant task-specific reach-to-grasp cycle experiment. We extended the egocentric vision feature extraction capabilities discovered in our earlier research [34]. To make it easier to extract these characteristics, we used object-centered coordinate transformation to make it simple to extract these characteristics.…”

Section: Discussion On Mesoscopic Levelmentioning

confidence: 99%

See 1 more Smart Citation

Hand–object interaction recognition based on visual attention using multiscopic cyber-physical-social system

Besari¹,

Saputra

Chin

et al. 2023

Int. J. Adv. Intell. Informatics

View full text Add to dashboard Cite

Computer vision-based cyber-physical-social systems (CPSS) are predicted to be the future of independent hand rehabilitation. However, there is a link between hand function and cognition in the elderly that this technology has not adequately supported. To investigate this issue, this paper proposes a multiscopic CPSS framework by developing hand–object interaction (HOI) based on visual attention. First, we use egocentric vision to extract features from hand posture at the microscopic level. With 94.87% testing accuracy, we use three layers of graph neural network (GNN) based on hand skeletal features to categorize 16 grasp postures. Second, we use a mesoscopic active perception ability to validate the HOI with eye tracking in the task-specific reach-to-grasp cycle. With 90.75% testing accuracy, the distance between the fingertips and the center of an object is used as input to a multi-layer gated recurrent unit based on recurrent neural network architecture. Third, we incorporate visual attention into the cognitive ability for classifying multiple objects at the macroscopic level. In two scenarios with four activities, we use GNN with three convolutional layers to categorize some objects. The outcome demonstrates that the system can successfully separate objects based on related activities. Further research and development are expected to support the CPSS application in independent rehabilitation.

show abstract

Section: Discussion On Microscopic Levelsupporting

confidence: 48%

Section: Discussion On Mesoscopic Levelmentioning

confidence: 99%

Hand–object interaction recognition based on visual attention using multiscopic cyber-physical-social system

Besari¹,

Saputra

Chin

et al. 2023

Int. J. Adv. Intell. Informatics

View full text Add to dashboard Cite

show abstract

“…Kinematic finger models [45] have been widely studied in various research, especially in the industrial sector, such as hand modeling and simulation, hand motion coordination, articulated human hands, prosthetic hands, and robotics. Our previous work [46] used a stereo infrared (SIR) camera from Leap Motion to capture the position of the hand and fingers. This sensor tracks fingers in a 3D zone extending from 10 cm to 75cm and from the device in a 170°x170° ordinary field of view.…”

Section: ) Kinematic Finger Modelmentioning

confidence: 99%

Finger Joint Angle Estimation With Visual Attention for Rehabilitation Support: A Case Study of the Chopsticks Manipulation Test

et al. 2022

Self Cite

View full text Add to dashboard Cite

Most East Asian rehabilitation centers offer chopsticks manipulation tests (CMT). In addition to impaired hand function, approximately two-thirds of stroke survivors have visual impairment related to eye movement. This article investigates the significance of combining finger joint angle estimation and a visual attention measurement in CMT. We present a multiscopic framework that consists of microscopic, mesoscopic, and macroscopic levels. We develop a feature extraction technique to extract the kinematic finger model at the microscopic level. At the mesoscopic level, we propose an active perception ability to detect the position and geometry of the finger on the chopsticks. The proposed framework estimates the proximal interphalangeal (PIP) joint angle on the index finger during CMT using fully connected cascade neural networks (FCC-NN). At the macroscopic level, we implement a cognitive ability by measuring visual attention during CMT. We further evaluate the proposed framework with a conventional test that counts the number of peanuts (NP) which are moved from one bowl to another using chopsticks within a particular time frame. For the evaluation, we introduce three parameters, namely joint angle estimation movement (JAEM), chopstick attention movement (CAM), and chopstick tip movement (CTM), by detecting the local minima and maxima of the signal. According to the experiment results, the velocity of these three parameters could indicate improvement in hand and eye function during CMT. We expect this study to benefit therapists and researchers by providing valuable information that is not accessible in the clinic. Code and datasets are available online at https://github.com/anom-tmu/cmt-attention.INDEX TERMS hand-eye interaction, rehabilitation evaluation, first-person vision, eye-gaze, eye-tracking.

show abstract

Multiscopic CPSS for Independent Block-Design Test Based on Hand–Object Interaction Recognition With Visual Attention

et al. 2023

View full text Add to dashboard Cite

Feature-based Egocentric Grasp Pose Classification for Expanding Human-Object Interactions

Cited by 6 publications

References 22 publications

Hand–object interaction recognition based on visual attention using multiscopic cyber-physical-social system

Hand–object interaction recognition based on visual attention using multiscopic cyber-physical-social system

Finger Joint Angle Estimation With Visual Attention for Rehabilitation Support: A Case Study of the Chopsticks Manipulation Test

Multiscopic CPSS for Independent Block-Design Test Based on Hand–Object Interaction Recognition With Visual Attention

Contact Info

Product

Resources

About