Multimodal Framework for Fine and Gross Upper-Limb Motor Coordination Assessment Using Serious Games and Robotics

Abstract

A critical element of neurological function is eye–hand coordination: the ability of our vision system to coordinate the information received through the eyes to control, guide, and direct the hands to accomplish a task. Recent evidence shows that this ability can be disturbed by strokes or other neurological disorders, with critical consequences for motor behaviour. This paper presents a system based on serious games and multimodal devices aimed at improving the assessment of eye–hand coordination. The system implements gameplay that involves drawing specific patterns (labyrinths) to capture hand trajectories. The user can draw the path using multimodal devices such as a mouse, a stylus with a tablet, or robotic devices. Multimodal input devices can allow for the evaluation of complex coordinated movements of the upper limb that involve the synergistic motion of arm joints, depending on the device. A preliminary test of technological validation with healthy volunteers was conducted in the laboratory. The Dynamic Time Warping (DTW) index was used to compare hand trajectories without considering time-series lag. The results suggest that this multimodal framework allows for measuring differences between fine and gross motor skills. Moreover, the results support the viability of this system for developing a high-resolution metric for measuring eye–hand coordination in neurorehabilitation.