Predicting Targets of Human Reaching Motions Using Different Sensing Technologies

Abstract

Rapid recognition of voluntary motions is crucial in human-computer interaction, but few studies compare the predictive abilities of different sensing technologies. This paper thus compares performances of different technologies when predicting targets of human reaching motions: electroencephalography (EEG), electrooculography, camera-based eye tracking, electromyography (EMG), hand position, and the user's preferences. Supervised machine learning is used to make predictions at different points in time (before and during limb motion) with each individual sensing modality. Different modalities are then combined using an algorithm that takes into account the different times at which modalities provide useful information. Results show that EEG can make predictions before limb motion onset, but requires subject-specific training and exhibits decreased performance as the number of possible targets increases. EMG and hand position give high accuracy, but only once the motion has begun. Eye tracking is robust and exhibits high accuracy at the very onset of limb motion. Several advantages of combining different modalities are also shown, including advantages of combining measurements with contextual data. Finally, some recommendations are given for sensing modalities with regard to different criteria and applications. The information could aid human-computer interaction designers in selecting and evaluating appropriate equipment for their applications.

Eye tracking for locomotion prediction in redirected walking

“…It was shown in consumer and cognitive science research that a person's gaze can be highly predictive of future actions in selection tasks. Various researchers found that if people are asked to make a decision between a number of options, there is a significant gaze bias towards the chosen option for a short time period before the decision is announced [3,17,25,26]. It is also known that the gaze direction leads the walking direction by a few seconds [6] and the same holds true for driving [8,12].…”

Multimodal Movement Prediction - Towards an Individual Assistance of Patients

“…This can be achieved by adapting the control of the device with respect to the patient’s intention. Movement intention of the patient can be detected from her/his brain activity, e.g., the electroencephalogram (EEG), as shown in healthy subjects [8], [10]–[13] as well as in stroke patients [14], and by the analysis of gaze direction and fixation [15], or by the analysis of the electromyogram (EMG) [16]. EMG activity is quite often solely used to trigger an orthosis or a prosthesis [17]–[19].…”

“…Assistive technology devices that are supported by integrated analysis of physiological or technical data to enable the detection of movement intention can support a patient for self-initiated movements. By analyzing the context of behavior even complex interaction, like grasping a certain object [15], can be triggered and executed by the device (Figure 1). What sources of physiological data should be combined depends on the requirements, e.g., the kind of disability and neuromuscular disorder [25] as well as the state and progress of the patient in rehabilitation.…”

“…What sources of physiological data should be combined depends on the requirements, e.g., the kind of disability and neuromuscular disorder [25] as well as the state and progress of the patient in rehabilitation. Further, the correctness of a prediction can in principle be improved by combining several sources of physiological data or even other information, like preferences of the patient [15], [23], [25]. Moreover, it is important to evaluate what effect a combination has on the control of the device.…”

Multimodal decoding and congruent sensory information enhance reaching performance in subjects with cervical spinal cord injury

“…Brain-machine interfaces (BMIs) have the potential to provide more natural control (Collinger et al, 2012; Ethier et al, 2012; Hochberg et al, 2012), although most BMIs that have successfully controlled reach involve invasive cortical recordings, a technology that is currently inaccessible in most clinical situations. Combining information from disparate sources has been proposed as a solution when there are few signals accessible (Batista et al, 2008; Pfurtscheller et al, 2010; Leeb et al, 2011; Corbett et al, 2013a; Novak et al, 2013; Kirchner et al, 2014). As the set of usable signals from each individual may be different, it is important to be able take advantage of all the useful channels available.…”