On the Use of Brain Decoded Signals for Online User Adaptive Gesture Recognition Systems

Abstract

Activity and context recognition in pervasive and wearable computing ought to continuously adapt to changes typical of open-ended scenarios, such as changing users, sensor characteristics, user expectations, or user motor patterns due to learning or aging. System performance inherently relates to the user's perception of the system behavior. Thus, the user should be guiding the adaptation process. This should be automatic, transparent, and unconscious. We capitalize on advances in electroencephalography (EEG) signal processing that allow for error related potentials (ErrP) recognition. ErrP are emitted when a human observes an unexpected behavior in a system. We propose and evaluate a hand gesture recognition system from wearable motion sensors that adapts online by taking advantage of ErrP. Thus the gesture recognition system becomes self-aware of its performance, and can self-improve through re-occurring detection of ErrP signals. Results show that our adaptation technique can improve the accuracy of a user independent gesture recognition system by 9.58% when ErrP recognition is perfect. When ErrP recognition errors are factored in, recognition accuracy increases by 3.29%. We characterize the boundary conditions of ErrP recognition guaranteeing beneficial adaptation. The adaptive algorithms are applicable to other forms of activity recognition, and can also use explicit user feedback rather than ErrP.

Collecting complex activity datasets in highly rich networked sensor environments

“…We showed principles that allows one sensor node to autonomously learn how to recognize user activity from another one, thus allowing an activity recognition system to autonomously expand to new resources discovered or introduced in the environment, without user intervention [23]. We showed that adaptive methods can lead to an autonomous system capable of self-improvement, by using minimalist or even unconscious user feedback [24].…”

Adaptation of hybrid human-computer interaction systems using EEG error-related potentials

“…This data was used offline to assess EEG-based adaptation of gesture recognition systems. More details about the experimental setup can be found in [8] B. Experimental protocol Seven healthy male subjects aged 25 to 47 took part in the the experiment.…”

“…In particular, we assume that a subject independent gesture classifier has been previously obtained and the EEG decoding signals will be used to adapt this classifier to a specific new user. Accordingly, trials that not classified as errors would signal that the last gesture was correctly recognized, it can thus be used as an example to further train the current classifier in a supervised manner [8].…”

Autonomous construction using scarce resources in unknown environments

“…One way to detect a fault in action is to compare the sensor values resulting from the action with those expected after a success. Experimental evidence has shown that humans employ this strategy to detect failure of predicted actions (Förster et al 2010). Another way is to build compliant behaviours, as we did for grasping resources (see Sect.…”