A closed-loop brain-computer interface with augmented reality feedback for industrial human-robot collaboration

Abstract

Industrial human-robot collaboration (HRC) aims to combine the human intelligence and robotic capability to achieve the higher productiveness. In industrial HRC, the communication between human and robot is essential to enhance the understanding the intent of each other to make a more fluent collaboration. Brain-computer interface (BCI) is a technology that could record the user's brain activity that can be translated into interaction messages (e.g., control commands) to the outside world, which is able to build a direct and efficient communication channel between human and robot. However, due to lacking of information feedback mechanism, it is challenging for BCI to control robots with high degree of freedom with the limited number of classifiable mental state.To address this problem, this paper proposes a close-loop BCI with contextual visual feedback by an augmented reality (AR) headset. In such BCI, the electroencephalogram (EEG) patterns from the multiple voluntary eye blinks are considered as the input and the its online detection algorithm is proposed whose average accuracy can reach 94.31%. Moreover, an AR-enable information feedback interface is designed which enable to achieve an interactive robotic path planning. A case study of an industrial HRC assembly task is also develop which shows that the proposed closed-up BCI could shorten the time of user input in human-robot interaction.

A Comprehensive Review of Endogenous EEG-Based BCIs for Dynamic Device Control

“…Furthermore, users had to make an effort to self-induce emotions despite their current mood, which may be challenging in everyday life. Self-inducing emotion may also be more mentally demanding than other paradigms, such as facial movement, which use relatively straightforward actions such as blinks [ 58 , 59 ].…”

“…Facial-movement paradigms have been touted as a viable alternative to other endogenous paradigms such as MI because they have much lower BCI illiteracy rates [ 59 ]. This is because facial movements such as blinks and eyebrow or mouth movements produce distinct signal characteristics in EEG data that can be detected with a relatively high accuracy [ 58 , 59 ]. Sequential facial-movement paradigms have also been used to augment the number of commands executed using a small number of facial actions.…”

“…In total, 23 studies used time-domain modeling features (such as autoregressive features [ 48 ]) or multichannel time series data for classification [ 58 ], whereas 17 used frequency [ 17 , 72 ] or time-frequency domain features [ 65 ]. Time-series data are sometimes directly used for thresholding classification, which will be discussed in detail in the next subsection [ 58 , 81 ].…”

“…In total, 23 studies used time-domain modeling features (such as autoregressive features [ 48 ]) or multichannel time series data for classification [ 58 ], whereas 17 used frequency [ 17 , 72 ] or time-frequency domain features [ 65 ]. Time-series data are sometimes directly used for thresholding classification, which will be discussed in detail in the next subsection [ 58 , 81 ]. In real-time systems, low execution times are essential, and past research has indicated that time-domain features can have a similar performance to frequency-domain features, with the added benefit of a lower computational complexity [ 82 ].…”

“…Thresholding-based classifiers were used for paradigms that had highly distinctive EEG patterns [ 24 , 58 ]. Blinks, for example, could be easily identified as relatively large-amplitude spikes in the signal, and an amplitude threshold on the time-series data could be used to identify them [ 58 ].…”