Acting thoughts: Towards a mobile robotic service assistant for users with limited communication skills

Burget, Felix; Fiederer, Lukas Dominique Josef; Kuhner, Daniel; Volker, Martin A.; Aldinger, Johannes; Schirrmeister, Robin Tibor; Do, Chau; Boedecker, Joschka; Nebel, Bernhard; Ball, Tonio; Burgard, Wolfram

doi:10.1109/ecmr.2017.8098658

Cited by 22 publications

(24 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In [18] using a Cybathlon complying BCI based in CSP-LDA, the performance between offline analysis and first online session dropped from 79.4% to 51.4% (35.3% drop). A similar drop exist between [7] and its online implementation by [9], from 84% to 76.7% (8.7% drop). However, in this case, the online implementation added one category to the offline implementation and the drop could be explained by the extra class as well.…”

Section: Discussionsupporting

confidence: 55%

Compact Convolutional Neural Networks for Multi-Class, Personalised, Closed-Loop EEG-BCI

Ortega

Colas

Faisal

2018

2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob)

View full text Add to dashboard Cite

For many people suffering from motor disabilities, assistive devices controlled with only brain activity are the only way to interact with their environment [1]. Natural tasks often require different kinds of interactions, involving different controllers the user should be able to select in a self-paced way. We developed a Brain-Computer Interface (BCI) allowing users to switch between four control modes in a self-paced way in real-time. Since the system is devised to be used in domestic environments in a user-friendly way, we selected non-invasive electroencephalographic (EEG) signals and convolutional neural networks (CNNs), known for their ability to find the optimal features in classification tasks. We tested our system using the Cybathlon BCI computer game, which embodies all the challenges inherent to real-time control. Our preliminary results show that an efficient architecture (SmallNet), with only one convolutional layer, can classify 4 mental activities chosen by the user. The BCI system is run and validated online. It is kept up-to-date through the use of newly collected signals along playing, reaching an online accuracy of 47.6% where most approaches only report results obtained offline. We found that models trained with data collected online better predicted the behaviour of the system in real-time. This suggests that similar (CNN based) offline classifying methods found in the literature might experience a drop in performance when applied online. Compared to our previous decoder of physiological signals relying on blinks, we increased by a factor 2 the amount of states among which the user can transit, bringing the opportunity for finer control of specific subtasks composing natural grasping in a self-paced way. Our results are comparable to those showed at the Cybathlon's BCI Race but further improvements on accuracy are required.

show abstract

Section: Discussionsupporting

confidence: 55%

Compact Convolutional Neural Networks for Multi-Class, Personalised, Closed-Loop EEG-BCI

Ortega

Colas

Faisal

2018

2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob)

View full text Add to dashboard Cite

show abstract

“…The possibility for evaluation of different deep learning techniques that is large-scale both with respect to the number of network models and datasets included, could be helpful for an informed initial choice of a suitable deep learning architecture also for novel EEG decoding problems. Such an informed initial choice would be especially important when considering deep learning for online BCIs, as in [7], where most of the data is acquired after having fixed the decoder architecture. Thus, in summary, the deep learning EEG framework as described here could help to tap the full potential of deep learning for BCI applications.…”

Section: Discussionmentioning

confidence: 99%

“…and excels at decoding information from raw data, e.g., in image recognition [1] and natural language processing [2]. Thus, currently many BCI researchers are starting to investigate the potential usefulness of deep learning techniques using a wide range of different network architectures and applying them to a wide range of EEG datasets [3]- [7].…”

Section: Introductionmentioning

confidence: 99%

A Large-Scale Evaluation Framework for EEG Deep Learning Architectures

Heilmeyer

Schirrmeister

Fiederer

et al. 2018

2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC)

View full text Add to dashboard Cite

EEG is the most common signal source for noninvasive BCI applications. For such applications, the EEG signal needs to be decoded and translated into appropriate actions. A recently emerging EEG decoding approach is deep learning with Convolutional or Recurrent Neural Networks (CNNs, RNNs) with many different architectures already published. Here we present a novel framework for the large-scale evaluation of different deep-learning architectures on different EEG datasets. This framework comprises (i) a collection of EEG datasets currently including 100 examples (recording sessions) from six different classification problems, (ii) a collection of different EEG decoding algorithms, and (iii) a wrapper linking the decoders to the data as well as handling structured documentation of all settings and (hyper-) parameters and statistics, designed to ensure transparency and reproducibility. As an applications example we used our framework by comparing three publicly available CNN architectures: the Braindecode Deep4 ConvNet, Braindecode Shallow ConvNet, and two versions of EEGNet. We also show how our framework can be used to study similarities and differences in the performance of different decoding methods across tasks. We argue that the deep learning EEG framework as described here could help to tap the full potential of deep learning for BCI applications.

show abstract

“…A first attempt at using shallow ConvNets for online BCI has recently been reported [22]. To the best of our knowledge, apart from our previous paper [23], there is no other work, which uses a deep ConvNet-based online control to implement an EEG-based brain-computer interface.…”

Section: Related Workmentioning

confidence: 99%

Deep Learning Based BCI Control of a Robotic Service Assistant Using Intelligent Goal Formulation

Kuhner

Fiederer

Aldinger

et al. 2018

Preprint

Self Cite

View full text Add to dashboard Cite

As autonomous service robots become more affordable and thus available for the general public, there is a growing need for user-friendly interfaces to control these systems. Control interfaces typically get more complicated with increasing complexity of the robotic tasks and the environment. Traditional control modalities as touch, speech or gesture commands are not necessarily suited for all users. While non-expert users can make the effort to familiarize themselves with a robotic system, paralyzed users may not be capable of controlling such systems even though they need robotic assistance most. In this paper, we present a novel framework, that allows these users to interact with a robotic service assistant in a closed-loop fashion, using only thoughts. The system is composed of several interacting components: non-invasive neuronal signal recording and co-adaptive deep learning which form the brain-computer interface (BCI), high-level task planning based on referring expressions, navigation and manipulation planning as well as environmental perception. We extensively evaluate the BCI in various tasks, determine the performance of the goal formulation user interface and investigate its intuitiveness in a user study. Furthermore, we demonstrate the applicability and robustness of the system in real world scenarios, considering fetch-and-carry tasks and tasks involving human-robot interaction. As our results show, the system is capable of adapting to frequent changes in the environment and reliably accomplishes given tasks within a reasonable amount of time. Combined with high-level planning using referring expressions and autonomous robotic systems, interesting new perspectives open up for non-invasive BCI-based human-robot interactions.

show abstract

Acting thoughts: Towards a mobile robotic service assistant for users with limited communication skills

Cited by 22 publications

References 29 publications

Compact Convolutional Neural Networks for Multi-Class, Personalised, Closed-Loop EEG-BCI

Compact Convolutional Neural Networks for Multi-Class, Personalised, Closed-Loop EEG-BCI

A Large-Scale Evaluation Framework for EEG Deep Learning Architectures

Deep Learning Based BCI Control of a Robotic Service Assistant Using Intelligent Goal Formulation

Contact Info

Product

Resources

About