Hierarchical Graphical Models for Context-Aware Hybrid Brain-Machine Interfaces

Özdenizci, Ozan; Gunay, Sezen Yagmur; Quivira, Fernando; Erdogmug, Deniz

doi:10.1109/embc.2018.8512677

Cited by 4 publications

(7 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While EEG alone may not be capable of discriminating a large number of intended grasp types, supplementing EMG evidence with EEG may lead to improved intent inference in the case of some amputations. Ozdenizci et al [19] demonstrated that five particular hand gestures (four of them grasp types) can be reliably inferred from EEG, and they provide a reasonable initial coverage of the grasp taxonomy variety established by Feix et al [5]. Specifically, these five grasp types, shown in Fig.…”

Section: Data Annotationmentioning

confidence: 89%

“…The quality of EMG signals may dramatically vary across individuals depending on the specifics of their amputation. Some studies also investigated the possibility of complementing EMG with electroencephalography (EEG) signals to improve inference accuracy [2,9,15,19]; however, in many practical situations the added information from EEG does not have significant impact on performance. Both EMG and EEG models may need frequent calibration to account for signal nonstationarity due to various factors, such as electrode locations or skin conductance.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

HANDS: a multimodal dataset for modeling toward human grasp intent inference in prosthetic hands

Han

Gunay

Schirner

et al. 2019

Intel Serv Robotics

View full text Add to dashboard Cite

Upper limb and hand functionality is critical to many activities of daily living, and the amputation of one can lead to significant functionality loss for individuals. From this perspective, advanced prosthetic hands of the future are anticipated to benefit from improved shared control between a robotic hand and its human user, but more importantly from the improved capability to infer human intent from multimodal sensor data to provide the robotic hand perception abilities regarding the operational context. Such multimodal sensor data may include various environment sensors including vision, as well as human physiology and behavior sensors including electromyography and inertial measurement units. A fusion methodology for environmental state and human intent estimation can combine these sources of evidence in order to help prosthetic hand motion planning and control. In this paper, we present a dataset of this type that was gathered with the anticipation of cameras being built into prosthetic hands, and computer vision methods will need to assess this hand-view visual evidence in order to estimate human intent. Specifically, paired images from human eye-view and hand-view of various objects placed at different orientations have been captured at the initial state of grasping trials, followed by paired video, EMG and IMU from the arm of the human during a grasp, lift, put-down, and retract style trial structure. For each trial, based on eye-view images of the scene showing the hand and object on a table, multiple humans were asked to sort in decreasing order of preference, five grasp types appropriate for the object in its given configuration relative to the hand. The potential utility of paired eye-view and hand-view images was illustrated by training a convolutional neural network to process hand-view images in order to predict eye-view labels assigned by humans. KeywordsMultimodal dataset • Human grasp intent classification • Prosthetic hand • Eye and hand-view images • EMG • Convolutional neural network

show abstract

Section: Data Annotationmentioning

confidence: 89%

Section: Introductionmentioning

confidence: 99%

HANDS: a multimodal dataset for modeling toward human grasp intent inference in prosthetic hands

Han

Gunay

Schirner

et al. 2019

Intel Serv Robotics

View full text Add to dashboard Cite

show abstract

“…In the context of BCIs, we argue that hierarchical arrangement of one-versus-rest binary sub-problems can be represented by an intuitive ordering rather than an arbitrary one. For instance in hand gesture decoding, upper hierarchical levels can discriminate the choice of hand and palm opening, whereas lower levels will be decoding power versus precision grasp type, or thumb abduction versus adduction of a specific grasp [39]. This decomposition provides an application specific multi-class decoding scheme which we demonstrate in Section IV.…”

Section: A Hierarchical Graphical Modelmentioning

confidence: 99%

Information Theoretic Feature Transformation Learning for Brain Interfaces

Özdenizci

Erdoğmuş

2020

IEEE Trans. Biomed. Eng.

Self Cite

View full text Add to dashboard Cite

Objective: A variety of pattern analysis techniques for model training in brain interfaces exploit neural feature dimensionality reduction based on feature ranking and selection heuristics. In the light of broad evidence demonstrating the potential sub-optimality of ranking based feature selection by any criterion, we propose to extend this focus with an information theoretic learning driven feature transformation concept. Methods:We present a maximum mutual information linear transformation (MMI-LinT), and a nonlinear transformation (MMI-NonLinT) framework derived by a general definition of the feature transformation learning problem. Empirical assessments are performed based on electroencephalographic (EEG) data recorded during a four class motor imagery brain-computer interface (BCI) task. Exploiting state-of-the-art methods for initial feature vector construction, we compare the proposed approaches with conventional feature selection based dimensionality reduction techniques which are widely used in brain interfaces. Furthermore, for the multi-class problem, we present and exploit a hierarchical graphical model based BCI decoding system.Results: Both binary and multi-class decoding analyses demonstrate significantly better performances with the proposed methods. Conclusion:Information theoretic feature transformations are capable of tackling potential confounders of conventional approaches in various settings.Significance: We argue that this concept provides significant insights to extend the focus on feature selection heuristics to a broader definition of feature transformation learning in brain interfaces.

show abstract

“…As proved in [9], some of the grasp tasks are very similar with the others and most of them could be handled successfully by several ones of all the 33 gestures. Moreover, it has been shown in [22] that, EEG signals are only capable of classifying 5 specic grasp types. Thus, for the future fusion with the EEG control, the label set is limited to 5 gestures: Open Palm, Medium Wrap, Power Sphere, Parallel Extension and Palmar Pinch, as shown in Fig.…”

Section: Set-up and Collectionmentioning

confidence: 99%

“…The newest myoelectrical activity-based designs of the prosthetic hand demonstrate promising results for patients with hand and wrist amputation, but the quality of the muscle activity signals decreases dramatically as amputation severity increases. Recent evidence indicates that decient electromyogram (EMG) activity could be compensated by electroencephalogram (EEG) signals [1,14,19,22], but because of the lower signal-to-noise ratio of EEG data, the results are still not sucient for real-world problems. Additionally, frequent calibration of the system is required to account for sensor sensitivity to external factors, such as electrode locations and skin variabilities.…”

Section: Introductionmentioning

confidence: 99%

From hand-perspective visual information to grasp type probabilities

Han¹,

Gunay²,

Yıldız³

et al. 2019

Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments

View full text Add to dashboard Cite

Limb deciency severely aects the daily lives of amputees and drives eorts to provide functional robotic prosthetic hands to compensate this deprivation. Convolutional neural network-based computer vision control of the prosthetic hand has received increased attention as a method to replace or complement physiological signals due to its reliability by training visual information to predict the hand gesture. Mounting a camera into the palm of a prosthetic hand is proved to be a promising approach to collect visual data. However, the grasp type labelled from the eye and hand perspective may dier as object shapes are not always symmetric. Thus, to represent this dierence in a realistic way, we employed a dataset containing synchronous images from eye-and hand-view, where the hand-perspective images are used for training while the eyeview images are only for manual labelling. Electromyogram (EMG) activity and movement kinematics data from the upper arm are also collected for multi-modal information fusion in future work. Moreover, in order to include human-in-the-loop control and combine the computer vision with physiological signal inputs, instead of making absolute positive or negative predictions, we build a novel probabilistic classier according to the Plackett-Luce model. To predict the probability distribution over grasps, we exploit the statistical model over label rankings to solve the permutation domain problems via a maximum likelihood estimation, utilizing the manually ranked lists of grasps as a new form of label. We indicate that the proposed model is applicable to the most popular and productive convolutional neural network frameworks.

show abstract

Hierarchical Graphical Models for Context-Aware Hybrid Brain-Machine Interfaces

Cited by 4 publications

References 17 publications

HANDS: a multimodal dataset for modeling toward human grasp intent inference in prosthetic hands

HANDS: a multimodal dataset for modeling toward human grasp intent inference in prosthetic hands

Information Theoretic Feature Transformation Learning for Brain Interfaces

From hand-perspective visual information to grasp type probabilities

Contact Info

Product

Resources

About