Force Myography-Based Human Robot Interactions via Deep Domain Adaptation and Generalization

Zakia, Umme; Menon, Carlo

doi:10.3390/s22010211

Cited by 7 publications

(6 citation statements)

References 35 publications

(35 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the literature, CNN based GAN models are more dominant and have shown good performances in image classifications and pattern recognition. In our previous studies, we found that the model could effectively learn discriminative features of time-series interactive data [16], [17]. In this study, window size of the FMG-DCGAN model was set to 10ms evaluate the smallest time-series input (every single row) in the system.…”

Section: Discussionmentioning

confidence: 99%

Unsupervised, Semi-Supervised Interactive Force Estimations During pHRI via Generated Synthetic Force Myography Signals

et al. 2022

Self Cite

View full text Add to dashboard Cite

Recognizing applied hand forces using force myography (FMG) biosignals requires adequate training data to facilitate physical human-robot interactions (pHRI). But in practice, data is often scarce, and labels are usually unavailable or time consuming to generate. Synthesizing FMG biosignals can be a viable solution. Therefore, in this paper, we propose for the first time a dual-phased algorithm based on semisupervised adversarial learning utilizing fewer labeled real FMG data with generated unlabeled synthetic FMG data. We conducted a pilot study to test this algorithm in estimating applied forces during interactions with a Kuka robot in 1D-X, Y, Z directions. Initially, an unsupervised FMG-based deep convolutional generative adversarial network (FMG-DCGAN) model was employed to generate real-like synthetic FMG data. A variety of transformation functions were used to observe domain randomization for increasing data variability and for representing authentic physiological, environmental changes. Cosine similarity score and generated-to-input-data ratio were used as decision criteria minimizing the reality gap between real and synthetic data and helped avoid risks associated with wrong predictions. Finally, the FMG-DCGAN model was pretrained to generate pseudo-labels for unlabeled real and synthetic data, further retrained using all labeled and pseudo-labeled data and was termed as the self-trained FMG-DCGAN model. Lastly, this model was evaluated on unseen real test data and achieved accuracies of 85%>R 2 >77% in force estimation compared to the corresponding supervised baseline model (89%>R 2 >78%). Therefore, the proposed method can be more practical for use in FMG-based HRI, rehabilitation, and prosthetic control for daily, repetitive usage even with few labeled data.

show abstract

Section: Discussionmentioning

confidence: 99%

Unsupervised, Semi-Supervised Interactive Force Estimations During pHRI via Generated Synthetic Force Myography Signals

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…Common industrial tasks such as object transportation or handover may require the worker to apply hand forces during interactions with the robot. Estimating the applied hand force to interact with a robot in dynamic motions using FMG signals was found favorable in physical interactions [ 48 , 49 , 50 , 51 , 52 , 53 ]. These studies implemented compliant collaboration, where a robot’s trajectory was dictated by the applied hand force of a human worker.…”

Section: Introductionmentioning

confidence: 99%

Detecting Safety Anomalies in pHRI Activities via Force Myography

Zakia

Menon

2023

Bioengineering

Self Cite

View full text Add to dashboard Cite

The potential application of using a wearable force myography (FMG) band for monitoring the occupational safety of a human participant working in collaboration with an industrial robot was studied. Regular physical human–robot interactions were considered as activities of daily life in pHRI (pHRI-ADL) to recognize human-intended motions during such interactions. The force myography technique was used to read volumetric changes in muscle movements while a human participant interacted with a robot. Data-driven models were used to observe human activities for useful insights. Using three unsupervised learning algorithms, isolation forest, one-class SVM, and Mahalanobis distance, models were trained to determine pHRI-ADL/regular, preset activities by learning the latent features’ distributions. The trained models were evaluated separately to recognize any unwanted interactions that differed from the normal activities, i.e., anomalies that were novel, inliers, or outliers to the normal distributions. The models were able to detect unusual, novel movements during a certain scenario that was considered an unsafe interaction. Once a safety hazard was detected, the control system generated a warning signal within seconds of the event. Hence, this study showed the viability of using FMG biofeedback to indicate risky interactions to prevent injuries, improve occupational health, and monitor safety in workplaces that require human participation.

show abstract

“…In recent sEMG-based human-robot interactions (HRI) studies, deep learning techniques such as convolutional neural networks (CNN) and long-term short memory (LSTM) were used for dynamic or static force estimations [7][8][9]. A few studies were conducted recently where FMG bio-signals were used for applied force estimations during physical human-robot interaction (pHRI) activities [10][11][12]. In [10], pHRI between several participants and a fixed linear robot was investigated using intra-session data.…”

Section: Introductionmentioning

confidence: 99%

“…Such an individual-specific, intra-session biased model predicted interactive forces (94%>R 2 >82%) in real-time during the same session. Interestingly, pHRI with inter-session FMG data using domain adaptation and generalization in a planar workspace was recently investigated with improved force estimations [11,12]. In these 2D-pHRI studies, a generalized model trained with long-term FMG distributions (collected over a period) predicted unseen target data during repetitive usage or during interactions with a new participant.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Human–Robot Collaboration in 3D via Force Myography Based Interactive Force Estimations Using Cross-Domain Generalization

Zakia

Menon

2022

IEEE Access

Self Cite

View full text Add to dashboard Cite

In this study, human robot collaboration (HRC) via force myography (FMG) bio-signal was investigated. Interactive hand force was estimated during moving a wooden rod in 3D with a Kuka robot. A baseline FMG-based deep convolutional neural network (FMG-DCNN) model could moderately estimate applied forces during the HRC task. Model performance can be improved with additional training data; however, collection of it was impractical and time-consuming. Available long-term multiple source data (32 feature spaces) during human robot interaction (HRI) with a linear robot collected over a long time period might be useful. Therefore, we explored a cross-domain generalization (CDG) technique that allowed pretraining a model to transfer knowledge between two unrelated source (2D-HRI) and target data (3D-HRC) for the first time. An FMG-based transfer learning with CDG (TL-CDG) model trained with these multiple source domains was examined in estimating applied forces from 16-channel FMG data during interactions with the Kuka robot. Two target scenarios were evaluated: case i) collaborative task of moving the wooden rod in 3D, and case ii) grasping interactions in 1D. In both cases, few calibration data finetuned the TL-CDG model and improved recognizing out-of-domain target data (case i: R 2 ≈60-63%, and case ii: R 2 ≈79-87%) compared to the baseline FMG-DCNN model. Hence, cross-domain generalization could be useful in platform-independent FMG-based HRI applications.

show abstract

Force Myography-Based Human Robot Interactions via Deep Domain Adaptation and Generalization

Cited by 7 publications

References 35 publications

Unsupervised, Semi-Supervised Interactive Force Estimations During pHRI via Generated Synthetic Force Myography Signals

Unsupervised, Semi-Supervised Interactive Force Estimations During pHRI via Generated Synthetic Force Myography Signals

Detecting Safety Anomalies in pHRI Activities via Force Myography

Human–Robot Collaboration in 3D via Force Myography Based Interactive Force Estimations Using Cross-Domain Generalization

Contact Info

Product

Resources

About