The Imaginative Generative Adversarial Network: Automatic Data Augmentation for Dynamic Skeleton-Based Hand Gesture and Human Action Recognition

Shen, Junxiao; Dudley, John J.; Kristensson, Per Ola

doi:10.1109/fg52635.2021.9666999

Cited by 16 publications

(10 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Synthetic data may be generated in large quantities to mimic actual data for training 41,42 , but may not be sufficiently realistic or diverse for generalizable learning 43 . Data augmentation is a widely used approach to training ML on slightly altered versions of the same input data to increase the size of the training set 44 , but does not capture variations in larger real data 45 .…”

Section: Discussionmentioning

confidence: 99%

Novel Domain Knowledge Encoding Enables Machine Learning of Rapid, Expert-level Segmentation of Cardiac Computed Tomography

Feng

Deb

Ganesan

et al. 2022

Preprint

View full text Add to dashboard Cite

Ablation is a common therapeutic procedure for atrial fibrillation (AF), that is guided to specific targets in the heart often after laborious segmentation of computed tomography. Machine learning (ML) can automate such tasks but requires large training datasets. Inspired by natural intelligence, which builds conceptual models to learn without large datasets, we mathematically encoded domain knowledge of atrial geometry to accelerate ML segmentation. In test cohorts (N=160) at 2 institutions, Dice scores were 96.7% (IQR: 95.3% to 97.7%) and 93.5% (IQR: 91.9% to 94.7%) with similar anatomic agreement to experts (r=0.99; p<0.0001). In a prospective study of patients undergoing AF ablation (N=42), our approach reduced segmentation time by 85% (2.28±0.8 vs 15.0±6.9 minutes; p<0.0001), with similar Dice (p=0.07) versus experts. This approach may broaden the availability of AF ablation, and more broadly shows that encoding of domain knowledge may reduce the dependence of ML on large training datasets.

show abstract

Section: Discussionmentioning

confidence: 99%

Novel Domain Knowledge Encoding Enables Machine Learning of Rapid, Expert-level Segmentation of Cardiac Computed Tomography

Feng

Deb

Ganesan

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…This strategy did not utilize other datasets, and the best model needed to be selected from the candidate model list, which needed to be more complex for application. Shen et al [ 60 ] presented an automatic data augmentation model called Imaginative Generative Adversarial Network that can sample new data from learned dataset distribution. Augmented datasets can improve the classification accuracy with the same neural network.…”

Section: Related Workmentioning

confidence: 99%

“…However, in the downstream tasks of small-scale datasets, the pre-training strategy is an effective way to improve recognition performance [ 73 ]. Therefore, our model framework based on the pre-training strategy is necessary for small-scale action recognition tasks, which has never been addressed in previous work [ 6 , 57 , 59 , 60 ]. We also give a complete analysis with intuitional t-SNE [ 14 ] visualization to demonstrate the effectiveness of the pre-training strategy.…”

Section: Related Workmentioning

confidence: 99%

Improving Small-Scale Human Action Recognition Performance Using a 3D Heatmap Volume

Yuan

Wang

et al. 2023

Sensors

View full text Add to dashboard Cite

In recent years, skeleton-based human action recognition has garnered significant research attention, with proposed recognition or segmentation methods typically validated on large-scale coarse-grained action datasets. However, there remains a lack of research on the recognition of small-scale fine-grained human actions using deep learning methods, which have greater practical significance. To address this gap, we propose a novel approach based on heatmap-based pseudo videos and a unified, general model applicable to all modality datasets. Leveraging anthropometric kinematics as prior information, we extract common human motion features among datasets through an ad hoc pre-trained model. To overcome joint mismatch issues, we partition the human skeleton into five parts, a simple yet effective technique for information sharing. Our approach is evaluated on two datasets, including the public Nursing Activities and our self-built Tai Chi Action dataset. Results from linear evaluation protocol and fine-tuned evaluation demonstrate that our pre-trained model effectively captures common motion features among human actions and achieves steady and precise accuracy across all training settings, while mitigating network overfitting. Notably, our model outperforms state-of-the-art models in recognition accuracy when fusing joint and limb modality features along the channel dimension.

show abstract

“…to generate realistic samples in image [12,43,50], speech [19], and motion trajectory [41] synthesis. We adopted the Imaginative GAN from Shen et al [40] as one of our data augmentation strategies, which can approximate the true distribution of the input data and sample new data from the approximated distribution. The goal of the Imaginative GAN is to learn the latent attributes, such as behavioral attributes (for example, the speed of performing the actions/gestures) and physical attributes (for example, hand sizes).…”

Section: Augmentationmentioning

confidence: 99%

Gesture Spotter: A Rapid Prototyping Tool for Key Gesture Spotting in Virtual and Augmented Reality Applications

Shen

Dudley

et al. 2022

IEEE Trans. Visual. Comput. Graphics

View full text Add to dashboard Cite

In this paper we examine the task of key gesture spotting: accurate and timely online recognition of hand gestures. We specifically seek to address two key challenges faced by developers when integrating key gesture spotting functionality into their applications. These are: i) achieving high accuracy and zero or negative activation lag with single-time activation; and ii) avoiding the requirement for deep domain expertise in machine learning. We address the first challenge by proposing a key gesture spotting architecture consisting of a novel gesture classifier model and a novel single-time activation algorithm. This key gesture spotting architecture was evaluated on four separate hand skeleton gesture datasets, and achieved high recognition accuracy with early detection. We address the second challenge by encapsulating different data processing and augmentation strategies, as well as the proposed key gesture spotting architecture, into a graphical user interface and an application programming interface. Two user studies demonstrate that developers are able to efficiently construct custom recognizers using both the graphical user interface and the application programming interface.

show abstract

The Imaginative Generative Adversarial Network: Automatic Data Augmentation for Dynamic Skeleton-Based Hand Gesture and Human Action Recognition

Cited by 16 publications

References 27 publications

Novel Domain Knowledge Encoding Enables Machine Learning of Rapid, Expert-level Segmentation of Cardiac Computed Tomography

Novel Domain Knowledge Encoding Enables Machine Learning of Rapid, Expert-level Segmentation of Cardiac Computed Tomography

Improving Small-Scale Human Action Recognition Performance Using a 3D Heatmap Volume

Gesture Spotter: A Rapid Prototyping Tool for Key Gesture Spotting in Virtual and Augmented Reality Applications

Contact Info

Product

Resources

About