Atrous Temporal Convolutional Network for Video Action Segmentation

Wang, Jiahao; Du, Zhengyin; Li, Annan; Wang, Yunlong

doi:10.1109/icip.2019.8803088

Cited by 8 publications

(6 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Built on a similar encoder-decoder backbone, more recent solutions rely instead on two-stream processing: one stream to capture multiscale contextual information and action dependencies in the input sequence, the other to extract lowlevel information for precise action boundary identification [47,48]. Basic temporal convolutions were replaced with a set of Deformable Temporal Residual Modules (TDRM) to capture the temporal variability of human actions and merge the two streams at multiple processing levels [47].…”

Section: A Convolutional Neural Networkmentioning

confidence: 99%

“…Basic temporal convolutions were replaced with a set of Deformable Temporal Residual Modules (TDRM) to capture the temporal variability of human actions and merge the two streams at multiple processing levels [47]. Alternatively, atrous (aka dilated) temporal convolutions and pyramid pooling were used to explicitly generate multi-scale encoding of the input data, that were merged with the lowlevel local features just before the decoding phase [48]. Both architectures were able to improve consistently both framewise and segmental evaluation scores (see Section VII for a description and explanation of the most common evaluation metrics) upon the competing methods.…”

Section: A Convolutional Neural Networkmentioning

confidence: 99%

See 1 more Smart Citation

Gesture Recognition in Robotic Surgery: A Review

Amsterdam

Clarkson²,

Stoyanov³

2021

IEEE Trans. Biomed. Eng.

View full text Add to dashboard Cite

Surgical activity recognition is a fundamental step in computer-assisted interventions. This paper reviews the state-of-the-art in methods for automatic recognition of fine-grained gestures in robotic surgery focusing on recent data-driven approaches and outlines the open questions and future research directions. Methods: An article search was performed on 5 bibliographic databases with combinations of the following search terms: robotic, robot-assisted, JIGSAWS, surgery, surgical, gesture, fine-grained, surgeme, action, trajectory, segmentation, recognition, parsing. Selected articles were classified based on the level of supervision required for training and divided into different groups representing major frameworks for time series analysis and data modelling. Results: A total of 52 articles were reviewed. The research field is showing rapid expansion, with the majority of articles published in the last 4 years. Deep-learning-based temporal models with discriminative feature extraction and multi-modal data integration have demonstrated promising results on small surgical datasets. Currently, unsupervised methods perform significantly less well than the supervised approaches. Conclusion: The development of large and diverse open-source datasets of annotated demonstrations is essential for development and validation of robust solutions for surgical gesture recognition. While new strategies for discriminative feature extraction and knowledge transfer, or unsupervised and semi-supervised approaches, can mitigate the need for data and labels, they have not yet been demonstrated to achieve comparable performance. Important future research directions include detection and forecast of gesture-specific errors and anomalies. Significance: This paper is a comprehensive and structured analysis of surgical gesture recognition methods aiming to summarize the status of this rapidly evolving field.

show abstract

Section: A Convolutional Neural Networkmentioning

confidence: 99%

Section: A Convolutional Neural Networkmentioning

confidence: 99%

Gesture Recognition in Robotic Surgery: A Review

Amsterdam

Clarkson²,

Stoyanov³

2021

IEEE Trans. Biomed. Eng.

View full text Add to dashboard Cite

show abstract

“…For example, the variations in facial expressions can only be exploited by modeling a sequence of consecutive facial images. Recent studies have demonstrated that temporal convolutional networks (TCN) are of great competitiveness in plenty of sequence modeling tasks including action segmentation [18,29,56], natural language processing [3,13] and so on. Compared with recurrent neural network (RNN) architectures, TCN has lower memory requirement and less training difficulty [3].…”

Section: Appearance-based Predictionmentioning

confidence: 99%

Will You Ever Become Popular? Learning to Predict Virality of Dance Clips

Wang

Weng

et al. 2022

ACM Trans. Multimedia Comput. Commun. Appl.

Self Cite

View full text Add to dashboard Cite

Dance challenges are going viral in video communities like TikTok nowadays. Once a challenge becomes popular, thousands of short-form videos will be uploaded within a couple of days. Therefore, virality prediction from dance challenges is of great commercial value and has a wide range of applications, such as smart recommendation and popularity promotion. In this article, a novel multi-modal framework that integrates skeletal, holistic appearance, facial and scenic cues is proposed for comprehensive dance virality prediction. To model body movements, we propose a pyramidal skeleton graph convolutional network (PSGCN) that hierarchically refines spatio-temporal skeleton graphs. Meanwhile, we introduce a relational temporal convolutional network (RTCN) to exploit appearance dynamics with non-local temporal relations. An attentive fusion approach is finally proposed to adaptively aggregate predictions from different modalities. To validate our method, we introduce a large-scale viral dance video (VDV) dataset, which contains over 4,000 dance clips of eight viral dance challenges. Extensive experiments on the VDV dataset well demonstrate the effectiveness of our approach. Furthermore, we show that short video applications such as multi-dimensional recommendation and action feedback can be derived from our model.

show abstract

“…Surgical video rather than kinematics also embeds gesture information which can be extracted with spatio-temporal CNNs [27], 3D CNNs [28], multi-scale temporal convolutions [29,30] or hybrid encoder-decoder networks with temporal-convolutional filters for local motion modelling and bidirectional LSTM for long-range dependency memorization [31].…”

Section: A Related Workmentioning

confidence: 99%

Multi-Task Recurrent Neural Network for Surgical Gesture Recognition and Progress Prediction

Amsterdam¹,

Clarkson²,

Stoyanov³

2020

Preprint

View full text Add to dashboard Cite

Surgical gesture recognition is important for surgical data science and computer-aided intervention. Even with robotic kinematic information, automatically segmenting surgical steps presents numerous challenges because surgical demonstrations are characterized by high variability in style, duration and order of actions. In order to extract discriminative features from the kinematic signals and boost recognition accuracy, we propose a multi-task recurrent neural network for simultaneous recognition of surgical gestures and estimation of a novel formulation of surgical task progress. To show the effectiveness of the presented approach, we evaluate its application on the JIGSAWS dataset, that is currently the only publicly available dataset for surgical gesture recognition featuring robot kinematic data. We demonstrate that recognition performance improves in multi-task frameworks with progress estimation without any additional manual labelling and training.

show abstract

Atrous Temporal Convolutional Network for Video Action Segmentation

Cited by 8 publications

References 13 publications

Gesture Recognition in Robotic Surgery: A Review

Gesture Recognition in Robotic Surgery: A Review

Will You Ever Become Popular? Learning to Predict Virality of Dance Clips

Multi-Task Recurrent Neural Network for Surgical Gesture Recognition and Progress Prediction

Contact Info

Product

Resources

About