Action Segmentation With Joint Self-Supervised Temporal Domain Adaptation

Chen, Min-Hung; Li, Baopu; Bao, Yingze; AlRegib, Ghassan; Kira, Zsolt

doi:10.1109/cvpr42600.2020.00947

Cited by 98 publications

(89 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, there exist several approaches to improve the performance of action segmentation models such as MS-TCN [3,26,9,10]. Chen et al [3] proposed to apply selfsupervised domain adaptation techniques when training a model such as MS-TCN, and it exploits unlabeled videos to boost the performance of action segmentation.…”

Section: Related Workmentioning

confidence: 99%

“…Researches in temporal action segmentation have been improved to successfully segment thousands of video frames recorded with 15 fps [5,3,10]. However, we find out that existing state-of-the-art models sometimes generate segmentation results including action labels that are out of overall context.…”

Section: Introductionmentioning

confidence: 99%

“…In this paper, we propose a Hierarchical Action Segmentation Refiner (HASR), which can extract the hierarchical video representations to understand the overall context, and exploit them to refine the results from action segmentation backbone models. Here, the action segmentation backbone models refer to any existing approaches for a temporal action segmentation task, such as MS-TCN [5], SSTDA [3] and ASRF [10]. Figure 1 shows an illustration of how HASR works.…”

Section: Introductionmentioning

confidence: 99%

“…• Our approach can boost up the performance of existing state-of-the-art action segmentation models. For example, based on the 50Salads dataset [22], our model improves the segmental edit score from 67.9% to 77.4% for MS-TCN [5], from 75.8% to 77.3% for SSTDA [3], from 79.3% to 81.0% for ASRF [10].…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Refining Action Segmentation with Hierarchical Video Representations

Ahn

Lee

2021

2021 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

In this paper, we propose Hierarchical Action Segmentation Refiner (HASR), which can refine temporal action segmentation results from various models by understanding the overall context of a given video in a hierarchical way. When a backbone model for action segmentation estimates how the given video can be segmented, our model extracts segment-level representations based on frame-level features, and extracts a video-level representation based on the segment-level representations. Based on these hierarchical representations, our model can refer to the overall context of the entire video, and predict how the segment labels that are out of context should be corrected. Our HASR can be plugged into various action segmentation models (MS-TCN, SSTDA, ASRF), and improve the performance of state-of-the-art models based on three challenging datasets (GTEA, 50Salads, and Breakfast). For example, in 50Salads dataset, the segmental edit score improves from 67.9% to 77.4% (MS-TCN), from 75.8% to 77.3% (SSTDA), from 79.3% to 81.0% (ASRF). In addition, our model can refine the segmentation result from the unseen backbone model, which was not referred to when training HASR. This generalization performance would make HASR be an effective tool for boosting up the existing approaches for temporal action segmentation. Our code is available at https: //github.com/cotton-ahn/HASR_iccv2021.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Refining Action Segmentation with Hierarchical Video Representations

Ahn

Lee

2021

2021 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

show abstract

“…It works by designing an auxiliary task that labels can be self-annotated. For example, [5] proposed an auxiliary task that predicts temporal permutation for cross-domain videos to tackle the problem of Spatio-temporal variations for action segmentation. This self-supervised approach combined with MS-TCN has improved the MS-TCN stand-alone version accuracy on all three datasets 50Salads [33], GTEA [9] and Breakfast [21], and requires only 65% of the labeled training data for comparable performance.…”

Section: Action Recognitionmentioning

confidence: 99%

Towards Neural-Symbolic AI for Media Understanding

Costa

Marques

Serra

et al. 2020

Anais Estendidos Do XXVI Simpósio Brasileiro De Sistemas Multimídia E Web (WebMedia 2020)

View full text Add to dashboard Cite

Methods based on Machine Learning have become state-of-the-art in various segments of computing, especially in the fields of computer vision, speech recognition, and natural language processing. Such methods, however, generally work best when applied to specific tasks in specific domains where large training datasets are available. This paper presents an overview of the state-of-the-art in the area of Deep Learning for Multimedia Content Analysis (image, audio, and video), and describe recent works that propose The integration of deep learning with symbolic AI reasoning. We draw a picture of the future by discussing envisaged use cases that address media understanding gaps which can be solved by the integration of machine learning and symbolic AI, the so-called Neuro-Symbolic integration.

show abstract