Temporal Action Segmentation from Timestamp Supervision

Abstract

Temporal action segmentation approaches have been very successful recently. However, annotating videos with frame-wise labels to train such models is very expensive and time consuming. While weakly supervised methods trained using only ordered action lists require less annotation effort, the performance is still worse than fully supervised approaches. In this paper, we propose to use timestamp supervision for the temporal action segmentation task. Timestamps require a comparable annotation effort to weakly supervised approaches, and yet provide a more supervisory signal. To demonstrate the effectiveness of timestamp supervision, we propose an approach to train a segmentation model using only timestamps annotations. Our approach uses the model output and the annotated timestamps to generate frame-wise labels by detecting the action changes. We further introduce a confidence loss that forces the predicted probabilities to monotonically decrease as the distance to the timestamps increases. This ensures that all and not only the most distinctive frames of an action are learned during training. The evaluation on four datasets shows that models trained with timestamps annotations achieve comparable performance to the fully supervised approaches.

Iterative Contrast-Classify for Semi-supervised Temporal Action Segmentation

“…Weakly supervised methods bypass per-frame annotations and use labels such as ordered lists of actions (Ding and Xu 2018;Richard et al 2018;Chang et al 2019;Li, Lei, and Todorovic 2019;Souri et al 2019) or a small percentage of action time-stamps (Kuehne, Richard, and Gall 2018;Li, Farha, and Gall 2021;Chen et al 2020a) for all videos.…”

“…There is a huge annotation cost to label each frame of all videos for action segmentation, especially as the videos are minutes long. Several works aim to reduce annotation requirements with weak supervision like transcripts (Chang et al 2019), or few frame labels (Li, Farha, and Gall 2021). In this work, we advocate using semisupervised learning, i.e.…”

“…To the best of our knowledge, our work is the first to apply semi-supervised learning for temporal action segmentation. The closest in spirit are SSTDA (Chen et al 2020a), TSS (Li, Farha, and Gall 2021). However, these works are weaklysupervised setups and require (weak) labels for every training video (see Fig.…”

Iterative Contrast-Classify For Semi-supervised Temporal Action Segmentation

“…Summarizing our contributions, we: Weakly supervised methods bypass per-frame annotations and use labels such as ordered lists of actions (Ding and Xu 2018;Richard et al 2018;Chang et al 2019;Li, Lei, and Todorovic 2019;Souri et al 2019) or a small percentage of action time-stamps (Kuehne, Richard, and Gall 2018;Li, Farha, and Gall 2021;Chen et al 2020a) for all videos.…”