C2F-TCN: A Framework for Semi- and Fully-Supervised Temporal Action Segmentation

Abstract

Temporal action segmentation tags action labels for every frame in an input untrimmed video containing multiple actions in a sequence. For the task of temporal action segmentation, we propose an encoder-decoder style architecture named C2F-TCN featuring a "coarse-to-fine" ensemble of decoder outputs. The C2F-TCN framework is enhanced with a novel model agnostic temporal feature augmentation strategy formed by the computationally inexpensive strategy of the stochastic max-pooling of segments. It produces more accurate and well-calibrated supervised results on three benchmark action segmentation datasets. We show that the architecture is flexible for both supervised and representation learning. In line with this, we present a novel unsupervised way to learn frame-wise representation from C2F-TCN. Our unsupervised learning approach hinges on the clustering capabilities of the input features and the formation of multi-resolution features from the decoder's implicit structure. Further, we provide first semi-supervised temporal action segmentation results by merging representation learning with conventional supervised learning. Our semi-supervised learning scheme, called "Iterative-Contrastive-Classify (ICC)", progressively improves in performance with more labeled data. The ICC semi-supervised learning in C2F-TCN, with 40% labeled videos, performs similar to fully supervised counterparts.

Early Drought Detection in Maize Using UAV Images and YOLOv8+

“…After the input of the C2F module undergoes processing via a CBS, the output dimensions are transformed into B, C, H, and W, where B denotes the number of images, C indicates the number of channels, and H and W represent the height and width of the feature map, respectively [20]. The detailed structure is depicted in Figure 5.…”

“…Essentially, the output of each BottleNeck must be preserved and utilized as input for the following BottleNeck. Eventually, all corresponding feature channels are fused, resulting in dimensions of B × (N + 2)C/2 × H × W. After the input of the C2F module undergoes processing via a CBS, the output dimensions are transformed into B, C, H, and W, where B denotes the number of images, C indicates the number of channels, and H and W represent the height and width of the feature map, respectively [20]. The detailed structure is depicted in Figure 5.…”

Regional Semantic Contrast and Aggregation for Weakly Supervised Semantic Segmentation

“…Recently, semi-supervised approaches [16]- [18] for this task have attracted increasing attention, with a small percentage of labelled videos in the training set. Iterative-Contrast-Classify (ICC) [16] is the first attempt to explore semi-supervised learning for human action segmentation, which consists of two steps, i.e., unsupervised representation learning based on contrastive learning [19] (Fig.…”