Learning Temporally Invariant and Localizable Features via Data Augmentation for Video Recognition

Abstract

Data augmentation has recently emerged as an essential component of modern training recipes for visual recognition tasks. However, data augmentation for video recognition has been rarely explored despite its effectiveness. Few existing augmentation recipes for video recognition naively extend the image augmentation methods by applying the same operations to the whole video frames. Our main idea is that the magnitude of augmentation operations for each frame needs to be changed over time to capture the real-world video's temporal variations. These variations should be generated as diverse as possible using fewer additional hyperparameters during training. Through this motivation, we propose a simple yet effective video data augmentation framework, DynaAugment. The magnitude of augmentation operations on each frame is changed by an effective mechanism, Fourier Sampling that parameterizes diverse, smooth, and realistic temporal variations. DynaAugment also includes an extended search space suitable for video for automatic data augmentation methods. DynaAugment experimentally demonstrates that there are additional performance rooms to be improved from static augmentations on diverse video models. Specifically, we show the effectiveness of DynaAugment on various video datasets and tasks: large-scale video recognition (Kinetics-400 and Something-Something-v2), small-scale video recognition (UCF-101 and HMDB-51), fine-grained video recognition (Diving-48 and FineGym), video action segmentation on Breakfast, video action localization on THUMOS'14, and video object detection on MOT17Det. DynaAugment also enables video models to learn more generalized representation to improve the model robustness on the corrupted videos.Preprint. Under review.

Extending Temporal Data Augmentation for Video Action Recognition

“…We propose to linearly change the bounding pox positions for cut-and-paste algorithms, such as CutOut [11], CutMix [65], and CutMixUp [64], and their extensions, as well as the mixing ratio in MixUp [66] and CutMixUp [64] extensions; 4. The recognition results of the aforementioned techniques on the UCF-101 [55] and the HMDB-51 [38] datasets either maintain competitive or exceed performance achieved by the previous work [34].…”

“…In this paper, we expand on the previous work [34]. We argue that some of the proposed techniques can be extended even further to fully utilise the time domain and achieve a deeper level of temporal perturbations, which results in more accurate and robust classifiers.…”

“…Data augmentation for videos, however, still remains an under-explored research area, as most works have been treating inputs as stacks of static images rather than temporally linked series of data. A recent study has shown that the time domain consideration while designing augmentations can be superior to its spatial-only variants for video recognition [34].…”

“…1. We expand the list of available augmentations in RandAugment-T [34] by adding VideoReverse, FrameFadeIn, and VideoCutMix, augmentations that are video-specific and are done within a single sample; 2. We increase the amount of magnitude checkpoints for all augmentation techniques to allow for non-linear temporal perturbations; 3.…”

“…Data augmentation for videos, however, still remains an under-explored research topic, as most works have been treating inputs as stacks of static images rather than temporally linked series of data. Recently, it has been shown that involving the time dimension when designing augmentations can be superior to its spatial-only variants for video action recognition [34]. In this paper, we propose several novel enhancements to these techniques to strengthen the relationship between the spatial and temporal domains and achieve a deeper level of perturbations.…”