Hierarchical Attention Network for Action Segmentation

Abstract

Temporal segmentation of events is an essential task and a precursor for the automatic recognition of human actions in the video. Several attempts have been made to capture frame-level salient aspects through attention but they lack the capacity to effectively map the temporal relationships in between the frames as they only capture a limited span of temporal dependencies. To this end we propose a complete end-to-end supervised learning approach that can better learn relationships between actions over time, thus improving the overall segmentation performance. The proposed hierarchical recurrent attention framework analyses the input video at multiple temporal scales, to form embeddings at frame level and segment level, and perform fine-grained action segmentation. This generates a simple, lightweight, yet extremely effective architecture for segmenting continuous video streams and has multiple application domains. We evaluate our system on multiple challenging public benchmark datasets, including MERL Shopping, 50 salads, and Georgia Tech Egocentric datasets and achieves state-of-the-art performance. The evaluated datasets encompass numerous video capture settings which are inclusive of static overhead camera views and dynamic, ego-centric headmounted camera views, demonstrating the direct applicability of the proposed framework in a variety of settings.

Continuous Human Action Recognition for Human-Machine Interaction: A Review

“…These ResNet models are less complex (in terms of parameters) compared to previous pre-trained networks such as VGG networks, where even the deeper ResNet network (i.e. ResNet152) is less complex (11.3 Due to the aforementioned advantages, pre-trained ResNet networks have been widely used as a feature extraction backbone within both the action recognition domain [144], [145], [146], [46], [147] and related problem domains [148], [149], [150].…”