Review of Video Predictive Understanding: Early Action Recognition and Future Action Prediction

Abstract

Video predictive understanding encompasses a wide range of efforts that are concerned with the anticipation of the unobserved future from the current as well as historical video observations. Action prediction is a major sub-area of video predictive understanding and is the focus of this review. This sub-area has two major subdivisions: early action recognition and future action prediction. Early action recognition is concerned with recognizing an ongoing action as soon as possible. Future action prediction is concerned with the anticipation of actions that follow those previously observed. In either case, the causal relationship between the past, current and potential future information is the main focus.Various mathematical tools such as Markov Chains, Gaussian Processes, Auto-Regressive modeling and Bayesian recursive filtering are widely adopted jointly with computer vision techniques for these two tasks.However, these approaches face challenges such as the curse of dimensionality, poor generalization and constraints from domain specific knowledge. Recently, structures that rely on deep convolutional neural networks and recurrent neural networks have been extensively proposed for improving performance of existing vision tasks, in general, and action prediction tasks, in particular. However, they have their own shortcomings, e.g., reliance on massive training data and lack of strong theoretical underpinnings. In this survey, we start by introducing the major sub-areas of the broad area of video predictive understanding, which recently have received intensive attention and proven to have practical value. Next, a thorough review of various early action recognition and future action prediction algorithms are provided with suitably organized divisions. Finally, we conclude our discussion with future research directions.

Multi-Head TrajectoryCNN: A New Multi-Task Framework for Action Prediction

“…its corresponding future motion sequence can be represented as M×J×D and its action semantic denoted by y, where N is the length of the partially observed motion sequence, M is the length of the future human motion sequence, J is the number of joints of the human body, D is the dimension of the joints (i.e., D = 3), and j i * is the * -th joint of the i-th pose. The action prediction model aims to predict the future motion sequence and action semantics using on a partially observed motion sequence, which can be formulated by Equation (1).…”

“…Predicting future action information is important due to its undeniable applications in computer vision and robotics, among which action semantics and human motion predictions are two key aspects [1]. As shown in Figure 1, our prediction task is different from the traditional recognition task.…”

Prediction of Social Dynamic Agents and Long-Tailed Learning Challenges: A Survey

“…provide a succinct table of leading vehicle prediction approaches and their characteristics (input modalities, output modalities, history/scene encoding, interaction encoding), and Rudenko et al (2020) detail pedestrian prediction metrics, datasets, and methods organised into an insightful taxonomy. Finally, Zhao and Wildes (2021) and Hu et al (2022) discuss datasets, state-of-the-art performance, and online methods for action prediction.…”