Predicting Actions in Videos and Action-Based Segmentation Using Deep Learning

Abstract

In this paper, we propose a technique to recognize multiple actions in a video using deep learning. The proposed approach is concerned with interpreting the overall context of a video and transforming it into one or more appropriate actions. In order to cope with multiple actions in a video, our proposed technique first determines the individual segments/shots in a video using intersections of color histograms. The segmented parts are then fed to the action recognition system comprising a combination of a Convolution Neural Network (CNN) and a Long-Short-Term Memory (LSTM) network trained on our action vocabulary. The segments are then labeled according to their predicted actions and a compact set of distinct actions is produced. Using the corpus generated by the shot detection phase, which includes the location of key frames in shots, and start/end timestamps of a shot, we can also perform video segmentation based on an action query. Hence, the proposed technique can be used for a number of tasks such as content censoring, on-demand scene retrieval, video summarizing, and query based scene/video retrieval, to name a few. The proposed technique also stands aprat from the existing approaches which either do not take into account the motion information for action prediction or do not perform action-based video segmentation. The experimental results presented in this paper show that the proposed technique not only finds the the complete set of actions present in the video, but can also find all the relevant parts in a video based on an action query.

ViGAT: Bottom-Up Event Recognition and Explanation in Video Using Factorized Graph Attention Network

“…In [52], a pretrained ResNet is used to derive a feature representation for each frame and an LSTM to process the temporal information. In [3], a 2D-CNN and LSTM are used to process the spatiotemporal video information, and in addition, shot boundary detection is applied to segment and predict multiple actions occurring in a video. PivotCorrNN [53] introduces contextual gated recurrent units (cGRUs) to exploit time-varying information among different modalities (MFCC, IDT, etc.).…”

Computer Vision: A Review of Detecting Objects in Videos – Challenges and Techniques

“…It could be one of the obstacles [18,19,20]. The significant challenging point in object detection is occlusion and it may be full/partial and can occur anytime an object passes behind another object [17,21].…”

Gymnastic Movement Recognition Based on Depth Convolution Neural Network

“…More importantly, the CNN can process large-scale high-definition resolution images. [7][8]. Therefore, the CNN reduces the amount of computation by means of sparse connections, as shown in Figure 1, which represents the difference between sparse connections and full connections.…”