HARTIV: Human Activity Recognition Using Temporal Information in Videos

Abstract

Nowadays, the most challenging and important problem of computer vision is to detect human activities and recognize the same with temporal information from video data. The video datasets are generated using cameras available in various devices that can be in a static or dynamic position and are referred to as untrimmed videos. Smarter monitoring is a historical necessity in which commonly occurring, regular, and out-of-the-ordinary activities can be automatically identified using intelligence systems and computer vision technology. In a long video, human activity may be present anywhere in the video. There can be a single or multiple human activities present in such videos. This paper presents a deep learning-based methodology to identify the locally present human activities in the video sequences captured by a single wide-view camera in a sports environment. The recognition process is split into four parts: firstly, the video is divided into different set of frames, then the human body part in a sequence of frames is identified, next process is to identify the human activity using a convolutional neural network and finally the time information of the observed postures for each activity is determined with the help of a deep learning algorithm. The proposed approach has been tested on two different sports datasets including ActivityNet and THUMOS. Three sports activities like swimming, cricket bowling and high jump have been considered in this paper and classified with the temporal information i.e., the start and end time for every activity present in the video. The convolutional neural network and long short-term memory are used for feature extraction of temporal action recognition from video data of sports activity. The outcomes show that the proposed method for activity recognition in the sports domain outperforms the existing methods.

Video Frame Prediction by Joint Optimization of Direct Frame Synthesis and Optical-Flow Estimation

“…At forget gate, ConvLSTM decides which information (f t ) should be forgotten from the cell states, as in (10). Based on the update at the input and forget gate, the old cell state c t−1 updates cell state (c t ) given, as in (11). At the output gate, the ConvLSTM decides which part of the cell state to send to the output (o t ), as in (12).…”

“…Such predictive cognitive neural networks are often considered the essence of computer vision. They play a critical role in a variety of applications, such as abnormal event detection [1], autonomous driving [2][3][4], intention prediction in robotics [5,6], video coding [7,8], collision avoidance systems [9,10], activity and event prediction [11,12], and pedestrian and traffic prediction [13][14][15]. However, modeling future image content and object motion is challenging due to dynamic evolution and image complexity, such as occlusions, camera movements, and illumination.…”