Fast Object Detection in Compressed Video

Abstract

Object detection in videos has drawn increasing attention since it is more practical in real scenarios. Most of the deep learning methods use CNNs to process each decoded frame in a video stream individually. However, the free of charge yet valuable motion information already embedded in the video compression format is usually overlooked. In this paper, we propose a fast object detection method by taking advantage of this with a novel Motion aided Memory Network (MMNet). The MMNet has two major advantages: 1) It significantly accelerates the procedure of feature extraction for compressed videos. It only need to run a complete recognition network for I-frames, i.e. a few reference frames in a video, and it produces the features for the following P frames (predictive frames) with a light weight memory network, which runs fast; 2) Unlike existing methods that establish an additional network to model motion of frames, we take full advantage of both motion vectors and residual errors that are freely available in video streams. To our best knowledge, the MMNet is the first work that investigates a deep convolutional detector on compressed videos. Our method is evaluated on the large-scale Ima-geNet VID dataset, and the results show that it is 3× times faster than single image detector R-FCN and 10× times faster than high-performance detector MANet at a minor accuracy loss.

Fast object detection in compressed JPEG Images

“…Another research stream designs dedicated networks to spectral input coefficients: harmonic networks [6] uses custom convolutions that produce high-level features by learning combinations of spectral filters defined by the 2D Discrete Cosine Transform; Ehrlich and Davis (2019) [7] introduce a ResNet able to operate on compressed JPEG images by including the compression transform into the network weights. From video side, two recent works on detection in compressed videos are [8], [9]. In [8], separate CNNs are used for temporally linked I-frame (RGB image), and P-frame (motion and residual arrays) are trained all together.…”

“…In [8], separate CNNs are used for temporally linked I-frame (RGB image), and P-frame (motion and residual arrays) are trained all together. In [9], the authors consider three networks: a CNN feature extraction module based on the raw I-image, a recurrent memory network to align the features of consecutive P-frames using compressed motion and residual vectors, and a detection network aiming at identifying the objects in the videos.…”

Real-Time Online Multi-Object Tracking in Compressed Domain

“…Alvar et al [17] constructed approximate bounding-box of the target object in the pixel level based on the bounding-box in previous frame for single object tracking, but each frame needs to be restored into RGB image for detection. The work in [18] fed MVs and residuals to a network to propagate the features across frames for object detection. However, the frames are processed in a batch manner which is inapplicable for online tasks.…”

“…Secondly, the motion information is readily provided in the compressed domain which is helpful for video tasks. A few works, such as tracking [8], [17], video object detection [18] and action recognition [19], have been done in the compressed domain. Generally, there are two goals of the implementations in the compressed domain: (1) Feature propagation [18].…”

Compact Scrutiny of Current Video Tracking System and its Associated Standard Approaches

“…The motion model for a three-dimensional object is usually related to the position and orientation of the object [10]. While dealing with video compression, the macroblocks are divided into keyframes, and selected motion motions are considered disruptions of these frames considering motion parameters [11]. In the case of a deformable object, the motion model generally considers the position of a target object over the mesh [12].…”