Co-Segmentation Inspired Attention Networks for Video-Based Person Re-Identification

Abstract

Video based computer vision tasks can benefit from estimation of the salient regions and interactions between those regions. Traditionally, this has been done by identifying the object regions in the images by utilizing pre-trained models to perform object detection, object segmentation and/or object pose estimation. Though using pre-trained models seems to be a viable approach, it is infeasible in practice due to the need for exhaustive annotation of object categories, domain gap between datasets and bias present in pre-trained models. To overcome these downsides, we propose to utilize the common rationale that a sequence of video frames capture a set of common objects and interactions between them, thus a notion of co-segmentation between the video frame features may equip the model with the ability to automatically focus on salient regions and improve underlying task's performance in an end-to-end manner. In this regard, we propose a generic module called "Co-Segmentation Activation Module" (COSAM) that can be plugged-in to any CNN to promote the notion of co-segmentation based attention among a sequence of video frame features. We show the application of COSAM in three video based tasks namely: 1) Video-based person re-ID, 2) Video captioning, & 3) Video action classification and demonstrate that COSAM is able to capture salient regions in the video frames, thus leading to notable performance improvements along with interpretable attention maps.

ASTA-Net: Adaptive Spatio-Temporal Attention Network for Person Re-Identification in Videos

“…We compare the proposed ASTA-Net against sixteen state-of-the-art methods on the MARS dataset in terms of CMC accuracy and mAP score, with the results shown in Table 1. The compared methods belong to two categories, i.e., image-based person reidentification methods including PartsNet [17], Re-ranking [51], MGCAM [38], Triplet [8], and video-based person reidentification methods including JST-RNN [53], QAN [29], DuATM [37], SpaAtn [20], RRU [30], M3D [19], COSAM [39], Snippet [1], STA [4], ADFA [49], GLTR [18], VRSTC [11]. As shown in Table 1, the proposed ASTA-Net achieves 90.4% rank-1 recognition rate and 84.1% mAP score.…”

Channel Transformer Network

“…It's really a challenging problem since there exist a lot of variations like view point, background clutter, occlusion, misalignment, etc. There have been lots of works focusing on two branches for the problem, one is image-based person ReID [12][13][14][15][16][17][18][19][20][21], the other one is video-based [8,10,[22][23][24][25][26][27][28][29][30][31][32][33][34].…”

“…They also provided a spatial and temporal efficient method to reduce FLOPs (floating-point operations per second) for saving computation resources. Subramaniam et al formulated a Co-segmentation based Attention Module (COSAM) [10] for video-based person ReID. The module could help to extract a common set of salient feature maps among video frames through a Normalized Cross Correlation (NCC) layer and a summarization layer.…”

“…Normally, we design an attention or transform block which can be inserted into any place in CNNs without affecting the size of input and output feature maps, such as Nonlocal [7,8], CBAM [9], COSAM [10], CosegCA [11], etc. In these blocks or modules, 1×1 convolutional layers or fully connected layers are used to reduce channel dimension at the beginning for saving computation complexity.…”

“…We also present the lightweight and efficient Video Co-segment Attentive Network (VCAN) based on CosegCA [11] for person ReID to better study CTN. Then we inject CTN into Non-local [7,8], CBAM [9], COSAM [10] and VCAN to make them parameter-free when transforming channel dimension. To the best of our knowledge, this is the first work that shows how to change channel dimension in a parameter-free way for saving network parameters and computation resources.…”