Video Playback Rate Perception for Self-Supervised Spatio-Temporal Representation Learning

Yao, Yuan; Liu, Chang; Luo, Dezhao; Zhou, Yu; Ye, Qixiang

doi:10.1109/cvpr42600.2020.00658

Cited by 166 publications

(135 citation statements)

References 23 publications

Supporting

Mentioning

135

Contrasting

Order By: Relevance

“…Due to the uniqueness and discrimination of corners, we look forward to extending this work to accurate arbitrary-shape text detection and end-to-end text spotting. We would also like to combine this work with self-supervised learning [27][28][29][30].…”

Section: Discussionmentioning

confidence: 99%

FC²RN: A Fully Convolutional Corner Refinement Network for Accurate Multi-Oriented Scene Text Detection

Qin

Zhou

Guo

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Accurate detection of multi-oriented text that accounts for a large proportion in real practice is of great significance. The performance has improved rapidly on common benchmarks in recent years. However, dense long text case and the quality of detection are easy to be overlooked. Direct regression may produce low-quality and incomplete detections due to the constrain of the receptive field; proposal-based methods could alleviate this but might introduce redundant context due to RoI operation, degrading the performance. To address the dilemma, a novel proposed corner-aware convolution in which the sampling positions tightly cover the text area is utilized to encode an initial corner prediction into the feature maps, which can be further used to produce a refined corner prediction. We embed the proposed module into an anchor-free baseline model, leading to a simple and effective fully convolutional corner refinement network (FC 2 RN). Experimental results on four public datasets including MSRA-TD500, ICDAR2015, RCTW-17, and COCO-Text demonstrate that FC 2 RN can outperform state-of-the-art methods.

show abstract

Section: Discussionmentioning

confidence: 99%

FC²RN: A Fully Convolutional Corner Refinement Network for Accurate Multi-Oriented Scene Text Detection

Qin

Zhou

Guo

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Instead of directly predicting low-level information, methods based on spatio- more dedicate pretext tasks, such as temporal order prediction [13,14,39,40,75] and video speed prediction [15,16,18]. Compared to dense prediction methods, the spatio-temporal reasoning methods are more efficient since they discard additional generators.…”

Section: Spatio-temporal Reasoning Methodsmentioning

confidence: 99%

“…Although improvement can be achieved by temporal order prediction, there is still a noticeable gap in performance when compared to fully-supervised methods. To narrow the gap of performance, recent methods [15,16,18] et al [18] propose a self-supervised pace prediction task, where they discard the generation task in PRP [15] and include an additional constrative learning task.…”

Section: Spatio-temporal Reasoning Methodsmentioning

confidence: 99%

“…Following previous works [15,16,75], we utilize R3D-18 which is the 18-layer variant of R3D. R(2+1)D [27] proposes to replace the traditional 3D kernel with the combination of a 2D kernel and a 1D kernel for spatial and temporal feature extraction, respectively.…”

Section: Methodsmentioning

confidence: 99%

“…optical flow [12]) or complicated computation process [37,38], leading to an expensive cost of time and resources. To improve the efficiency, recent spatio-temporal reasoning methods, such as clip order prediction [13,14,39,40], video speed prediction [15,16,18]…”

Section: Self-supervised Learningmentioning

confidence: 99%

See 2 more Smart Citations

Effective action recognition with fully supervised and self-supervised methods

Cao¹

View full text Add to dashboard Cite

Action recognition in videos has attracted interest in computer vision and machine learning communities thanks to its applications such as surveillance and smart homes. In addition to spatial information in individual frames, videos contain temporal information across the temporal dimension. Therefore, effective spatiotemporal representation is the key to accurate action recognition in videos.Previous works have proposed various fully-supervised and self-supervised methods for video representation learning. For fully-supervised methods, most of them utilize convolution neural networks (CNNs) to extract spatial representation while temporal representation is usually modelled by pixel-wise correlations. However, it is inefficient to extract correlations between all pixels since some of them may relate to in-salient area (e.g. backgrounds or environments). On the other hand, self-supervised methods are proposed to leverage more accessible unlabled data on the Internet and transfer the extracted representation for different downstream tasks. The core of self-supervised methods is to design a pretext task where supervision signal is automatically generated based on characteristics of unlabeled data.Although self-supervised methods avoid the annotation of labeled data, compared to fully-supervised methods, there is room for performance improvement of selfsupervised methods. In this thesis, we address the above research gap with two novel deep learning methods, to advance fully-supervised and self-supervised methods, respectively. For fully-supervised learning, we propose a novel Key Point Shift Embedding Module (KPSEM) to adaptively extract channel-wise key point shifts across video frames without key point annotation for temporal feature extraction. Key points are adaptively extracted as feature points with maximum feature values at split regions, while key point shifts are the spatial displacements of corresponding key points. The key point shifts are encoded as the overall temporal features via linear embedding layers in a multi-set manner. vi To advance self-supervised learning, we propose a novel self-supervised learning method, called Video Incoherence Detection (VID), that leverages incoherence detection for spatio-temporal feature extraction. It roots from the observation that visual systems of human beings can easily identify video incoherence based on their comprehensive understanding of videos. Specifically, the training sample, denoted as the incoherent clip, is constructed by multiple sub-clips hierarchically sampled from the same raw video with various lengths of incoherence between each other. The network is trained to learn high-level representation by predicting the relative location and length of incoherence given the incoherent clip as input. Additionally, intra-video contrastive learning is introduced to maximize the mutual information between different incoherent clips from the same raw video. Our experiments show that both KPSEM and VID achieve state-of-the-art performance on action recognition wi...

show abstract

Self-supervised Video Representation Learning by Pace Prediction

Wang

Jiao

Liu

2020

Computer Vision – ECCV 2020

159

195

View full text Add to dashboard Cite

Video Playback Rate Perception for Self-Supervised Spatio-Temporal Representation Learning

Cited by 166 publications

References 23 publications

FC²RN: A Fully Convolutional Corner Refinement Network for Accurate Multi-Oriented Scene Text Detection

FC²RN: A Fully Convolutional Corner Refinement Network for Accurate Multi-Oriented Scene Text Detection

Effective action recognition with fully supervised and self-supervised methods

Self-supervised Video Representation Learning by Pace Prediction

Contact Info

Product

Resources

About

Video Playback Rate Perception for Self-Supervised Spatio-Temporal Representation Learning

Cited by 166 publications

References 23 publications

FC2RN: A Fully Convolutional Corner Refinement Network for Accurate Multi-Oriented Scene Text Detection

FC2RN: A Fully Convolutional Corner Refinement Network for Accurate Multi-Oriented Scene Text Detection

Effective action recognition with fully supervised and self-supervised methods

Self-supervised Video Representation Learning by Pace Prediction

Contact Info

Product

Resources

About

FC²RN: A Fully Convolutional Corner Refinement Network for Accurate Multi-Oriented Scene Text Detection

FC²RN: A Fully Convolutional Corner Refinement Network for Accurate Multi-Oriented Scene Text Detection