STFlow: Self-Taught Optical Flow Estimation Using Pseudo Labels

Ren, Zhe; Luo, Wenhan; Yan, Junchi; Liao, Wenlong; Yang, Xiaokang; Yuille, Alan; Zha, Hongyuan

doi:10.1109/tip.2020.3024015

Cited by 17 publications

(6 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In future work, we are exploring the way of integrating optical flow networks into our learning pipeline, especially for those unsupervised flow models based on our previous work (Ren et al 2017(Ren et al , 2020a, as the two tasks for action recognition and flow estimation can be of mutual benefit to each other.…”

Section: Discussionmentioning

confidence: 99%

Learning Comprehensive Motion Representation for Action Recognition

Jiang

Luo

et al. 2021

AAAI

Self Cite

View full text Add to dashboard Cite

For action recognition learning, 2D CNN-based methods are efficient but may yield redundant features due to applying the same 2D convolution kernel to each frame. Recent efforts attempt to capture motion information by establishing inter-frame connections while still suffering the limited temporal receptive field or high latency. Moreover, the feature enhancement is often only performed by channel or space dimension in action recognition. To address these issues, we first devise a Channel-wise Motion Enhancement (CME) module to adaptively emphasize the channels related to dynamic information with a channel-wise gate vector. The channel gates generated by CME incorporate the information from all the other frames in the video. We further propose a Spatial-wise Motion Enhancement (SME) module to focus on the regions with the critical target in motion, according to the point-to-point similarity between adjacent feature maps. The intuition is that the change of background is typically slower than the motion area. Both CME and SME have clear physical meaning in capturing action clues. By integrating the two modules into the off-the-shelf 2D network, we finally obtain a Comprehensive Motion Representation (CMR) learning method for action recognition, which achieves competitive performance on Something-Something V1 & V2 and Kinetics-400. On the temporal reasoning datasets Something-Something V1 and V2, our method outperforms the current state-of-the-art by 2.3% and 1.9% when using 16 frames as input, respectively.

show abstract

Section: Discussionmentioning

confidence: 99%

Learning Comprehensive Motion Representation for Action Recognition

Jiang

Luo

et al. 2021

AAAI

Self Cite

View full text Add to dashboard Cite

show abstract

Section: Discussionmentioning

confidence: 99%

Learning Comprehensive Motion Representation for Action Recognition

Wu¹,

Jiang²,

Luo³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

For action recognition learning, 2D CNN-based methods are efficient but may yield redundant features due to applying the same 2D convolution kernel to each frame. Recent efforts attempt to capture motion information by establishing interframe connections while still suffering the limited temporal receptive field or high latency. Moreover, the feature enhancement is often only performed by channel or space dimension in action recognition. To address these issues, we first devise a Channel-wise Motion Enhancement (CME) module to adaptively emphasize the channels related to dynamic information with a channel-wise gate vector. The channel gates generated by CME incorporate the information from all the other frames in the video. We further propose a Spatial-wise Motion Enhancement (SME) module to focus on the regions with the critical target in motion, according to the point-to-point similarity between adjacent feature maps. The intuition is that the change of background is typically slower than the motion area. Both CME and SME have clear physical meaning in capturing action clues. By integrating the two modules into the off-the-shelf 2D network, we finally obtain a Comprehensive Motion Representation (CMR) learning method for action recognition, which achieves competitive performance on Something-Something V1 & V2 and Kinetics-400. On the temporal reasoning datasets Something-Something V1 and V2, our method outperforms the current state-of-the-art by 2.3% and 1.9% when using 16 frames as input, respectively.

show abstract

“…However, the teacher can not benefit from the improved student. STFlow [41] integrates variational refinement with unsuperivsed learning and propose a self-taught framework for continual improvement. SimFlow [26] explores learnable feature similarity for regulating previous census reconstruction loss.…”

Section: B Learning Unsupervised Optical Flowmentioning

confidence: 99%

MDFlow: Unsupervised Optical Flow Learning by Reliable Mutual Knowledge Distillation

Kong

Yang

2023

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

Recent works have shown that optical flow can be learned by deep networks from unlabelled image pairs based on brightness constancy assumption and smoothness prior. Current approaches additionally impose an augmentation regularization term for continual self-supervision, which has been proved to be effective on difficult matching regions. However, this method also amplify the inevitable mismatch in unsupervised setting, blocking the learning process towards optimal solution. To break the dilemma, we propose a novel mutual distillation framework to transfer reliable knowledge back and forth between the teacher and student networks for alternate improvement. Concretely, taking estimation of off-the-shelf unsupervised approach as pseudo labels, our insight locates at defining a confidence selection mechanism to extract relative good matches, and then add diverse data augmentation for distilling adequate and reliable knowledge from teacher to student. Thanks to the decouple nature of our method, we can choose a stronger student architecture for sufficient learning. Finally, better student prediction is adopted to transfer knowledge back to the efficient teacher without additional costs in real deployment. Rather than formulating it as a supervised task, we find that introducing an extra unsupervised term for multi-target learning achieves best final results. Extensive experiments show that our approach, termed MDFlow, achieves state-of-the-art real-time accuracy and generalization ability on challenging benchmarks. Code is available at https://github.com/ltkong218/MDFlow.

show abstract

STFlow: Self-Taught Optical Flow Estimation Using Pseudo Labels

Cited by 17 publications

References 30 publications

Learning Comprehensive Motion Representation for Action Recognition

Learning Comprehensive Motion Representation for Action Recognition

Learning Comprehensive Motion Representation for Action Recognition

MDFlow: Unsupervised Optical Flow Learning by Reliable Mutual Knowledge Distillation

Contact Info

Product

Resources

About