Multi-Scale Attention Generative Adversarial Networks for Video Frame Interpolation

Abstract

Video frame interpolation is a fundamental task in computer vision. Recent methods usually apply convolutional neural networks to generate intermediate frame with two consecutive frames as inputs. But sometimes existing methods fail to handle with complex motion and long-range dependencies. In this paper, a multi-scale dense attention generative adversarial network is proposed. First, a multi-scale generative adversarial framework is established for video frame interpolation. Generators from coarse to fine can better combine global and local information. Second, an attention module introduced to generator makes network accurately focus on moving objects. Third, a sequence discriminator is designed to improve the ability of capturing spatial and temporal consistency in frame sequence. Experimental results of ablation study prove the effectiveness of our three contributions. And results on several datasets demonstrate that our approach attains higher performance and produce more photo-realistic in-between frame comparing with previous works. INDEX TERMSVideo frame interpolation, generative adversarial networks, multi-scale pyramid, spatial and temporal consistency, sequence discriminator. JIAN XIAO received the B.S. degree in communication engineering from Northeast Electric Power University, China, in 2015. He is currently pursuing the Ph.D. degree in information and communication engineering with Harbin Engineering University. His research interests include image processing, generative adversarial networks, video analysis, and computer vision. XIAOJUN BI received the Ph.D. degree from the

Towards Intelligent Electromagnetic Inverse Scattering Using Deep Learning Techniques and Information Metasurfaces

“…In (35), the function Re[ * ] means taking the real value part of a complex value vector or matrix, and the vector division is element-wise operator. 5) Do one-dimensional search at the direction p n to minimize the objective function:…”

“…Technically, the aforementioned EMIS methods are developed in the context of data postprocessing. Parallelly, the deep learning techniques are gained ever-increasingly attentions mainly in the digital world since the huge success of artificial intelligence (AI) board-game-Go program (AlphaGo) [22], such as in the areas of speech recognition [23], [24], [25], image recognition [26], [27], [28], automatic translation [29], [30], [31], image editing [32], [33], [34], [35], robot control [36], [37], [38], etc. Naturally, the deep learning techniques can be explored to solve the difficulties arising in the current EMIS algorithms.…”

Information metasurfaces and intelligent metasurfaces

“…The basic problem of AI is how to let a machine learn the experience from collected data or interact with the environment, and therefore, a variety of machinelearning and deep-learning algorithms have been developed [117] . Artificial neural networks (ANNs) have proved to be able to handle various intelligent tasks, such as speech recognition [118][119][120] , image recognition [121][122][123] , automatic translation [124][125][126] , image editing [127][128][129][130] , and robot control [131][132][133] . Due to its unparalleled specialty, AI has been integrated into metasurface structure and function designs.…”