Motion-Aware Dynamic Architecture for Efficient Frame Interpolation

Choi, Myungsub; Lee, Suyong; Kim, Heewon; Lee, Kyoung Mu

doi:10.1109/iccv48922.2021.01358

Cited by 18 publications

(8 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The area of video frame interpolation is much more diverse than these categories though. There is research on using more than two frames as input [6,35,55,65], interpolating footage from event cameras [34,60,62,67], efficient model design [10,11,13], test-time adaptation [9,53], utilizing hybrid imaging systems [49], handling quantization artifacts [61], as well as joint deblurring [55] and joint superresolution [30,64]. Our proposed splatting-based synthesis technique is orthogonal to such research directions.…”

Section: Related Workmentioning

confidence: 99%

Softmax Splatting for Video Frame Interpolation

Niklaus

Liu

2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

294

263

View full text Add to dashboard Cite

Frame interpolation is an essential video processing technique that adjusts the temporal resolution of an image sequence. An effective approach to perform frame interpolation is based on splatting, also known as forward warping. Specifically, splatting can be used to warp the input images to an arbitrary temporal location based on an optical flow estimate. A synthesis network, also sometimes referred to as refinement network, can then be used to generate the output frame from the warped images. In doing so, it is common to not only warp the images but also various feature representations which provide rich contextual cues to the synthesis network. However, while this approach has been shown to work well and enables arbitrary-time interpolation due to using splatting, the involved synthesis network is prohibitively slow. In contrast, we propose to solely rely on splatting to synthesize the output without any subsequent refinement. This splatting-based synthesis is much faster than similar approaches, especially for multi-frame interpolation, while enabling new state-of-the-art results at high resolutions.

show abstract

Section: Related Workmentioning

confidence: 99%

Softmax Splatting for Video Frame Interpolation

Niklaus

Liu

2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

294

263

View full text Add to dashboard Cite

show abstract

“…To improve model efficiency, Ding et al [24] introduce model compression [55]. Spatio-temporal decoding methods are also proposed to directly convert spatio-temporal features into target frames via channel attention [27], [28] or 3D convolutions [29]. However, most of these methods generate outputs at a fixed time, typically halfway between the input images, which limits arbitrary-time interpolation and linearly increases the runtime for multi-frame interpolation.…”

Section: Related Workmentioning

confidence: 99%

“…The referenced research can roughly be categorized into motion-free and motion-based, depending on whether or not cues like optical flow are incorporated [18], [19], [20], [21], [22]. Motion-free models typically rely on kernel prediction [23], [24], [25], [26] or spatio-temporal decoding [27], [28], [29], which are effective but limited to interpolating frames at fixed time steps and their runtime increases linearly in the number of desired output frames. On the other end of the spectrum, motion-based approaches establish dense correspondences between frames and apply warping to render the intermediate pixels.…”

Section: Introductionmentioning

confidence: 99%

Many-to-many Splatting for Efficient Video Frame Interpolation

Niklaus

Sclaroff

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

In this work, we first propose a fully differentiable Many-to-Many (M2M) splatting framework to interpolate frames efficiently. Given a frame pair, we estimate multiple bidirectional flows to directly forward warp the pixels to the desired time step before fusing any overlapping pixels. In doing so, each source pixel renders multiple target pixels and each target pixel can be synthesized from a larger area of visual context, establishing a many-to-many splatting scheme with robustness to undesirable artifacts. For each input frame pair, M2M has a minuscule computational overhead when interpolating an arbitrary number of in-between frames, hence achieving fast multi-frame interpolation. However, directly warping and fusing pixels in the intensity domain is sensitive to the quality of motion estimation and may suffer from less effective representation capacity. To improve interpolation accuracy, we further extend an M2M++ framework by introducing a flexible Spatial Selective Refinement (SSR) component, which allows for trading computational efficiency for interpolation quality and vice versa. Instead of refining the entire interpolated frame, SSR only processes difficult regions selected under the guidance of an estimated error map, thereby avoiding redundant computation. Evaluation on multiple benchmark datasets shows that our method is able to improve the efficiency while maintaining competitive video interpolation quality, and it can be adjusted to use more or less compute as needed.

show abstract

“…Recent work has explored a few strategies for improving the performance of such methods. These efforts include utilizing additional contextual information to interpolate high-quality results [32], developing unsupervised techniques by cycle consistency [39], detecting the occlusion by exploring the depth information [2], forwardwarping input frames using softmax splatting [33], using quadratic interpolation to overcome the limitation of linear models [24,50], leveraging the distillation loss to supervise the intermediate flows [17], and constructing efficient architectures for large resolution images [10,43]. We note that methods built upon convolutional networks generally face challenges of modeling long-term dependencies thus limiting large motion handling.…”

Section: Video Frame Interpolationmentioning

confidence: 99%

Video Frame Interpolation with Transformer

Lu¹,

Wu²,

Lin³

et al. 2022

Preprint

View full text Add to dashboard Cite

Video frame interpolation (VFI), which aims to synthesize intermediate frames of a video, has made remarkable progress with development of deep convolutional networks over past years. Existing methods built upon convolutional networks generally face challenges of handling large motion due to the locality of convolution operations. To overcome this limitation, we introduce a novel framework, which takes advantage of Transformer to model long-range pixel correlation among video frames. Further, our network is equipped with a novel cross-scale window-based attention mechanism, where cross-scale windows interact with each other. This design effectively enlarges the receptive field and aggregates multi-scale information. Extensive quantitative and qualitative experiments demonstrate that our method achieves new state-of-the-art results on various benchmarks. The source code is available at https://github.com/dvlab-research/VFIformer.

show abstract

Motion-Aware Dynamic Architecture for Efficient Frame Interpolation

Cited by 18 publications

References 35 publications

Softmax Splatting for Video Frame Interpolation

Softmax Splatting for Video Frame Interpolation

Many-to-many Splatting for Efficient Video Frame Interpolation

Video Frame Interpolation with Transformer

Contact Info

Product

Resources

About