A Hybrid Approach to Wide Baseline View Synthesis with Convolutional Neural Networks

Hybrid dual stream blender for wide baseline view synthesis

“…J o u r n a l P r e -p r o o f large baselines with small kernels, it does not solve the problem with the artifacts in the synthesized view. In our previous work [20], we took inspiration from convolutional architectures for image super resolution [43]. Conversely, in this work we take a different approach inspired by recent advances in image-to-image (I2I) translation [21,22].…”

“…By comparison, other image-to-image translation architectures deal with monoscopic images only, so they do not need to address this additional problem. In [20], features extracted from the input views did not undergo any spatial downsampling and they were only concatenated to synthesize the target view. We will experimentally show by ablation study the advantages of blending the features in a spatially subsampled feature space.…”

“…CNN-VB is the method in [20], it shares with the approach proposed here the preliminary algorithmic warping followed by a learnable blending. Namely, first the the textures are warped to the target view position, then holes are disoccluded.…”

“…Large kernels increase the network complexity at deployment time and makes the network prone to overfitting during training. In our previous work [20], we explored the idea of preliminary warping the reference views to the target position, then blending the warped views with the aid of a ConvNet.…”

“…The present work builds upon our previous research [20] retaining the ideas of a hybrid algorithmic-learning scheme where reference views are preliminary warped to the target position and using an inpainting method built around a median filter to handle occlusions. However, this work improves our previous approach under several aspects:…”