Robust Image Translation and Completion Based on Dual Auto-Encoder With Bidirectional Latent Space Regression

Abstract

Automated image translation and completion is a subject of keen interest due to their impact on image representation, interpretation, and enhancement. To date, a conditional or a dual adversarial framework with a convolutional auto-encoder embedded as a generator is known to offer the best accuracy in image translation. However, although the frequency is excellent, the adversarial framework may suffer from a lack of generality, i.e., the accuracy drops when translating incomplete and corrupted data given as untrained noisy input data. This paper proposes an approach to robust image-to-image translation that offers a high level of generality while keeping accuracy high as well. The proposed approach is referred to here as a dual autoencoder with bidirectional latent space regression or Bi-directionally Associative DualAE, for short. The proposed BA-DualAE is configured with two auto-encoders the individual latent spaces of which are tightly associated by a bidirectional regression network. Once the two auto-encoders are trained independently for their respective domains, and then, the bidirectional regression network is trained to learn the general association between data pairs. With its capability of robust and bidirectional image translation, BA-DualAE performed direct image completion with no iterative search. The experiments with photo-sketch datasets demonstrated that the proposed BA-DualAE is highly robust under incomplete or corrupted data conditions and is far superior to adversarial frameworks in terms of generality and robustness.INDEX TERMS Convolutional neural network, auto-encoder, bidirectional latent space regression, imageto-image translation, generative adversarial network.

Depth Estimation From a Single RGB Image Using Fine-Tuned Generative Adversarial Network

“…Our approach aims to perform an accurate translation of the RGB images into their corresponding depth maps with a FIGURE 4. RGB input testing samples in the first column translated into the depth maps shown in the third column by the proposed and BA-DualAE [29] approaches. The corresponding ground truth depth maps are shown in the second column.…”

“…However, this approach requires the availability of the entire training set at the test time. Image-to-image translations [28][29][30][31][32], which translate images from one domain to another, play a key role in estimating depth from RGB images. Considering image-toimage translation, the authors in [29] recently proposed BA-DualAE, which is composed of two auto-encoders, where the latent spaces of the different domains are linked with a bidirectional regression network.…”

Accurate and Consistent Image-to-Image Conditional Adversarial Network

“…Recently, in [28], BA-DualAEcomprising two AEs with individual latent spaces associated with a bidirectional regression network was proposed. It can translate images between different domains with an additional capability of image completion, proving its generality.…”

“…We used two different quantitative measures, the mean squared error (MSE) and SSIM, for the comparative analysis [10]. First, we randomly selected two batches of input test samples from the NYU [30] and Cityscapes [31] datasets and translated these samples to their corresponding cross-domains by using the proposed approach, the cGAN-based approach [10], and the BA-DualAE-based approach [28]. The sample-wise comparative analysis for the NYU and Cityscapes datasets in terms of the MSE is shown in Figure 7a,b, respectively.…”

“…The per-pixel and per-image SSIM values were higher for the proposed approach than the cGAN-and the BA-DualAE-based approaches for both datasets, showing that the proposed approach outperformed the cGAN-and the BA-DualAE-based approaches. [10] and BA-DualAE [28], respectively.…”