Multi-Channel Attention Selection GAN With Cascaded Semantic Guidance for Cross-View Image Translation

Abstract

Cross-view image translation is challenging because it involves images with drastically different views and severe deformation. In this paper, we propose a novel approach named Multi-Channel Attention SelectionGAN (Selection-GAN) that makes it possible to generate images of natural scenes in arbitrary viewpoints, based on an image of the scene and a novel semantic map. The proposed SelectionGAN explicitly utilizes the semantic information and consists of two stages. In the first stage, the condition image and the target semantic map are fed into a cycled semantic-guided generation network to produce initial coarse results. In the second stage, we refine the initial results by using a multi-channel attention selection mechanism. Moreover, uncertainty maps automatically learned from attentions are used to guide the pixel loss for better network optimization. Extensive experiments on Dayton [42], CVUSA [44] and Ego2Top [1] datasets show that our model is able to generate significantly better results than the state-of-the-art methods. The source code, data and trained models are available at https://github. com/Ha0Tang/SelectionGAN .

Exocentric to Egocentric Image Generation Via Parallel Generative Adversarial Network

“…We compare our P-GAN with both single-view image generation methods [4,5] and cross-view image generation methods [6,7]. We adopt the same experimental setup as in [4,6,7]. All images are scaled to 256×256.…”

“…In our experiments, we set λ 1 =10, λ 2 =10, λ 3 =100, λ 4 =10, λ 5 =1, λ 6 =1 in Eq. (4), (7), (8) and (9), respectively. The stateof-the-art cross-view generation methods, i.e., X-Fork [6], X-Seq [6] and SelectionGAN [7] utilize segmentation map to facilitate target view image generation.…”

“…(4), (7), (8) and (9), respectively. The stateof-the-art cross-view generation methods, i.e., X-Fork [6], X-Seq [6] and SelectionGAN [7] utilize segmentation map to facilitate target view image generation. To compare with these cross-view methods, we adopt RefineNet [17,18] to generate segmentation maps on Side2Ego and Top2Ego subsets as Fig.…”

“…These samples were randomly selected for visualization purposes. Columns from left to right are: Input, Pix2pix [4], CycleGAN [5], P-GAN (ours), X-Fork [6], X-Seq [6], Selection-GAN [7], P-GAN + Segmentation map (ours), Ground Truth. Table 1: SSIM, PSNR, Sharpness Difference (SD), KL score (KL) and Accuracy of different single-view image generation methods.…”

“…For these metrics except KL score, higher is better. in [6,7]. The generated segmentation maps are used as the conditional input of G 1 and G 2 .…”