Generative Adversarial Frontal View to Bird View Synthesis

Zhu, Xinge; Yin, Zhichao; Shi, Jianping; Li, Hongsheng; Lin, Dahua

doi:10.1109/3dv.2018.00059

Cited by 51 publications

(28 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We demonstrate in this paper that we are able to generate reliable, improved IPM for larger scenes than in [16], which are therefore able to directly aid scene understanding tasks. We achieve this in real time using real-world data collected under different conditions with a single front-facing camera.…”

Section: Arxiv:181200913v2 [Cscv] 2 May 2019mentioning

confidence: 91%

“…State-of-the-art approaches for cross-domain image translation tasks train (conditional) Generative Adversarial Networks (GANs) to transform images to a new domain [14], [15]. However, these methods are designed to perform aligned appearance transformations and struggle when views change drastically [16]. The latter work, in which a synthetic dataset with perfect ground-truth labels is used to learn IPM, is closest to ours.…”

Section: Arxiv:181200913v2 [Cscv] 2 May 2019mentioning

confidence: 99%

“…Very recently, [16] proposed the first learning approach for IPM using a synthetic dataset. The authors introduced BridgeGAN which employs the homography IPM to bridge the significant appearance gap between the frontal view and bird's-eye view.…”

Section: Arxiv:181200913v2 [Cscv] 2 May 2019mentioning

confidence: 99%

“…As the training labels are created from real-world data (in contrast to the synthetic data of [16]), their quality is limited by several aspects (see examples in Fig. 4):…”

Section: Creating Training Data For Boosted Ipmmentioning

confidence: 99%

See 3 more Smart Citations

The Right (Angled) Perspective: Improving the Understanding of Road Scenes Using Boosted Inverse Perspective Mapping

Bruls

Porav

Kunze

et al. 2019

2019 IEEE Intelligent Vehicles Symposium (IV)

View full text Add to dashboard Cite

Many tasks performed by autonomous vehicles such as road marking detection, object tracking, and path planning are simpler in bird's-eye view. Hence, Inverse Perspective Mapping (IPM) is often applied to remove the perspective effect from a vehicle's front-facing camera and to remap its images into a 2D domain, resulting in a top-down view. Unfortunately, however, this leads to unnatural blurring and stretching of objects at further distance, due to the resolution of the camera, limiting applicability. In this paper, we present an adversarial learning approach for generating a significantly improved IPM from a single camera image in real time. The generated bird'seye-view images contain sharper features (e.g. road markings) and a more homogeneous illumination, while (dynamic) objects are automatically removed from the scene, thus revealing the underlying road layout in an improved fashion. We demonstrate our framework using real-world data from the Oxford Robot-Car Dataset and show that scene understanding tasks directly benefit from our boosted IPM approach.

show abstract

Section: Arxiv:181200913v2 [Cscv] 2 May 2019mentioning

confidence: 91%

Section: Arxiv:181200913v2 [Cscv] 2 May 2019mentioning

confidence: 99%

Section: Arxiv:181200913v2 [Cscv] 2 May 2019mentioning

confidence: 99%

“…As the training labels are created from real-world data (in contrast to the synthetic data of [16]), their quality is limited by several aspects (see examples in Fig. 4):…”

Section: Creating Training Data For Boosted Ipmmentioning

confidence: 99%

See 2 more Smart Citations

The Right (Angled) Perspective: Improving the Understanding of Road Scenes Using Boosted Inverse Perspective Mapping

Bruls

Porav

Kunze

et al. 2019

2019 IEEE Intelligent Vehicles Symposium (IV)

View full text Add to dashboard Cite

show abstract

“…Cross-view relations have been explored in [46,32,11] with more challenging settings of aerial and ground views, where there is minimum semantic and viewpoint overlap between the objects in the images. Cross-view image synthesis between these contrasting domains has attracted wide interests lately [32,33,9,49] with the popularity of GANs; these works have been successful in image translation between aerial and ground-level cropped (single camera) images. Zhai et al [46] explored the possibility of synthesizing ground-level panorama from ground semantic layouts wherein the layouts were predicted from the semantic maps of the aerial images.…”

Section: Domain Transfer and Gansmentioning

confidence: 99%

Bridging the Domain Gap for Ground-to-Aerial Image Matching

Regmi

Shah

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

137

View full text Add to dashboard Cite

The visual entities in cross-view (e.g. ground and aerial) images exhibit drastic domain changes due to the differences in viewpoints each set of images is captured from. Existing state-of-the-art methods address the problem by learning view-invariant images descriptors. We propose a novel method for solving this task by exploiting the generative powers of conditional GANs to synthesize an aerial representation of a ground-level panorama query and use it to minimize the domain gap between the two views. The synthesized image being from the same view as the reference (target) image, helps the network to preserve important cues in aerial images following our Joint Feature Learning approach. We fuse the complementary features from a synthesized aerial image with the original groundlevel panorama features to obtain a robust query representation. In addition, we employ multi-scale feature aggregation in order to preserve image representations at different scales useful for solving this complex task. Experimental results show that our proposed approach performs significantly better than the state-of-the-art methods on the challenging CVUSA dataset in terms of top-1 and top-1% retrieval accuracies. Furthermore, we evaluate the generalization of the proposed method for urban landscapes on our newly collected cross-view localization dataset with geo-reference information.

show abstract

Penalizing Top Performers: Conservative Loss for Semantic Segmentation Adaptation

Zhu

Zhou

Yang

et al. 2018

Computer Vision – ECCV 2018

Self Cite

View full text Add to dashboard Cite

Due to the expensive and time-consuming annotations (e.g., segmentation) for real-world images, recent works in computer vision resort to synthetic data. However, the performance on the real image often drops significantly because of the domain shift between the synthetic data and the real images. In this setting, domain adaptation brings an appealing option. The effective approaches of domain adaptation shape the representations that (1) are discriminative for the main task and (2) have good generalization capability for domain shift. To this end, we propose a novel loss function, i.e., Conservative Loss, which penalizes the extreme good and bad cases while encouraging the moderate examples. More specifically, it enables the network to learn features that are discriminative by gradient descent and are invariant to the change of domains via gradient ascend method. Extensive experiments on synthetic to real segmentation adaptation show our proposed method achieves state of the art results. Ablation studies give more insights into properties of the Conservative Loss. Exploratory experiments and discussion demonstrate that our Conservative Loss has good flexibility rather than restricting an exact form.

show abstract

Generative Adversarial Frontal View to Bird View Synthesis

Cited by 51 publications

References 47 publications

The Right (Angled) Perspective: Improving the Understanding of Road Scenes Using Boosted Inverse Perspective Mapping

The Right (Angled) Perspective: Improving the Understanding of Road Scenes Using Boosted Inverse Perspective Mapping

Bridging the Domain Gap for Ground-to-Aerial Image Matching

Penalizing Top Performers: Conservative Loss for Semantic Segmentation Adaptation

Contact Info

Product

Resources

About