Learning to Find Unpaired Cross-Spectral Correspondences

Jeong, Somi; Kim, Seungryong; Park, Kihong; Sohn, Kwanghoon

doi:10.1109/tip.2019.2917864

Cited by 10 publications

(6 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We compare the proposed method with state-of-the-art methods for image-to-image translation: CycleGAN [37], FUCC [22], and the spectral translation network (STN) [17]. Cy-cleGAN [37] has been widely used in other image-to-image translation fields, while FUCC [22] is an unpaired image training method based on hidden features. Fig.…”

Section: ) Pseudo Nir Generationmentioning

confidence: 99%

See 1 more Smart Citation

Joint Disparity Estimation and Pseudo NIR Generation From Cross Spectral Image Pairs

Duan

Jung

2022

IEEE Access

View full text Add to dashboard Cite

Cross spectral stereo matching is a challenging task due to different spectral properties causing unreliable results in correspondence estimation. In this paper, we propose joint disparity estimation and pseudo near infrared (NIR) generation from cross spectral image pairs. To bridge the spectral gap between paired images, we adopt differential map operations and non-local blocks to improve the local attention and global attention of the network. The proposed network is based on unsupervised learning that consists of one encoder and two decoders, which performs both spectral translation and disparity estimation. For cooperative learning, we use difference map operation to connect two decoders, thus improving the inference ability of the decoder in regions even with large spectral differences. Experimental results show that the proposed network achieves good performance in cross spectral stereo matching for unreliable regions such as shadows and glasses. Moreover, the proposed network generates pseudo NIR images nearly the same as the ground truth even in the regions with large spectral difference. Besides, we achieve real-time speed of 27 FPS for 582×429 image pairs on RTX 2060 6G GPU due to the low computational complexity.

show abstract

Section: ) Pseudo Nir Generationmentioning

confidence: 99%

“…Then, we fix the pseudo NIR generation method to get the disparity estimation results. When we use CybleGAN [37] and FUCC [22] for pseudo NIR generation, DASC [21] achieves the best PSNR performance by 10.47dB and 11.1dB, respectively. When we use STN [17] for pseudo NIR generation, DMC has the best PSNR performance by 22.51dB.…”

Section: ) Ablation Studymentioning

confidence: 99%

Joint Disparity Estimation and Pseudo NIR Generation From Cross Spectral Image Pairs

Duan

Jung

2022

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Generative approaches for RGB and NIR use a cycleGAN to match across generated stereo pairs for both spectra [42], [43]. Performance of these image-to-image translation approaches are subject to spectral similarity.…”

Section: B Machine Learning and Self-supervised Trainingmentioning

confidence: 99%

“…Performance of these image-to-image translation approaches are subject to spectral similarity. Jeong et al show that for unsynchronized pairs, most approaches score lower on FIR compared to NIR when making feature-based comparisons [43]. We use synchronized pairs and assume all geometric differences are caused by the camera viewpoints.…”

Section: B Machine Learning and Self-supervised Trainingmentioning

confidence: 99%

There and Back Again: Self-supervised Multispectral Correspondence Estimation

Walters¹,

Méndez²,

Johnson³

et al. 2021

Preprint

View full text Add to dashboard Cite

Across a wide range of applications, from autonomous vehicles to medical imaging, multi-spectral images provide an opportunity to extract additional information not present in color images. One of the most important steps in making this information readily available is the accurate estimation of dense correspondences between different spectra.Due to the nature of cross-spectral images, most correspondence solving techniques for the visual domain are simply not applicable. Furthermore, most cross-spectral techniques utilize spectra-specific characteristics to perform the alignment. In this work, we aim to address the dense correspondence estimation problem in a way that generalizes to more than one spectrum. We do this by introducing a novel cycle-consistency metric that allows us to self-supervise. This, combined with our spectraagnostic loss functions, allows us to train the same network across multiple spectra.We demonstrate our approach on the challenging task of dense RGB-FIR correspondence estimation. We also show the performance of our unmodified network on the cases of RGB-NIR and RGB-RGB, where we achieve higher accuracy than similar self-supervised approaches. Our work shows that crossspectral correspondence estimation can be solved in a common framework that learns to generalize alignment across spectra.

show abstract

“…Li et al [53] proposed knowledge transfer between unpaired CT and X-ray images based on cycle-consistency loss, facilitating chest X-ray image decomposition. Jeong et al [54] adopted the structure of Cycle-GAN and explored the use of cross-spectral correspondence between visible and infrared images in an unpaired setting. However, noise characteristics and data range representation are fundamentally different when using depth and rgb data.…”

Section: B Domain Adaptationmentioning

confidence: 99%

Coupled Real-Synthetic Domain Adaptation for Real-World Deep Depth Enhancement

Guo

Deligianni

et al. 2020

IEEE Trans. on Image Process.

View full text Add to dashboard Cite

Advances in depth sensing technologies have allowed simultaneous acquisition of both color and depth data under different environments. However, most depth sensors have lower resolution than that of the associated color channels and such a mismatch can affect applications that require accurate depth recovery. Existing depth enhancement methods use simplistic noise models and cannot generalize well under real-world conditions. In this paper, a coupled real-synthetic domain adaptation method is proposed, which enables domain transfer between high-quality depth simulators and real depth camera information for super-resolution depth recovery. The method first enables the realistic degradation from synthetic images, and then enhances degraded depth data to high quality with a color-guided sub-network. The key advantage of the work is that it generalizes well to real-world datasets without further training or fine-tuning. Detailed quantitative and qualitative results are presented, and it is demonstrated that the proposed method achieves improved performance compared to previous methods fine-tuned on the specific datasets.

show abstract

Learning to Find Unpaired Cross-Spectral Correspondences

Cited by 10 publications

References 33 publications

Joint Disparity Estimation and Pseudo NIR Generation From Cross Spectral Image Pairs

Joint Disparity Estimation and Pseudo NIR Generation From Cross Spectral Image Pairs

There and Back Again: Self-supervised Multispectral Correspondence Estimation

Coupled Real-Synthetic Domain Adaptation for Real-World Deep Depth Enhancement

Contact Info

Product

Resources

About