I2V-GAN: Unpaired Infrared-to-Visible Video Translation

Abstract

Human vision is often adversely affected by complex environmental factors, especially in night vision scenarios. Thus, infrared cameras are often leveraged to help enhance the visual effects via detecting infrared radiation in the surrounding environment, but the infrared videos are undesirable due to the lack of detailed semantic information. In such a case, an effective video-to-video translation method from the infrared domain to the visible light counterpart is strongly needed by overcoming the intrinsic huge gap between infrared and visible fields. To address this challenging problem, we propose an infrared-to-visible (I2V) video translation method I2V-GAN to generate fine-grained and spatial-temporal consistent visible light videos by given unpaired infrared videos. Technically, our model capitalizes on three types of constraints: 1) adversarial constraint to generate synthetic frames that are similar to the real ones, 2) cyclic consistency with the introduced perceptual loss for effective content conversion as well as style preservation, and 3) similarity constraints across and within domains to enhance the content and motion consistency in both spatial and temporal spaces at a finegrained level. Furthermore, the current public available infrared and visible light datasets are mainly used for object detection or tracking, and some are composed of discontinuous images which are not suitable for video tasks. Thus, we provide a new dataset for infrared-to-visible video translation, which is named IRVI. Specifically, it has 12 consecutive video clips of vehicle and monitoring scenes, and both infrared and visible light videos could be apart into 24352 frames. Comprehensive experiments on IRVI validate that I2V-GAN is superior to the compared state-of-the-art methods in the translation of infrared-to-visible videos with higher fluency and finer semantic details. Moreover, additional experimental results on the flower-to-flower dataset indicate I2V-GAN is also applicable to other video translation tasks. The code and IRVI dataset are available at https://github.com/BIT-DA/I2V-GAN.

ROMA: Cross-Domain Region Similarity Matching for Unpaired Nighttime Infrared to Daytime Visible Video Translation

“…Similarly, mocycle-GAN [8] utilizes cycle loss for structural consistency and motion consistency loss based on optical flow to maintain the temporal coherence. Especially, I2V-GAN [24] is a tailored approach for daytime infrared-to-visible Eq. 2 translation based on the Recycle-GAN framework, which additionally proposes perceptual cyclic losses and similarity losses to enhance the spatiotemporal consistency.…”

“…Since a long-ware infrared camera conforms to the requirements of recognition and vehicle driving scenes at night, we utilize the binocular infrared color camera (DTC equipment) to capture nighttime infrared and visible videos. Additionally, a visible camera is [24] which is provided for daytime infrared-to-visible video translation. Besides, we capture these infrared videos at night from three scenes (city, highway, and monitoring scenarios).…”

“…6. We comprehensively select four popular infrared related datasets (IRVI [24], VOT2019-RGBTIR [29], FLIR [13] and KAIST [18]) for comparison and details are shown in Tab. 2.…”

“…Manual labeling is not only time-consuming and expensive, but also prone to introduce errors. Thus, some unpaired video-to-video translation methods [2,3,8,24] loosen the requirements for paired data. In particular, [24] is a framework that adapts for daytime infrared and visible video translation.…”

“…Thus, some unpaired video-to-video translation methods [2,3,8,24] loosen the requirements for paired data. In particular, [24] is a framework that adapts for daytime infrared and visible video translation. The huger domain gap between nighttime infrared and daytime visible data will inevitably diminish their translated performance, since the critical semantic structural correspondence across the two domains are largely overlooked.…”