SCGAN: Saliency Map-Guided Colorization With Generative Adversarial Network

Zhao, Yuzhi; Po, Lai-Man; Cheung, Kmc; Yu, Wing Yiu; Rehman, Yasar Abbas Ur

doi:10.1109/tcsvt.2020.3037688

Cited by 44 publications

(34 citation statements)

References 92 publications

(180 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For object recognition performance, Table 2 shows that our method performs better on PQ, SQ, RQ, PQ Th , SQ Th , RQ Th than other image-level image translation methods (i.e., pix2pix [9], MUNIT [7] , TIC-CGAN [19] and BicycleGAN [43]). Although the object-level method (i.e., SCGAN [40]) achieved the best scores in PQ, SQ, RQ, and PQ St , SQ St , RQ St , our method performed best in the PQ Th , SQ Th , RQ Th . This shows that our method can finely separate the unclear object boundary problem in a complex scene with multiple discrepant objects caused by the overlap of things and stuff.…”

Section: Quantitative Resultsmentioning

confidence: 83%

“…As shown in Figure 7 and Figure 8, we added more experimental results compared our method with various I2I translation models including pix2pix [9], MUNIT [7], TIC-CGAN [19], BicycleGAN [43], and SCGAN [40]. The results show the evaluation of image quality and object recognition performance for the translated images on the thermal-to-color translation task using different methods respectively.…”

Section: Additional Experiments Resultsmentioning

confidence: 99%

“…For competing methods, Pix2Pix [9], MUNIT [7] and BicycleGAN [43] belong to imagelevel I2I translation. As INIT [30] has no open implementation code and SCGAN [40] uses a saliency map for object perception, we use SCGAN as object-level I2I translation. TIC-CGAN [19] tends to show superiority on the thermal-to-color translation task.…”

Section: Methodsmentioning

confidence: 99%

“…DA-GAN [26] learned a deep attention encoder to enable the instance-level correspondences to be discovered consequently. SCGAN [40] and SalG-GAN [10] regarded saliency maps as object perception for object-level I2I translation. Shen et al [30], Su et al [31] and Chen et al [2] combined an object instance-level feature with an imagelevel feature for a higher quality instance-level I2I translation.…”

Section: Related Workmentioning

confidence: 99%

See 3 more Smart Citations

Panoptic-aware Image-to-Image Translation

Zhang¹,

Ratsamee²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Despite remarkable recent progress in image translation, the complex scene with multiple discrepant objects remains a challenging problem. Because the translated images have low fidelity and tiny objects in fewer details and obtain unsatisfactory performance in object recognition. Without the thorough object perception (i.e., bounding boxes, categories, and masks) of the image as prior knowledge, the style transformation of each object will be difficult to track in the image translation process. We propose panopticbased object style-align generative adversarial networks (POSA-GANs) for image-to-image translation together with a compact panoptic segmentation dataset. The panoptic segmentation model is utilized to extract panoptic-level perception (i.e., overlap-removed foreground object instances and background semantic regions in the image). This is utilized to guide the alignment between the object content codes of the input domain image and object style codes sampled from the style space of the target domain. The style-aligned object representations are further transformed to obtain precise boundaries layout for higher fidelity object generation. The proposed method was systematically compared with different competing methods and obtained significant improvement on both image quality and object recognition performance for translated images.

show abstract

Section: Quantitative Resultsmentioning

confidence: 83%

Section: Additional Experiments Resultsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Panoptic-aware Image-to-Image Translation

Zhang¹,

Ratsamee²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Neural networks are used since they have also been shown to mirror the behavior and the neuronal architecture of the early primate visual system [ 8 ]. In fact, with the evolution of neural networks, they have been more and more used for this purpose and, for instance, Generative Adversarial Networks (GANs) are used in [ 11 ] to color salciency maps.…”

Section: Technical Background and Related Workmentioning

confidence: 99%

Video Compression for Screen Recorded Sequences Following Eye Movements

Serrano-Carrasco

Díaz-Honrubia

Cuenca

2021

J Sign Process Syst

View full text Add to dashboard Cite

With the advent of smartphones and tablets, video traffic on the Internet has increased enormously. With this in mind, in 2013 the High Efficiency Video Coding (HEVC) standard was released with the aim of reducing the bit rate (at the same quality) by 50% with respect to its predecessor. However, new contents with greater resolutions and requirements appear every day, making it necessary to further reduce the bit rate. Perceptual video coding has recently been recognized as a promising approach to achieving high-performance video compression and eye tracking data can be used to create and verify these models. In this paper, we present a new algorithm for the bit rate reduction of screen recorded sequences based on the visual perception of videos. An eye tracking system is used during the recording to locate the fixation point of the viewer. Then, the area around that point is encoded with the base quantization parameter (QP) value, which increases when moving away from it. The results show that up to 31.3% of the bit rate may be saved when compared with the original HEVC-encoded sequence, without a significant impact on the perceived quality.

show abstract

Nighttime Thermal Infrared Image Colorization with Dynamic Label Mining

Luo

Cao

2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Stable imaging in adverse environments (e.g., total darkness) makes thermal infrared (TIR) cameras a prevalent option for night scene perception. However, the low contrast and lack of chromaticity of TIR images are detrimental to human interpretation and subsequent deployment of RGB-based vision algorithms. Therefore, it makes sense to colorize the nighttime TIR images by translating them into the corresponding daytime color images (NTIR2DC). Despite the impressive progress made in the NTIR2DC task, how to improve the translation performance of small object classes is under-explored.To address this problem, we propose a generative adversarial network incorporating feedback-based object appearance learning (FoalGAN). Specifically, an occlusion-aware mixup module and corresponding appearance consistency loss are proposed to reduce the context dependence of object translation. As a representative example of small objects in nighttime street scenes, we illustrate how to enhance the realism of traffic light by designing a traffic light appearance loss. To further improve the appearance learning of small objects, we devise a dual feedback learning strategy to selectively adjust the learning frequency of different samples. In addition, we provide pixel-level annotation for a subset of the Brno dataset, which can facilitate the research of NTIR image understanding under multiple weather conditions. Extensive experiments illustrate that the proposed FoalGAN is not only effective for appearance learning of small objects, but also outperforms other image translation methods in terms of semantic preservation and edge consistency for the NTIR2DC task.

show abstract

SCGAN: Saliency Map-Guided Colorization With Generative Adversarial Network

Cited by 44 publications

References 92 publications

Panoptic-aware Image-to-Image Translation

Panoptic-aware Image-to-Image Translation

Video Compression for Screen Recorded Sequences Following Eye Movements

Nighttime Thermal Infrared Image Colorization with Dynamic Label Mining

Contact Info

Product

Resources

About