CoCoNet: Coupled Contrastive Learning Network with Multi-level Feature Ensemble for Multi-modality Image Fusion

Fusion of Single and Integral Multispectral Aerial Images

“…While some of the evaluated fusion methods support only two input channels [16,[19][20][21], the work in [13] scales to multiple channels. In cases only two channels are supported but three channels could be used, we selected SRGB and IT.…”

“…The work in [19] cannot accurately estimate complementary information between input images if target objects are discriminative in both images. The architecture in [21] fails in generating the saliency mask, as it was only trained on a small dataset [32] for extracting the mask, which leads to a poor fusion output at the end.…”

“…However, despite their remarkable fusion performance over model-based approaches, deeper networks often lose important details due to stacking a series of pooled convolution layers. Transformer architectures [19][20][21][22] can overcome this by extracting complementary features from the input images. With GANs, the problem of a limited number of ground-truth data for infrared-visible image fusion can be overcome.…”

“…For all integral images (IT and IRGB) the synthetic focal plane was set to the ground surface.Figure5illustrates a visual comparison of our approach with the state of the art for wildlife observation (nesting observation with data from[2]) and wildfire detection and monitoring (with data from[29]). Fused are IRGB and color-coded IT for nesting locations with breeding birds at lower layers just below tree crowns (a), and SRGB of tree vegetation and color-coded IT of ground fires (b,c).While some of the evaluated fusion methods support only two input channels[16,[19][20][21], the work in[13] scales to multiple channels. In cases only two channels are supported but three channels could be used, we selected SRGB and IT.…”