HyperTDP-Net: A Hyper-densely Connected Compression-and-Decomposition Network Based on Trident Dilated Perception for PET and MRI Image Fusion

Abstract

Conventional methods based on deep learning for medical image fusion usually only connect source images by convolution operations of separate paths to extract local features, without considering their global features, which often leads to the problem of unclear detail information in the final fusion images. Toward this end, we propose a novel end-to-end fusion model for PET and MRI images to achieve information interaction between different pathways, termed as Hyper-densely connected compression-and-decomposition Network based on Trident Dilated Perception for PET and MRI Image Fusion (HyperTDP-Net). In particular, in the compression network, a dual residual hyper densely module is constructed to take full advantage of middle layer information. Moreover, we establish the trident dilated perception module to precisely determine the location information of features, and improve the feature representation capability of the network. In addition, we abandon the ordinary mean square error as the content loss function and propose a new content-aware loss consisting of structural similarity loss and gradient loss, so that the fused image not only contains rich texture details but also maintains sufficient structural similarity with the source images. Abundant experiments demonstrate that our HyperTDP-Net obtains significant fusion performance, which exceeds other advanced fusion methods in terms of qualitative visual description and quantitative assessment.