Conventional methods based on deep learning for medical image fusion usually only connect source images by convolution operations of separate paths to extract local features, without considering their global features, which often leads to the problem of unclear detail information in the final fusion images. Toward this end, we propose a novel end-to-end fusion model for PET and MRI images to achieve information interaction between different pathways, termed as Hyper-densely connected compression-and-decomposition Network based on Trident Dilated Perception for PET and MRI Image Fusion (HyperTDP-Net). In particular, in the compression network, a dual residual hyper densely module is constructed to take full advantage of middle layer information. Moreover, we establish the trident dilated perception module to precisely determine the location information of features, and improve the feature representation capability of the network. In addition, we abandon the ordinary mean square error as the content loss function and propose a new content-aware loss consisting of structural similarity loss and gradient loss, so that the fused image not only contains rich texture details but also maintains sufficient structural similarity with the source images. Abundant experiments demonstrate that our HyperTDP-Net obtains significant fusion performance, which exceeds other advanced fusion methods in terms of qualitative visual description and quantitative assessment.
Traditional fusion approaches and most deep learning-based methods usually generate the intermediate decision map, resulting in detail loss of source images or fusion results. In this work, to enhance the detailed features and structured information from source images, we propose a dual cascade attention network (DCAN) to obtain a more informative fusion image for PET and MRI images. In our approach, channel attention is employed to improve the ability of features representation and spatial attention can highlight informative regions in the proposed fusion network. Additionally, channel and spatial attention are sequential arrangement in channel-first. Moreover, to achieve good performance in the procedure of feature extraction and image reconstruction, two-stage training strategy is adopted to train our fusion model. Experimental results demonstrate that the proposed approach achieves remarkable performance for PET and MRI images fusion.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.