Brain Medical Image Fusion Based on Dual-Branch CNNs in NSST Domain

Abstract

Computed tomography (CT) images show structural features, while magnetic resonance imaging (MRI) images represent brain tissue anatomy but do not contain any functional information. How to effectively combine the images of the two modes has become a research challenge. In this paper, a new framework for medical image fusion is proposed which combines convolutional neural networks (CNNs) and non-subsampled shearlet transform (NSST) to simultaneously cover the advantages of them both. This method effectively retains the functional information of the CT image and reduces the loss of brain structure information and spatial distortion of the MRI image. In our fusion framework, the initial weights integrate the pixel activity information from two source images that is generated by a dual-branch convolutional network and is decomposed by NSST. Firstly, the NSST is performed on the source images and the initial weights to obtain their low-frequency and high-frequency coefficients. Then, the first component of the low-frequency coefficients is fused by a novel fusion strategy, which simultaneously copes with two key issues in the fusion processing which are named energy conservation and detail extraction. The second component of the low-frequency coefficients is fused by the strategy that is designed according to the spatial frequency of the weight map. Moreover, the high-frequency coefficients are fused by the high-frequency components of the initial weight. Finally, the final image is reconstructed by the inverse NSST. The effectiveness of the proposed method is verified using pairs of multimodality images, and the sufficient experiments indicate that our method performs well especially for medical image fusion.

Medical Image Fusion Based on Low-Level Features

“…The traditional fusion methods are the Laplacian pyramid- (LAP-) based fusion algorithm, discrete wavelet transform- (DWT-) based fusion algorithm, and non-subsampled contourlet- (NSCT-) based fusion algorithm. The recently proposed fusion methods included cartoon-texture decomposition- (CTD-) based fusion algorithm [ 34 ], multiscale image decomposition- (MSID-) based fusion algorithm [ 35 ], cross-bilateral filtering- (CBF-) based fusion algorithm [ 24 ], and guided filtering fusion- (GFF-) based algorithm [ 22 ]. The compared algorithms and proposed algorithm are all programmed in MATLAB language, and all the experiments are conducted with MATLAB R2011b in a Windows environment, on a computer with an Intel Core (TM) i7-4770 and 4G memory.…”

“…Hou et al [ 23 ] designs a novel fusion scheme for CT and MRI medical images based on convolutional neural networks and a dual-channel spiking cortical model. Ding et al [ 24 ] fused medical images by combining convolutional neural networks and non-subsampled shear-let transform to simultaneously cover the advantages of them both for medical image fusion. Wang et al [ 25 ] proposed a medical image algorithm based on the Siamese convolutional network and contrast pyramid.…”

A Deep Probabilistic Sensing and Learning Model for Brain Tumor Classification With Fusion-Net and HFCMIK Segmentation

“…All conventional approaches are improperly localizing the tumor area. [15], (e) DB-CNN [17], (f) GF-SDL [18], (g) LDNSD [19],…”

Multimodal medical image fusion algorithm based on pulse coupled neural networks and nonsubsampled contourlet transform

“…The AG metric is given by this equation ( 12) PSNR = 10 log 10 (255 -The nonlinear correlation information entropy Q ncie : measures the nonlinear information of the fused image. Q ncie is denoted by the following formula [3]:…”