Hybrid pre training algorithm of Deep Neural Networks

Abstract

Abstract. This paper proposes a hybrid algorithm of pre training deep networks, using both marked and unmarked data. The algorithm combines and extends the ideas of Self-Taught learning and pre training of neural networks approaches on the one hand, as well as supervised learning and transfer learning on the other. Thus, the algorithm tries to integrate in itself the advantages of each approach. The article gives some examples of applying of the algorithm, as well as its comparison with the classical approach to pre training of neural networks. These examples show the effectiveness of the proposed algorithm.

KIKI‐net: cross‐domain convolutional neural networks for reconstructing undersampled magnetic resonance images

“…The number of layers linearly increases in CD‐CNNs with the number CD‐CNN iterations. For example, the KIKI‐net used in this study, which combines 4 25‐layer CNNs, corresponds to a CNN that consists of 100 layers (25 layers × 4); however, if this extremely deep KIKI‐net is trained in an end‐to‐end manner, there are several factors that degrade the training performance, such as the difficulty of hyperparameter tuning, the vanishing gradient problem due to a larger number of parameters, and graphic processing unit memory shortage. Furthermore, FT and IFT must be included in the networks, which increases the computational complexity of the network and worsen these problems.…”

“…The number of layers linearly increases in CD-CNNs with the number CD-CNN iterations. For example, the KIKInet used in this study, which combines 4 25-layer CNNs, corresponds to a CNN that consists of 100 layers (25 layers 3 4); however, if this extremely deep KIKI-net is trained in an end-to-end manner, there are several factors that degrade the training performance, such as the difficulty of hyperparameter tuning, [38][39][40] the vanishing gradient problem 41,42 due to a larger number of parameters, and graphic processing unit memory shortage. Furthermore, FT and IFT must be included in the networks, which increases the computational complexity of the network and worsen these F IGUR E 6 Reconstruction results from conventional algorithms and KIKI-net at R 5 4 undersampling for the T 2 fluid-attenuated inversion recovery Alzheimer's Disease Neuroimaging Initiative data set (T 2 -FLAIR_-ADNI): fully sampled image (A1); zero-filling image (B1); image reconstructed with compressed-sensing MRI (CS-MRI) (C1); image reconstructed with dictionary learning (DL) MRI (D1); image reconstructed with block-matching and 3D filtering (BM3D) MRI (E1); image reconstructed with Wang's algorithm (F1); image reconstructed with PANO (patch-based nonlocal operator) (G1); image reconstructed with FDLCP (fast dictionary learning method on classified patches) (H1); and image reconstructed with our proposed algorithm, KIKI-net (I1).…”