Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis

Abstract

Transfer learning from natural image to medical image has established as one of the most practical paradigms in deep learning for medical image analysis. However, to fit this paradigm, 3D imaging tasks in the most prominent imaging modalities (e.g., CT and MRI) have to be reformulated and solved in 2D, losing rich 3D anatomical information and inevitably compromising the performance. To overcome this limitation, we have built a set of models, called Generic Autodidactic Models, nicknamed Models Genesis, because they are created ex nihilo (with no manual labeling), self-taught (learned by self-supervision), and generic (served as source models for generating application-specific target models). Our extensive experiments demonstrate that our Models Genesis significantly outperform learning from scratch in all five target 3D applications covering both segmentation and classification. More importantly, learning a model from scratch simply in 3D may not necessarily yield performance better than transfer learning from ImageNet in 2D, but our Models Genesis consistently top any 2D approaches including fine-tuning the models pre-trained from ImageNet as well as fine-tuning the 2D versions of our Models Genesis, confirming the importance of 3D anatomical information and significance of our Models Genesis for 3D medical imaging. This performance is attributed to our unified self-supervised learning framework, built on a simple yet powerful observation: the sophisticated yet recurrent anatomy in medical images can serve as strong supervision signals for deep models to learn common anatomical representation automatically via selfsupervision. As open science, all pre-trained Models Genesis are available at https://github.com/MrGiovanni/ModelsGenesis.

Toward data‐efficient learning: A benchmark for COVID‐19 CT lung and infection segmentation

“…In knowledge transfer task (Task 3), simply fusing non-COVID-19 and COVID-19 dataset with the SOTA network could bias the model to learn more non-COVID-19 features. One can use more robust and powerful domain adaptation methods to handle heterogeneous datasets, such as self-supervised learning [43], cross-domain adaptation [44], [45].…”

Efficient and Effective Training of COVID-19 Classification Networks With Self-Supervised Dual-Track Learning to Rank

“…Zhuang et al [12] proposed to pre-train 3D networks by playing a Rubik's cube game, which can be seen as an extension of 2D jigsaw puzzles [35]. Zhou et al [36] formulated a content restoration pretext task for 3D medical volumes, which achieved impressive improvements on multiple medical image processing tasks. In this study, we extend the Rubik's cube approach by adding a random masking operation, which improves the robustness of the self-supervised feature representation.…”

Deep learning applications in pulmonary medical imaging: recent updates and insights on COVID-19

“…Unsupervised learning can thus be the solution, even though training data is becoming more relevant. It is early to have public datasets fully satisfying the needs of deep learning models, particularly for developing countries, but promising contributions can be expanded to different image modalities and populations, as in [119] which do self-supervised learning covering segmentation and classification in 2D and 3D versions can be expanded.…”

“…At a larger scale, models genesis developed by Zhou et al [119] aimed to generate powerful application-specific target models through transfer learning. Models genesis is a collection of generic source models built directly from unlabeled 3D image 3D data with a unified self-supervised method.…”