Style Transfer Using Generative Adversarial Networks for Multi-Site MRI Harmonization

Abstract

Large data initiatives and high-powered brain imaging analyses require the pooling of MR images acquired across multiple scanners, often using different protocols. Prospective cross-site harmonization often involves the use of a phantom or traveling subjects. However, as more datasets are becoming publicly available, there is a growing need for retrospective harmonization, pooling data from sites not originally coordinated together. Several retrospective harmonization techniques have shown promise in removing cross-site image variation. However, most unsupervised methods cannot distinguish between image-acquisition based variability and cross-site population variability, so they require that datasets contain subjects or patient groups with similar clinical or demographic information. To overcome this limitation, we consider cross-site MRI image harmonization as a style transfer problem rather than a domain transfer problem. Using a fully unsupervised deep-learning framework based on a generative adversarial network (GAN), we show that MR images can be harmonized by inserting the style information encoded from a reference image directly, without knowing their site/scanner labels a priori. We trained our model using data from five large-scale multi-site datasets with varied demographics. Results demonstrated that our styleencoding model can harmonize MR images, and match intensity profiles, successfully, without relying on traveling subjects. This model also avoids the need to control for clinical, diagnostic, or demographic information. Moreover, we further demonstrated that if we included diverse enough images into the training set, our method successfully harmonized MR images collected from unseen scanners and protocols, suggesting a promising novel tool for ongoing collaborative studies.

3D Convolutional Neural Networks for Classification of Alzheimer’s and Parkinson’s Disease with T1-Weighted Brain MRI

“…Future work will also assess the added value of additional imaging protocols, such as quantitative MRI, 25 which may boost classification performance. We plan to test the added value of MRI data harmonization approaches based on generative adversarial networks, such as CycleGANs 26 or CALAMITI, which can make MRI data from different scanners or protocols more comparable.…”

“…We plan to perform additional experiments with 2D variants as this might also help to reduce overfitting issues for low data problems such as the tasks defined here. We are also working on creating activation maps as a way to improve interpretability of the model's prediction, and plan to test the added value of MRI data harmonization approaches based on generative adversarial networks, such as CycleGANs 26 or CALAMITI, 27 which can make MRI data from different scanners or protocols more comparable.…”

A Comparison of Methods to Harmonize Cortical Thickness Measurements Across Scanners and Sites

“…There has been a recent explosion in methods that apply machine learning and other advanced multivariate techniques to tackle harmonization. Machine learning methods, including deep-learning approaches, have been developed in recent years to harmonize neuroimaging data without a priori hypotheses about data distributions (Blumberg et al, 2019; Dinsdale et al, 2021; Liu et al, 2021; Moyer et al, 2020; Ning et al, 2020; Tax et al, 2019). A universal machine learning method that is capable of harmonizing data across sites, time, and imaging modalities may enhance our ability to use both existing data and yet to be acquired data (Zuo et al, 2021).…”

Making Radiomics More Reproducible across Scanner and Imaging Protocol Variations: A Review of Harmonization Methods

“…The qualitative results showed that their methods performed better however, in the quantitative evaluation (peak signal to noise ratio and structure similarity index), supervised learning performed better than unsupervised learning. A similar study by Liu et al [ 115 ] was carried out to harmonize MRI images from multiple arbitrary sites using a style transferable GAN. They treated harmonization as a style transfer problem and proved that their model applied to unseen images provided there was enough data available from multiple sites for training purposes.…”