Test–retest reliability of freesurfer measurements within and between sites: Effects of visual approval process

Abstract

In the last decade, many studies have used automated processes to analyze magnetic resonance imaging (MRI) data such as cortical thickness, which is one indicator of neuronal health. Due to the convenience of image processing software (e.g., FreeSurfer), standard practice is to rely on automated results without performing visual inspection of intermediate processing. In this work, structural MRIs of 40 healthy controls who were scanned twice were used to determine the test–retest reliability of FreeSurfer-derived cortical measures in four groups of subjects—those 25 that passed visual inspection (approved), those 15 that failed visual inspection (disapproved), a combined group, and a subset of 10 subjects (Travel) whose test and retest scans occurred at different sites. Test–retest correlation (TRC), intraclass correlation coefficient (ICC), and percent difference (PD) were used to measure the reliability in the Destrieux and Desikan–Killiany (DK) atlases. In the approved subjects, reliability of cortical thickness/surface area/volume (DK atlas only) were: TRC (0.82/0.88/0.88), ICC (0.81/0.87/0.88), PD (0.86/1.19/1.39), which represent a significant improvement over these measures when disapproved subjects are included. Travel subjects’ results show that cortical thickness reliability is more sensitive to site differences than the cortical surface area and volume. To determine the effect of visual inspection on sample size required for studies of MRI-derived cortical thickness, the number of subjects required to show group differences was calculated. Significant differences observed across imaging sites, between visually approved/disapproved subjects, and across regions with different sizes suggest that these measures should be used with caution.

Reproducibility of quantitative structural and physiological MRI measurements

“…Overall whole‐brain average and regional cortical gray matter thickness and volumetric measurements showed excellent ICC and other measures of reproducibility that were consistent with other published results (Iscan et al., 2015; Liem et al., 2015; Yang et al., 2016). The cortical gray matter thickness of the entorhinal, insula, and medial orbitofrontal demonstrated lower reproducibility.…”

“…Therefore, caution should be recommended when interpreting cortical gray matter thickness findings from these areas. The power analyses estimates provided here showed a smaller number of subjects per group ( N  ~ 10) than Liem ( N  = 40) (Liem et al., 2015) but similar to that provided by Iscan ( N  = 19) (Iscan et al., 2015). This is due to a difference in methodology.…”

“…This is due to a difference in methodology. Our approach was based on the variance in the average gray matter (GM) thickness measurements that was also used by Iscan (Iscan et al., 2015). The power analysis by Liem and colleagues provided the number of subjects needed to detect the vertex‐specific difference in mean thickness by accounting for vertex‐wise variance across the surface.…”

Intra-Rater and Inter-Rater Reliability of Brain Surface Intensity Model (BSIM)-Based Cortical Thickness Analysis Using 3T MRI

“…ICCs for cortical thickness measurements and Z-score values indicated excellent reliability (0.751 -0.940) in most brain regions. This is the first study to evaluate the intrarater and inter-rater reliability of BSIM-based cortical (27). Freesurfer and other segmentation-based methods require 3D surface measurements obtained by segmentation of gray matter from white matter in order to calculate cortical thicknesses (28).…”

Breaking down the genetics of depression using brain endophenotypes

“…Brain MRI phenotypes represent ideal candidate endophenotypes for mental health disorders, as they do not rely on self-report [16] and are more and more reliably measured thanks to methodological and technological improvement [18][19][20][21][22][23][24][25]. In addition, the heritability of many MRI brain phenotypes has been established: from structural [26][27][28][29][30], to resting state and task fMRI [31][32][33][34].…”