Reduced accuracy of MRI deep grey matter segmentation in multiple sclerosis: an evaluation of four automated methods against manual reference segmentations in a multi-center cohort

Abstract

Background Deep grey matter (DGM) atrophy in multiple sclerosis (MS) and its relation to cognitive and clinical decline requires accurate measurements. MS pathology may deteriorate the performance of automated segmentation methods. Accuracy of DGM segmentation methods is compared between MS and controls, and the relation of performance with lesions and atrophy is studied. Methods On images of 21 MS subjects and 11 controls, three raters manually outlined caudate nucleus, putamen and thalamus; outlines were combined by majority voting. FSL-FIRST, FreeSurfer, Geodesic Information Flow and volBrain were evaluated. Performance was evaluated volumetrically (intra-class correlation coefficient (ICC)) and spatially (Dice similarity coefficient (DSC)). Spearman's correlations of DSC with global and local lesion volume, structure of interest volume (ROIV), and normalized brain volume (NBV) were assessed. Results ICC with manual volumes was mostly good and spatial agreement was high. MS exhibited significantly lower DSC than controls for thalamus and putamen. For some combinations of structure and method, DSC correlated negatively with lesion volume or positively with NBV or ROIV. Lesion-filling did not substantially change segmentations. Conclusions Automated methods have impaired performance in patients. Performance generally deteriorated with higher lesion volume and lower NBV and ROIV, suggesting that these may contribute to the impaired performance.

Manual and automated tissue segmentation confirm the impact of thalamus atrophy on cognition in multiple sclerosis: A multicenter study

“…This is probably due to their low contrast compared to surrounding tissue in T1-weighted MRI, which makes it more complicated to trace the edges of the thalamus in these subregions, also manually. The Bland Altman plots revealed that thalamus volumes were on average overestimated by FSL-FIRST and FreeSurfer (excepted left thalamus measurements), while they were systematically underestimated by CAT12, GIF and VolBrain, which is in line with an earlier publication on this topic ( de Sitter et al, 2020 ). It appeared that the absolute agreement for CAT12 (ICC: 0.20–0.21), GIF and VolBrain (ICCs between 0.39 and 0.47) in our study were much worse than previously reported by de Sitter et al (2020) .…”

“…The Bland Altman plots revealed that thalamus volumes were on average overestimated by FSL-FIRST and FreeSurfer (excepted left thalamus measurements), while they were systematically underestimated by CAT12, GIF and VolBrain, which is in line with an earlier publication on this topic ( de Sitter et al, 2020 ). It appeared that the absolute agreement for CAT12 (ICC: 0.20–0.21), GIF and VolBrain (ICCs between 0.39 and 0.47) in our study were much worse than previously reported by de Sitter et al (2020) . However, different study populations and combined manual segmentations created by majority voting were used in previous work.…”

“…and other physiological / pathological factors (e.g., age, sex, hydration, vascular risk factors etc.) ( Amiri et al, 2018 , de Sitter et al, 2020 , Gelineau-Morel et al, 2012 , Rocca et al, 2017a , Rocca et al, 2017b , Sastre-Garriga et al, 2020 ). Given the previously reported limitations of image analysis methods, it is important to understand how consistent and reliable the association between thalamus atrophy and cognition is when using different segmentation approaches in MS patients.…”

Development and evaluation of a manual segmentation protocol for deep grey matter in multiple sclerosis: Towards accelerated semi-automated references

“…A possible limitation of this study was that we did not compare FASTSURF with other existing automated segmentation techniques However, two other studies that were recently published by our group already evaluated existing automated segmentations methods against manual references, using (partly) the same dataset ( de Sitter et al, 2020 , Burggraaff et al, 2020 ). Moreover, since this comparison would reveal any systematic difference between methods, e.g.…”

“…Current state-of-the-art and frequently used automated segmentation methods suffer from substantial limitations with respect to both reproducibility and accuracy, which is partly due to the presence of MS pathological changes. ( Popescu et al, 2014 , Popescu et al, 2016 , Gelineau-Morel et al, 2012 , Meijerman et al, 2018 , Amiri et al, 2018 , de Sitter et al, 2020 ) Specifically, there are various confounds that can affect the measurement of dGM atrophy: image registration and segmentation can be negatively affected by the presence of white matter lesions, ( Gelineau-Morel et al, 2012 , de Sitter et al, 2020 ) generalized or local atrophy, or subtle tissue contrast changes ( Amiri et al, 2018 , Westlye et al, 2009 ). To achieve accurate automated dGM segmentation in the presence of MS abnormalities, it is important that new methods are validated against expert reference outlines of dGM in representative MS samples.…”