Factors affecting the labelling accuracy of brain MRI studies relevant for deep learning abnormality detection

Abstract

Unlocking the vast potential of deep learning-based computer vision classification systems necessitates large data sets for model training. Natural Language Processing (NLP)—involving automation of dataset labelling—represents a potential avenue to achieve this. However, many aspects of NLP for dataset labelling remain unvalidated. Expert radiologists manually labelled over 5,000 MRI head reports in order to develop a deep learning-based neuroradiology NLP report classifier. Our results demonstrate that binary labels (normal vs. abnormal) showed high rates of accuracy, even when only two MRI sequences (T2-weighted and those based on diffusion weighted imaging) were employed as opposed to all sequences in an examination. Meanwhile, the accuracy of more specific labelling for multiple disease categories was variable and dependent on the category. Finally, resultant model performance was shown to be dependent on the expertise of the original labeller, with worse performance seen with non-expert vs. expert labellers.

Optimising brain age estimation through transfer learning: A suite of pre‐trained foundation models for improved performance and generalisability in a clinical setting

“…A subset of 'radiologically normal for age' examinations was identified using a dedicated transformer-based neuroradiology report classifier (Wood et al, 2020(Wood et al, , 2021Wood et al, 2022b). This model was trained using a large dataset of neuroradiology reports from KCH (N = 5000) which had been annotated by a team of five expert neuroradiologists (UK consultant grade; US attending equivalent) as either 'radiologically normal for age' or 'radiologically abnormal for age' based on well-defined criteria (Benger et al, 2023;Wood et al, 2020;Wood et al, 2022b). Briefly, findings that could lead to a subsequent clinical intervention were labelled as 'abnormal' (a referral for case discussion at a multidisciplinary team meeting was considered the minimal intervention).…”