On Modelling Label Uncertainty in Deep Neural Networks: Automatic Estimation of Intra- Observer Variability in 2D Echocardiography Quality Assessment

Abstract

Uncertainty of labels in clinical data resulting from intra-observer variability can have direct impact on the reliability of assessments made by deep neural networks. In this paper, we propose a method for modelling such uncertainty in the context of 2D echocardiography (echo), which is a routine procedure for detecting cardiovascular disease at point-of-care. Echo imaging quality and acquisition time is highly dependent on the operator's experience level. Recent developments have shown the possibility of automating echo image quality quantification by mapping an expert's assessment of quality to the echo image via deep learning techniques. Nevertheless, the observer variability in the expert's assessment can impact the quality quantification accuracy. Here, we aim to model the intra-observer variability in echo quality assessment as an aleatoric uncertainty modelling regression problem with the introduction of a novel method that handles the regression problem with categorical labels. A key feature of our design is that only a single forward pass is sufficient to estimate the level of uncertainty for the network output. Compared to the 0.11±0.09 absolute error (in a scale from 0 to 1) archived by the conventional regression method, the proposed method brings the error down to 0.09 ± 0.08, where the improvement is statistically significant and equivalents to 5.7% test accuracy improvement. The simplicity of the proposed approach means that it could be generalized to other applications of deep learning in medical imaging, where there is often uncertainty in clinical labels.

Artificial intelligence versus expert: a comparison of rapid visual inferior vena cava collapsibility assessment between POCUS experts and a deep learning algorithm

“…As with many medical and non‐medical tasks, deep learning‐based automation should significantly decrease the amount of training required for nurses in this task. Additionally, evidence shows implemented artificial intelligence can significantly improve sonographer intra‐observer reliability during cardiac measurements, suggesting artificial intelligence should accomplish this for novice ultrasound users also 10 …”

“…Automation of the process holds the potential to improve inter-rater reliability and even automating the steps of performing and documenting the ultrasound examination in evaluating the IVC. 10 Artificial intelligence is rapidly infiltrating modern medicine. Deep learning, a branch of artificial intelligence, is currently the most promising application for medical image analysis and interpretation.…”

“…Although not surprising, this raises the question of reliability in clinical situations and also highlights the workload burden imposed on novices if they have to freeze images, move forward and backward 1 frame at a time through a cine loop to find maximal and minimal diameters of an IVC, and then carefully measure at the same location. Automation of the process holds the potential to improve inter‐rater reliability and even automating the steps of performing and documenting the ultrasound examination in evaluating the IVC 10 …”

Steps to use artificial intelligence in echocardiography

“…An overview of the studies for the diagnostic ability of current deep learning models in the field of echocardiography in Table 1 [16][17][18][19][20][21][22][23][24][25][26][27]. The accuracy of AI models has been achieved around 80-90%.…”

“…A recent paper showed that the accuracy of AI-based classification for image quality was excellent (score error: 0.11 ± 0.09). This proposed approach could also be generalized to other images involving deep learning in the cardiovascular field, where there are frequent gaps in clinical labeling [16].…”