Was that so Hard? Estimating Human Classification Difficulty

Abstract

When doctors are trained to diagnose a specific disease, they learn faster when presented with cases in order of increasing difficulty. This creates the need for automatically estimating how difficult it is for doctors to classify a given case. In this paper, we introduce methods for estimating how hard it is for a doctor to diagnose a case represented by a medical image, both when ground truth difficulties are available for training, and when they are not. Our methods are based on embeddings obtained with deep metric learning. Additionally, we introduce a practical method for obtaining ground truth human difficulty for each image case in a dataset using self-assessed certainty. We apply our methods to two different medical datasets, achieving high Kendall rank correlation coefficients on both, showing that we outperform existing methods by a large margin on our problem and data.

Multi‐modal deep learning for joint prediction of otitis media and diagnostic difficulty

“…It can be seen that the easy cases are in the center of the OME and AOM clusters, while the difficult cases are on the boundary of the cluster or inside other clusters. This shows that there is a relationship between the image embeddings and diagnostic difficulty, as also shown by Hannemose et al 18 Table 1 shows the performance in both tasks: otitis media classification and estimation of the diagnostic difficulty for all proposed models. For classification, both accuracy and class‐wise F1‐scores are reported.…”

“…Classification is performed in the embedding space by predicting the class with the closest training data cluster center to the current test example. Difficulty estimation is performed with the supervised method employing Extra Trees 23 with both embeddings and ground truth labels as input 18 …”

“…These annotations allow finding of the difficulty of each case based on the fraction of correct ENT answers (compared to the original ENT), , and the average self‐evaluated certainty, . The difficulty of each case is then given as 18 : …”

“…Difficulty estimation is performed with the supervised method employing Extra Trees 23 with both embeddings and ground truth labels as input. 18 All networks were trained with Adam optimizer and a learning rate of 0.0001, decreased by a factor of 0.1 every 50th epoch. The…”

“…Combining the diagnosis with the estimated difficulty allows the operator to assess the output of the model and to evaluate whether to redo the otoscopy or WBT or refer the patient to an expert ENT for further examination. Diagnostic difficulty estimation was investigated by Hannemose et al 18 They present several supervised and unsupervised methods for estimating the difficulty from the distribution of the dataset in the embedding space from a metric learning‐based neural network.…”