Validation and algorithmic audit of a deep learning system for the detection of proximal femoral fractures in patients in the emergency department: a diagnostic accuracy study

The medical algorithmic audit

“…We apply the principles of the FMEA tool, a known mechanism in engineering, to facilitate assessment, prioritisation, and mitigation of risk. For illustrative purposes, an example of an audit for a hip fracture detection algorithm is published as supplementary information in a study by Oakden-Rayner and colleagues, 20 alongside a detailed breakdown of the FMEA. The benefits of performing the FMEA is to initiate and guide a critical thought process, rather than to establish whether the artificial intelligence system is acceptable or unacceptable or to provide certainty that all risks can be anticipated and minimised.…”

“…It should also be known to the user, who decides whether the intended use statement matches the clinical task and clinical pathway in which the algorithm is intended to be deployed. For example, in the hip fracture audit, 20 scoping of the intended use refers to the function of the algorithm (detecting proximal femoral fractures) as well as its integration into a clinical pathway (in which detection leads to admission under an orthopaedic team and booking of further imaging if necessary). Other considerations include any limits on the health-care environment for use (eg, inpatient or outpatient) and the intended users or oversight (eg, health professionals, patients, or autonomous).…”

“…The developer might be able to define, in theory, the intended impact, but the user is better placed to consider this. The hip fracture audit 20 has several intended impacts, including reduced time to admission and surgery, reduced resource use in the emergency department and unnecessary imaging, and downstream improvement in health outcomes. Different users of the same algorithm might have different target impacts specific to their health setting and needs.…”

“…For example, in the hip fracture audit, 20 severity and occurrence were scored between 1 to 4. A severity score of 4 was catastrophic (failure could cause injury or death, and extreme loss of trust) and 1 was minor (no intervention needed and no injury to patient).…”

Topological structure of complex predictions

“…Introduction Deep learning is a successful strategy where a highly parameterized model makes human-like predictions across many fields [1,9,33,44]. Yet challenges in generalization may keep deep learning from use in practice [50,29]. Detailed prediction mechanisms are also difficult to assess directly due to the large collection of model parameters.…”