Association of Clinician Diagnostic Performance With Machine Learning–Based Decision Support Systems

Vasey, Baptiste; Ursprung, Stephan; Beddoe, Benjamin; Taylor, Elliott H; Marlow, Neale; Bilbro, Nicole; Watkinson, Peter; McCulloch, Peter

doi:10.1001/jamanetworkopen.2021.1276

Cited by 63 publications

(59 citation statements)

References 68 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Recent systematic reviews have found that most of the studies on automated diagnosis using artificial intelligence have high risk of bias, mostly due to patient selection methodology and absence of validation on external data [54][55][56] . Systematic reviews on computer-based clinical-decision support systems also highlight the need for more robust patient selection [57][58][59][60][61][62] .…”

Section: Discussionmentioning

confidence: 99%

Computer-assisted analysis of routine EEG to identify hidden biomarkers of epilepsy: protocol for a systematic review

Lemoine

Briard

Rioux

et al. 2022

Preprint

View full text Add to dashboard Cite

Background: The diagnosis of epilepsy frequently relies on the visual interpretation of the electroencephalogram (EEG) by a neurologist. The hallmark of epilepsy on EEG is the interictal epileptiform discharge (IED). This marker lacks sensitivity: it is only captured in a small percentage of 30-minute routine EEGs in patients with epilepsy. In the past three decades, there has been growing interest in the use of computational methods to analyze the EEG without relying on the detection of IEDs, but none have made it to the clinical practice. We aim to review the diagnostic accuracy of quantitative methods applied to ambulatory EEG analysis to guide the diagnosis and management of epilepsy. Methods: The protocol complies with the recommendations for systematic reviews of diagnostic test accuracy by Cochrane. We will search MEDLINE, EMBASE, EBM reviews, IEEE Explore along with grey literature for articles, conference papers and conference abstracts published after 1961. We will include observational studies that present a computational method to analyze the EEG for the diagnosis of epilepsy in adults or children without relying on the identification of IEDs or seizures. The reference standard is the diagnosis of epilepsy by a physician. We will report the estimated pooled sensitivity and specificity, and receiver operating characteristic area-under-the-curve (ROC AUC) for each marker. If possible, we will perform a meta-analysis of the sensitivity and specificity and ROC AUC for each individual marker. We will assess the risk of bias using an adapted QUADAS-2 tool. We will also describe the algorithms used for signal processing, feature extraction and predictive modeling, and comment on the reproducibility of the different studies. Discussion: Despite the promise to unveil epileptiform patterns that cannot be seen by the naked eye, computational analysis of ambulatory EEG has not yet been successfully translated to the clinical setting. We hope to produce recommendations for future studies on computer-assisted EEG interpretation for the diagnosis and management of epilepsy. Systematic review registration: PROSPERO #292261

show abstract

Section: Discussionmentioning

confidence: 99%

Computer-assisted analysis of routine EEG to identify hidden biomarkers of epilepsy: protocol for a systematic review

Lemoine

Briard

Rioux

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…More recently, there has been a surge in interest in using artificial intelligence, and more specifically machine learning (ML), to assist clinicians in both diagnostic reasoning using EMR data 16 and, more notably, correct interpretation of diagnostic investigations that involve imaged data, such as electrocardiograms, radiological films and histopathological slides. 17 While many in silico studies in research settings suggest equivalent or better performance than clinicians, especially less experienced ones, relatively few studies have been conducted in clearly reported, representative clinical work environments, with most showing no difference in performance. Many studies are rated as being at high risk of bias, lack external validation and have not assessed implementation and effectiveness in real-world clinical practice.…”

Section: Using Artificial Intelligencementioning

confidence: 99%

Using information technology to reduce diagnostic error: still a bridge too far?

Scott

2022

Internal Medicine Journal

View full text Add to dashboard Cite

“…The breadth of applications of machine learning (ML) intelligent systems in predictive decision-making scenarios is substantial, and users are interfacing with these assistant systems in a multitude of ways. This human-ML collaboration is seen in a variety of fields, from medical diagnosis and treatment [1,2,3], to automated driving systems [4,5], to threat detection [6]. ML assistance has also been utilized in aiding visual search and predicting the presence of targets in modalities such as baggage screens [7,8,9] and simulated combat images [10].…”

Section: Introductionmentioning

confidence: 99%

The Cognitive Effects of Machine Learning Aid in Domain-Specific and Domain-General Tasks

Divis¹,

Howell²,

Matzen³

et al. 2022

Proceedings of the Annual Hawaii International Conference on System Sciences

View full text Add to dashboard Cite

With machine learning (ML) technologies rapidly expanding to new applications and domains, users are collaborating with artificial intelligence-assisted diagnostic tools to a larger and larger extent. But what impact does ML aid have on cognitive performance, especially when the ML output is not always accurate? Here, we examined the cognitive effects of the presence of simulated ML assistance-including both accurate and inaccurate output-on two tasks (a domain-specific nuclear safeguards task and domain-general visual search task). Patterns of performance varied across the two tasks for both the presence of ML aid as well as the category of ML feedback (e.g., false alarm). These results indicate that differences such as domain could influence users' performance with ML aid, and suggest the need to test the effects of ML output (and associated errors) in the specific context of use, especially when the stimuli of interest are vague or ill-defined.

show abstract

Association of Clinician Diagnostic Performance With Machine Learning–Based Decision Support Systems

Cited by 63 publications

References 68 publications

Computer-assisted analysis of routine EEG to identify hidden biomarkers of epilepsy: protocol for a systematic review

Computer-assisted analysis of routine EEG to identify hidden biomarkers of epilepsy: protocol for a systematic review

Using information technology to reduce diagnostic error: still a bridge too far?

The Cognitive Effects of Machine Learning Aid in Domain-Specific and Domain-General Tasks

Contact Info

Product

Resources

About