Evaluating pointwise reliability of machine learning prediction

Nicora, Giovanna; Rios, Miguel; Abu‐Hanna, Ameen; Bellazzi, Riccardo

doi:10.1016/j.jbi.2022.103996

Cited by 33 publications

(10 citation statements)

References 67 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most classification studies using dFNC and sFNC have given insight into models performance for individual classes [9][10] but not subtypes. Discerning from the data of a patient whether a CDSS is likely to be accurate for them will be vital to ensuring that they receive proper care [2].…”

Section: Introductionmentioning

confidence: 99%

“…If neuroimaging clinical decision support systems (CDSS) are ever to be implemented in a clinical setting, they must be both robust and reliable [1]. One aspect of this reliability is that clinicians and need to not only know whether there are systematic differences in how the model will perform for different patients [2]. Neurological and neuropsychological disorder subtyping could contribute to more reliable CDSS [3] [4].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Identifying Neuropsychiatric Disorder Subtypes and Subtype-dependent Variation in Diagnostic Deep Learning Classifier Performance

Ellis

Miller

Calhoun

2022

Preprint

View full text Add to dashboard Cite

Clinicians and developers of deep learning-based neuroimaging clinical decision support systems (CDSS) need to know whether those systems will perform well for specific individuals. However, relatively few methods provide this capability. Identifying neuropsychiatric disorder subtypes for which CDSS may have varying performance could offer a solution. Dynamic functional network connectivity (dFNC) is often used to study disorders and develop neuroimaging classifiers. Unfortunately, few studies have identified neurological disorder subtypes using dFNC. In this study, we present a novel approach with which we identify 5 states of dFNC activity and 4 schizophrenia subtypes based on their time spent in each state. We also show how the performance of an explainable diagnostic deep learning classifier is subtype-dependent. We lastly examine how the dFNC features used by the classifier vary across subtypes. Our study provides a novel approach for subtyping disorders that (1) has implications for future scientific studies and (2) could lead to more reliable CDSS.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Identifying Neuropsychiatric Disorder Subtypes and Subtype-dependent Variation in Diagnostic Deep Learning Classifier Performance

Ellis

Miller

Calhoun

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…A confusion matrix is a technique for summarizing and describing the performance of classification algorithms on a set of test data for which the true values are known [31]. The accuracy of classification is sometimes misleading due to the difference in the number of observations in each category or the multiplicity of categories in the data set [32]. In our work, we evaluated model performance by measures of accuracy, precision, and recall in ML.…”

Section: Data Preprocessingmentioning

confidence: 99%

Bluetooth beacons based indoor positioning in a shopping malls using machine learning

Maaloul¹,

Lejdel²,

Clementini

et al. 2023

Bulletin EEI

View full text Add to dashboard Cite

The adoption of Bluetooth beacon technology demonstrates a broad interest in indoor positioning technology because of its low cost and ease of use. Bluetooth beacons usually have an accuracy of fewer than 4 meters. The use of machine learning (ML) leads to results with greater accuracy compared to using traditional filtering methods. In this paper, we provide indoor localization based on Bluetooth beacons using several different ML techniques. We used ML algorithms to locate customers' devices in shopping malls. The extra-trees classifier and k-neighbors classifier found the device with greater than 90% accuracy. Other algorithms were able to determine the location with less accuracy. The results also showed that Bluetooth technology is a valid solution to find the data used to analyze the spatial-temporal behavior of individuals.

show abstract

“…simulating dataset shift. This out-of-distribution samples can be exploited to test the robustness, reliability and explainability of ML classifiers [5] .…”

Section: Value Of the Datamentioning

confidence: 99%