Confounds in the Data—Comments on “Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features”

Ahmed, Hamad; Wilbur, Ronnie Β.; Bharadwaj, Hari; Siskind, Jeffrey Mark

doi:10.1109/tpami.2021.3121268

Cited by 5 publications

(2 citation statements)

References 8 publications

(21 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also highlight other issues that are reported in the papers: (1) computational reproducibility (the lack of availability of code, data, and computing environment to reproduce the exact results reported in the paper); (2) data quality (e.g., small size or large amounts of missing data); (3) metric choice (using incorrect metrics for the task at hand, e.g., using accuracy for measuring model performance in the presence of heavy class imbalance); and (4) standard dataset use, where issues are found despite the use of standard datasets in a field. 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 …”

Section: Introductionmentioning

confidence: 99%

Leakage and the reproducibility crisis in machine-learning-based science

Kapoor

Narayanan

2023

Patterns

View full text Add to dashboard Cite

Section: Introductionmentioning

confidence: 99%

Leakage and the reproducibility crisis in machine-learning-based science

Kapoor

Narayanan

2023

Patterns

View full text Add to dashboard Cite

“…The OpenML website [160] provides access to experimental data including data from the University of California, Irvine machine learning database [161] that can be used in pipeline validation studies. For confounds specific to EEG classification see Li, et al [162] and Ahmed, et al [163].…”

Section: Response To Observation 10mentioning

confidence: 99%

Cautionary Observations Concerning the Introduction of Psychophysiological Biomarkers into Neuropsychiatric Practice

Rapp¹,

Cellucci²,

Darmon³

et al. 2022

Preprint

View full text Add to dashboard Cite

The combination of statistical learning technologies with large databases of psychophysiological data has appropriately generated enthusiastic interest in future clinical applicability. It is argued here that this enthusiasm should be tempered with the understanding that significant obstacles must be overcome before the systematic introduction of psychophysiological measures into neuropsychiatric practice becomes possible. The nonspecificity of psychophysiological measures complicates their use in diagnosis. Low test-retest reliability complicates use in longitudinal assessment, and quantitative psychophysiological measures can normalize in response to placebo intervention. Ten cautionary observations are introduced and, in some instances, possible directions for remediation are suggested.

show abstract

Machine learning seizure prediction: one problematic but accepted practice

et al. 2023

View full text Add to dashboard Cite

Objective. Epilepsy is one of the most common neurological disorders and can have a devastating effect on a person's quality of life. As such, the search for markers which indicate an upcoming seizure is a critically important area of research which would allow either on-demand treatment or early warning for people suffering with these disorders. There is a growing body of work which uses machine learning methods to detect pre-seizure biomarkers from electroencephalography (EEG), however the high prediction rates published do not translate into the clinical setting. Our objective is to investigate a potential reason for this. Approach. We conduct an empirical study of a commonly used data labelling method for EEG seizure prediction which relies on labelling small windows of EEG data in temporal groups then selecting randomly from those windows to validate results. We investigate a confound for this approach for seizure prediction and demonstrate the ease at which it can be inadvertently learned by a machine learning system. Main results.We find that non-seizure signals can create decision surfaces for machine learning approaches which can result in false high prediction accuracy on validation datasets. We prove this by training an artificial neural network to learn fake seizures (fully decoupled from biology) in real EEG. Significance. The significance of our findings is that many existing works may be reporting results based on this confound and that future work should adhere to stricter requirements in mitigating this confound. The problematic, but commonly accepted approach in the literature for seizure prediction labelling is potentially preventing real advances in developing solutions for these sufferers. By adhering to the guidelines in this paper future work in machine learning seizure prediction is more likely to be clinically relevant.

show abstract

Confounds in the Data—Comments on “Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features”

Cited by 5 publications

References 8 publications

Leakage and the reproducibility crisis in machine-learning-based science

Leakage and the reproducibility crisis in machine-learning-based science

Cautionary Observations Concerning the Introduction of Psychophysiological Biomarkers into Neuropsychiatric Practice

Machine learning seizure prediction: one problematic but accepted practice

Contact Info

Product

Resources

About