Overly optimistic prediction results on imbalanced data: a case study of flaws and benefits when applying over-sampling

Vandewiele, Gilles; Dehaene, Isabelle; Kovács, György; Sterckx, Lucas; Janssens, Olivier; Ongenae, Femke; Backere, Femke De; Turck, Filip De; Roelens, Kristien; Decruyenaere, Johan; Hoecke, Sofie Van; Demeester, Thomas

doi:10.1016/j.artmed.2020.101987

Cited by 54 publications

(71 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To computationally deal with this imbalance, we used the SMOTE technique [ 55 ] to oversample the minority class in the training sets after partitioning the data during the learning process. It is essential to oversample after data partitioning to keep the test data representative of the original distribution of the dataset and avoid information leakage that can lead to overly optimistic prediction results [ 56 ].…”

Section: Methodsmentioning

confidence: 99%

Acoustic and language analysis of speech for suicidal ideation among US veterans

et al. 2021

View full text Add to dashboard Cite

Background Screening for suicidal ideation in high-risk groups such as U.S. veterans is crucial for early detection and suicide prevention. Currently, screening is based on clinical interviews or self-report measures. Both approaches rely on subjects to disclose their suicidal thoughts. Innovative approaches are necessary to develop objective and clinically applicable assessments. Speech has been investigated as an objective marker to understand various mental states including suicidal ideation. In this work, we developed a machine learning and natural language processing classifier based on speech markers to screen for suicidal ideation in US veterans. Methodology Veterans submitted 588 narrative audio recordings via a mobile app in a real-life setting. In addition, participants completed self-report psychiatric scales and questionnaires. Recordings were analyzed to extract voice characteristics including prosodic, phonation, and glottal. The audios were also transcribed to extract textual features for linguistic analysis. We evaluated the acoustic and linguistic features using both statistical significance and ensemble feature selection. We also examined the performance of different machine learning algorithms on multiple combinations of features to classify suicidal and non-suicidal audios. Results A combined set of 15 acoustic and linguistic features of speech were identified by the ensemble feature selection. Random Forest classifier, using the selected set of features, correctly identified suicidal ideation in veterans with 86% sensitivity, 70% specificity, and an area under the receiver operating characteristic curve (AUC) of 80%. Conclusions Speech analysis of audios collected from veterans in everyday life settings using smartphones offers a promising approach for suicidal ideation detection. A machine learning classifier may eventually help clinicians identify and monitor high-risk veterans.

show abstract

Section: Methodsmentioning

confidence: 99%

Acoustic and language analysis of speech for suicidal ideation among US veterans

et al. 2021

View full text Add to dashboard Cite

show abstract

Section: Methodsmentioning

confidence: 99%

“…It is essential to oversample after data partitioning to keep the test data representative of the original distribution of the dataset and avoid information leakage that can lead to overly optimistic prediction results. 38 We investigated different supervised classification algorithms on the selected features and evaluated the results. Specifically, we applied six algorithms: logistic regression (LR), random forest (RF), support vector machines (SVM), XGBoost (XGB), k-nearest neighbors (KNN), and deep neural network (DNN).…”

Section: Machine Learning Approachmentioning

confidence: 99%

Acoustic and language analysis of speech for suicide ideation among US veterans

Belouali

Gupta

Sourirajan

et al. 2020

Preprint

View full text Add to dashboard Cite

U.S. veterans are 1.5 times more likely to die by suicide than Americans who never served in the military. Considering such high rates, there is an urgent need to develop innovative approaches for objective and clinically applicable assessments to detect individuals at high risk. We hypothesize that speech in suicidal veterans has a range of distinctive acoustic and linguistic features. The purpose of this work is to build an automated machine learning and natural language processing tool to screen for suicidality. Veterans made 588 narrative audio recordings via a mobile app in a real-life setting. In addition, veterans completed self-report psychiatric scales and questionnaires. Recordings were analyzed to extract voice characteristics including prosodic, phonation, and glottal. The audios were also transcribed to extract textual features for linguistic analysis. We evaluated the acoustic and linguistic features using both statistical significance and ensemble feature selection. We also examined the performance of different machine learning algorithms on multiple combinations of features to classify suicidal and non-suicidal audios. Random Forest classifier correctly identified suicidal ideation in veterans based on the combined set of acoustic and linguistic features of speech with 86% sensitivity, 70% specificity, and an area under the receiver operating characteristic curve (AUC) of 80%. Speech analysis of audios collected from veterans in everyday life settings using smartphones is a promising approach for suicidal ideation detection. A machine learning classifier may eventually help clinicians identify and monitor high-risk veterans.

show abstract

“…Apart from the above augmentation approaches, we also look into a specific oversampling technique [38] to compare the methodological flaw if there any exists in Polar unrolling as key part of the augmentation step. The oversampling of the data can be an alternative step to augmentation due to given imbalanced data.…”

Section: Data Augmentationmentioning

confidence: 99%

Deep Learning-Based Smart IoT Health System for Blindness Detection Using Retina Images

et al. 2021

View full text Add to dashboard Cite

Deep Learning-based Smart Healthcare is getting so much attention due to real-time applicability in everyone life's, and It has obtained more attention with the convergence of IoT. Diabetic eye disease is the primary cause of blindness between working aged peoples. The major populated Asian countries such as India and China presently account for millions of people and at the verge of an eruption of diabetic inhabitants. These growing number of diabetic patients posed a major challenge among trained doctors to provide medical screening and diagnosis. Our goal is to leverage the deep learning techniques to automate the detection of blind spot in an eye and identify how severe the stage may be. In this paper, we propose an optimized technique on top of recently released pre-trained EfficientNet models for blindness identification in retinal images along with a comparative analysis among various other neural network models. Our fine-tuned EfficientNet-B5 based model evaluation follows the benchmark dataset of retina images captured using fundus photography during varied imaging stages and outperforms CNN and ResNet50 models.

show abstract

Overly optimistic prediction results on imbalanced data: a case study of flaws and benefits when applying over-sampling

Cited by 54 publications

References 35 publications

Acoustic and language analysis of speech for suicidal ideation among US veterans

Acoustic and language analysis of speech for suicidal ideation among US veterans

Acoustic and language analysis of speech for suicide ideation among US veterans

Deep Learning-Based Smart IoT Health System for Blindness Detection Using Retina Images

Contact Info

Product

Resources

About