Efficient permutation testing of variable importance measures by the example of random forests

Hapfelmeier, Alexander; Hornung, Roman; Haller, Bernhard

doi:10.1016/j.csda.2022.107689

Cited by 10 publications

(8 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A decrease in model performance post-permutation suggests the predictor variable’s pivotal role in model accuracy, while minimal impact indicates less influence [ 80 ]. However, this technique may pose computational challenges, particularly with large datasets [ 81 ].…”

Section: Methodsmentioning

confidence: 99%

Enhancing credit scoring accuracy with a comprehensive evaluation of alternative data

Hlongwane,

Ramaboa,

Mongwe

2024

PLoS ONE

View full text Add to dashboard Cite

This study explores the potential of utilizing alternative data sources to enhance the accuracy of credit scoring models, compared to relying solely on traditional data sources, such as credit bureau data. A comprehensive dataset from the Home Credit Group’s home loan portfolio is analysed. The research examines the impact of incorporating alternative predictors that are typically overlooked, such as an applicant’s social network default status, regional economic ratings, and local population characteristics. The modelling approach applies the model-X knockoffs framework for systematic variable selection. By including these alternative data sources, the credit scoring models demonstrate improved predictive performance, achieving an area under the curve metric of 0.79360 on the Kaggle Home Credit default risk competition dataset, outperforming models that relied solely on traditional data sources, such as credit bureau data. The findings highlight the significance of leveraging diverse, non-traditional data sources to augment credit risk assessment capabilities and overall model accuracy.

show abstract

Section: Methodsmentioning

confidence: 99%

Enhancing credit scoring accuracy with a comprehensive evaluation of alternative data

Hlongwane,

Ramaboa,

Mongwe

2024

PLoS ONE

View full text Add to dashboard Cite

show abstract

“…Likelihood-based permutation variable importance measures (VIMPs) of this final model were used to identify informative predictor variables. 30 , 31 Predictor variables with a VIMP lower than the VIMP of an additionally included random variable were excluded from VIMP display. 32 To further evaluate the counterfactual analysis, the MS-TDS was calculated assuming both treatment alternatives for each patient and compared between the actual medication groups.…”

Section: Methodsmentioning

confidence: 99%

Retrospective cohort study to devise a treatment decision score predicting adverse 24-month radiological activity in early multiple sclerosis

Hapfelmeier

Ön

Mühlau

et al. 2023

Ther Adv Neurol Disord

Self Cite

View full text Add to dashboard Cite

Background: Multiple sclerosis (MS) is a chronic neuroinflammatory disease affecting about 2.8 million people worldwide. Disease course after the most common diagnoses of relapsing-remitting multiple sclerosis (RRMS) and clinically isolated syndrome (CIS) is highly variable and cannot be reliably predicted. This impairs early personalized treatment decisions. Objectives: The main objective of this study was to algorithmically support clinical decision-making regarding the options of early platform medication or no immediate treatment of patients with early RRMS and CIS. Design: Retrospective monocentric cohort study within the Data Integration for Future Medicine (DIFUTURE) Consortium. Methods: Multiple data sources of routine clinical, imaging and laboratory data derived from a large and deeply characterized cohort of patients with MS were integrated to conduct a retrospective study to create and internally validate a treatment decision score [Multiple Sclerosis Treatment Decision Score (MS-TDS)] through model-based random forests (RFs). The MS-TDS predicts the probability of no new or enlarging lesions in cerebral magnetic resonance images (cMRIs) between 6 and 24 months after the first cMRI. Results: Data from 65 predictors collected for 475 patients between 2008 and 2017 were included. No medication and platform medication were administered to 277 (58.3%) and 198 (41.7%) patients. The MS-TDS predicted individual outcomes with a cross-validated area under the receiver operating characteristics curve (AUROC) of 0.624. The respective RF prediction model provides patient-specific MS-TDS and probabilities of treatment success. The latter may increase by 5–20% for half of the patients if the treatment considered superior by the MS-TDS is used. Conclusion: Routine clinical data from multiple sources can be successfully integrated to build prediction models to support treatment decision-making. In this study, the resulting MS-TDS estimates individualized treatment success probabilities that can identify patients who benefit from early platform medication. External validation of the MS-TDS is required, and a prospective study is currently being conducted. In addition, the clinical relevance of the MS-TDS needs to be established.

show abstract

“…Before building our prediction models we wanted to see if we could build predictive features from group level differences in frequency band power across the different states. We used Welch's method to calculate power in delta (0.5-4 Hz), theta (4-8 Hz), alpha (8-13 Hz), beta (13)(14)(15)(16)(17)(18)(19)(20)(21)(22)(23)(24)(25)(26)(27)(28)(29)(30), low gamma (30-50 Hz) and high gamma (50-100 Hz). The power was calculated per 3 s samples and normalized to overall power in the sample and then averaged within and across channels, resulting in one power value per round, band, side and state for each subject.…”

Section: Data Acquisition and Modelingmentioning

confidence: 99%

“…A permutation test provides only approximate p-values unless all possible permutations are computed for an exact test [25]. A sequential approximation to the general permutation test (SAPT) was used to stop permuting when adequate number of permutations had been done to either reject or accept the null hypothesis of no difference [26,27]. This method controls the type I error rate while achieving power close to the exact permutation test but with a significant reduction in number of permutations.…”

Section: Data Acquisition and Modelingmentioning

confidence: 99%

Patient specific intracranial neural signatures of obsessions and compulsions in the ventral striatum

et al. 2023

View full text Add to dashboard Cite

Objective: Deep brain stimulation is a treatment option for patients with refractory obsessive-compulsive disorder. A new generation of stimulators hold promise for closed loop stimulation, with adaptive stimulation in response to biologic signals. Here we aimed to discover a suitable biomarker in the ventral striatum in patients with obsessive compulsive disorder using local field potentials. Approach: We induced obsessions and compulsions in 11 patients undergoing deep brain stimulation treatment using a symptom provocation task. Then we trained machine learning models to predict symptoms using the recorded intracranial signal from the deep brain stimulation electrodes. Main results: Average areas under the receiver operating characteristics curve were 62.1% for obsessions and 78.2% for compulsions for patient specific models. For obsessions it reached over 85% in one patient, whereas performance was near chance level when the model was trained across patients. Optimal performances for obsessions and compulsions was obtained at different recording sites. Significance: The results from this study suggest that closed loop stimulation may be a viable option for obsessive-compulsive disorder, but that intracranial biomarkers are patient and not disorder specific. Clinical Trial: Netherlands trial registry NL7486

show abstract

Efficient permutation testing of variable importance measures by the example of random forests

Cited by 10 publications

References 41 publications

Enhancing credit scoring accuracy with a comprehensive evaluation of alternative data

Enhancing credit scoring accuracy with a comprehensive evaluation of alternative data

Retrospective cohort study to devise a treatment decision score predicting adverse 24-month radiological activity in early multiple sclerosis

Patient specific intracranial neural signatures of obsessions and compulsions in the ventral striatum

Contact Info

Product

Resources

About