Post-Hoc Explanations Fail to Achieve their Purpose in Adversarial Contexts

Bordt, Sebastian; Finck, Michèle; Raidl, Éric; Luxburg, Ulrike von

doi:10.1145/3531146.3533153

Cited by 25 publications

(8 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…First, interpreting SHAP values warrants caution 57 , as is generally the case for methods using model explanations in medicine 58,59 . Unstable explanations are not uncommon for complex models trained on large datasets 60,61 . While the ranking of importance may fluctuate, features with higher mean absolute SHAP values generally maintain consistent attributions.…”

Section: Limitations and Recommendationmentioning

confidence: 99%

Predicting Postoperative Delirium in Older Patients: a multicenter retrospective cohort study

Wu,

Sharma,

Bauch

et al. 2024

Preprint

View full text Add to dashboard Cite

BackgroundThe number of elective surgeries for older individuals is on the rise globally. Machine learning may improve risk assessment with impact on surgical planning and postoperative care. Preoperative cognitive assessment may facilitate early identification of postoperative delirium (POD). This study aim to estimate the predictive ability of machine learning models for POD using pre- and/or perioperative features, with a specific focus on adding neuropsychological assessments prior to surgery.Materials and MethodsThis retrospective cohort study analyzed data from the multicenter PAWEL study and its PAWEL-R substudy, encompassing older patients (≥70 years) undergoing elective surgeries across five medical centers from July 2017 to April 2019. A total of 1624 patients were included, with POD diagnosis made before discharge. Data included demographics, clinical, surgical, and neuropsychological features collected pre- and perioperatively. Machine learning model performance was evaluated using the area under the receiver operating characteristic curve (AUC), with permutation testing for significance and SHapley Additive exPlanations (SHAP) for effective neuropsychological assessments identification.ResultsIn this cohort of 1624 patients, 52.3% (N=850) were male, with a mean [SD] age of 77.9 [4.9] years. Predicting POD before surgery using demographic, clinical, surgical, and neuropsychological features achieved an AUC of 0.79. Incorporating all pre- and perioperative features into the model yielded a slightly higher AUC of 0.82, with no significant difference observed (P= .19). Notably, cognitive factors alone were not strong predictors (AUC=0.61). However, specific tests within neuropsychological assessments, such as the Montreal Cognitive Assessment memory subdomain and Trail Making Test Part B, were found to be crucial for prediction according to SHAP analysis.Conclusion and RelevancePreoperative risk prediction for POD can increase risk awareness in presurgical assessment and improve postoperative management in patients with a high risk for delirium.HighlightsAnalyzed 1624 older patients (≥70 years) undergoing elective surgeries across five medical centers from July 2017 to April 2019.Established machine learning model predicts postoperative delirium before surgery.Preoperative cognition enhances predictive performance, comparable to models incorporating all pre- and perioperative features.Montreal Cognitive Assessment memory subdomain and Trail Making Test Part B drive the cognition-based prediction.Perioperative surgical features, such as the duration of the surgery, are important predictors.

show abstract

Section: Limitations and Recommendationmentioning

confidence: 99%

Predicting Postoperative Delirium in Older Patients: a multicenter retrospective cohort study

Wu,

Sharma,

Bauch

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…The duty of care of an AI provider focuses on demonstrating to a mandated supervisory body that the system was compliant only with its instructions for use and documentation. This likely invalidates the possibility for end-users of interpreting system outputs or receiving explanations as a burden of proof under litigation [19].…”

Section: Ai Liability Directivementioning

confidence: 99%

Operationalizing Explainable AI in the EU Regulatory Ecosystem

Nannini,

Alonso-Moral,

Catala

et al. 2024

IEEE Intell. Syst.

View full text Add to dashboard Cite

The European Union's regulatory ecosystem presents challenges balancing legal and sociotechnical drivers for explainable AI systems. Core tensions emerge on dimensions of oversight, user needs and litigation. This paper maps provisions on algorithmic transparency and explainability across major EU data, AI, and platform policies using qualitative analysis. We characterize involved stakeholders and organizational implementation targets. Constraints become visible between useful transparency for accountability versus confidentiality protections. Through an AI hiring system example, we explore complications operationalizing explainability. Customization is required satisfying explainability desires within confidentiality and proportionality bounds. Findings advise technologists on prudent eXplainable AI technique selection given multi-dimensional tensions. Outcomes recommend policymakers balance worthy transparency goals with cohesive legislation enabling equitable dispute resolution. KEYWORDS K.4.1.c Ethics, K.4.1.g Regulation, Explainable AI, Trustworthy AI 1 https://joinup.ec.europa.eu/collection/ rolling-plan-ict-standardisation/rolling-plan-2022 2 https://eur-lex.europa.eu/legal-content/en/TXT/?uri= CELEX3A52021DC0118 IEEE Intelligent Systems

show abstract

“…The advantage of post hoc methods is that they can be used with any model architecture and, therefore, do not require a trade-off of predictive model performance. However, such approaches explain model decisions on specified data points and do not provide a holistic view of the entire decision function (Bordt et al, 2022). An alternative approach is to use intrinsically explainable machine learning models (Rudin, 2019).…”

Section: Introductionmentioning

confidence: 99%

Identifying compound weather drivers of forest biomass loss with generative deep learning

Anand,

Bohn,

Camps-Valls

et al. 2024

Environ. Data Science

View full text Add to dashboard Cite

Globally, forests are net carbon sinks that partly mitigates anthropogenic climate change. However, there is evidence of increasing weather-induced tree mortality, which needs to be better understood to improve forest management under future climate conditions. Disentangling drivers of tree mortality is challenging because of their interacting behavior over multiple temporal scales. In this study, we take a data-driven approach to the problem. We generate hourly temperate weather data using a stochastic weather generator to simulate 160,000 years of beech, pine, and spruce forest dynamics with a forest gap model. These data are used to train a generative deep learning model (a modified variational autoencoder) to learn representations of three-year-long monthly weather conditions (precipitation, temperature, and solar radiation) in an unsupervised way. We then associate these weather representations with years of high biomass loss in the forests and derive weather prototypes associated with such years. The identified prototype weather conditions are associated with 5–22% higher median biomass loss compared to the median of all samples, depending on the forest type and the prototype. When prototype weather conditions co-occur, these numbers increase to 10–25%. Our research illustrates how generative deep learning can discover compounding weather patterns associated with extreme impacts.

show abstract

Post-Hoc Explanations Fail to Achieve their Purpose in Adversarial Contexts

Cited by 25 publications

References 18 publications

Predicting Postoperative Delirium in Older Patients: a multicenter retrospective cohort study

Predicting Postoperative Delirium in Older Patients: a multicenter retrospective cohort study

Operationalizing Explainable AI in the EU Regulatory Ecosystem

Identifying compound weather drivers of forest biomass loss with generative deep learning

Contact Info

Product

Resources

About