General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models

Abstract

An increasing number of model-agnostic interpretation techniques for machine learning (ML) models such as partial dependence plots (PDP), permutation feature importance (PFI) and Shapley values provide insightful model interpretations, but can lead to wrong conclusions if applied incorrectly. We highlight many general pitfalls of ML model interpretation, such as using interpretation techniques in the wrong context, interpreting models that do not generalize well, ignoring feature dependencies, interactions, uncertainty estimates and issues in high-dimensional settings, or making unjustified causal interpretations, and illustrate them with examples. We focus on pitfalls for global methods that describe the average model behavior, but many pitfalls also apply to local methods that explain individual predictions. Our paper addresses ML practitioners by raising awareness of pitfalls and identifying solutions for correct model interpretation, but also addresses ML researchers by discussing open issues for further research.

On the importance of interpretable machine learning predictions to inform clinical decision making in oncology

“…However, appropriate selection of the approaches to use can be challenging and depends on the characteristics of the datasets used for model training and validation. Use of inappropriate approaches, such as applying PDP to a dataset containing intercorrelated features, can generate misleading information that is not easy to distinguish and may result in unintentional harm (68). Unfortunately, there is no guideline or standard guiding the use of these approaches, however, increasing the awareness of these techniques in the oncology community is an important initial step to establishing the interdisciplinary collaboration involving clinical experts, data scientists, and ML engineers that will lead to more robust interpretation.…”

“…Model interpretations are not detached from model performance. Misleading information can be a result of interpreting under-or over-fitted models (63,68). Therefore, we suggest prioritizing model generalizability and applying the interpretation approaches to those high-performing models for additional insights.…”

Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks

“…See also a number of previous surveys and critiques of interpretability work that have overlap with ours [3], [58], [60], [68], [95], [118], [136], [173]- [175], [208], [215], [218], [219]. This survey, however, is distinct in its focus on inner interpretability, AI safety, and the intersections between interpretability and several other research paradigms.…”

Using SHAP Values and Machine Learning to Understand Trends in the Transient Stability Limit

“…Feature effects offer insights into the impact of a feature on the model outcome, which are particularly advantageous for transient stability enhancement measure design and thus we deem them more suitable to the application proposed in this paper. Many such post-hoc IML techniques exist (reported in [20]) and authors in [21] highlight many of the pitfalls, urging caution when using IML to avoid drawing incorrect conclusions. Local Interpretable Model-agnostic Explanations (LIME) is a local technique capable of providing feature effects for individual points that can be extrapolated to form global explanations [22].…”

“…Permutation Feature Importance (PFI) is a global technique [24], used in [25] to interpret DT models trained to predict the transient stability limit. PFI can provide feature importance, but not feature effects [21] and is limited in that feature importance is based on the decrease in model performance (i.e., is linked to the error of the model).…”