A review of possible effects of cognitive biases on the interpretation of rule-based machine learning models

Kliegr, Tomáš; Bahník, Štěpán; Fürnkranz, Johannes

doi:10.48550/arxiv.1804.02969

Cited by 6 publications

(10 citation statements)

References 124 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We summarized some of these hypotheses, such as the conjunctive fallacy, and started to investigate to what extent these can serve as explanations for human preferences between different learned hypotheses. There are numerous other cognitive effects that can demonstrate how people assess rule plausibility, some of which are briefly listed in Appendix 11 and discussed more extensively in Kliegr et al (2018). Clearly, more work along these lines is needed.…”

Section: Discussionmentioning

confidence: 99%

“…However, we neither claim completeness, nor can we provide more than a very short summary of each phenomenon. A more extensive treatment can be found in (Kliegr et al, 2018). An extensive treatment of the subject can be found in (Kliegr et al, 2018).…”

Section: Appendix -A Brief Overview Of Relevant Cognitive Heuristics ...mentioning

confidence: 99%

“…A more extensive treatment can be found in (Kliegr et al, 2018). An extensive treatment of the subject can be found in (Kliegr et al, 2018).…”

Section: Appendix -A Brief Overview Of Relevant Cognitive Heuristics ...mentioning

confidence: 99%

See 2 more Smart Citations

On cognitive preferences and the plausibility of rule-based models

2019

Self Cite

View full text Add to dashboard Cite

It is conventional wisdom in machine learning and data mining that logical models such as rule sets are more interpretable than other models, and that among such rule-based models, simpler models are more interpretable than more complex ones. In this position paper, we question this latter assumption by focusing on one particular aspect of interpretability, namely the plausibility of models. Roughly speaking, we equate the plausibility of a model with the likeliness that a user accepts it as an explanation for a prediction. In particular, we argue that-all other things being equal-longer explanations may be more convincing than shorter ones, and that the predominant bias for shorter models, which is typically necessary for learning powerful discriminative models, may not be suitable when it comes to user acceptance of the learned models. To that end, we first recapitulate evidence for and against this postulate, and then report the results of an evaluation in a crowdsourcing study based on about 3,000 judgments. The results do not reveal a strong preference for simple rules, whereas we can observe a weak preference for longer rules in some domains. We then relate these results to well-known cognitive biases such as the conjunction fallacy, the representative heuristic, or the recogition heuristic, and investigate their relation to rule length and plausibility.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Appendix -A Brief Overview Of Relevant Cognitive Heuristics ...mentioning

confidence: 99%

See 1 more Smart Citation

On cognitive preferences and the plausibility of rule-based models

2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…Another important aspect of this empirical line of research is the study of cognitive biases in the understanding of interpretable models. Kliegr et al (2018) study the possible effects of biases on symbolic machine learning models. 16 As noted in the Introduction, none of these methods is intrinsically interpretable.…”

Section: Alternative Paths To Understandingmentioning

confidence: 99%

The Pragmatic Turn in Explainable Artificial Intelligence (XAI)

Páez

2019

Minds & Machines

149

View full text Add to dashboard Cite

In this paper I argue that the search for explainable models and interpretable decisions in AI must be reformulated in terms of the broader project of offering a pragmatic and naturalistic account of understanding in AI. Intuitively, the purpose of providing an explanation of a model or a decision is to make it understandable to its stakeholders. But without a previous grasp of what it means to say that an agent understands a model or a decision, the explanatory strategies will lack a well-defined goal. Aside from providing a clearer objective for XAI, focusing on understanding also allows us to relax the factivity condition on explanation, which is impossible to fulfill in many machine learning models, and to focus instead on the pragmatic conditions that determine the best fit between a model and the methods and devices deployed to understand it. After an examination of the different types of understanding discussed in the philosophical and psychological literature, I conclude that interpretative or approximation models not only provide the best way to achieve the objectual understanding of a machine learning model, but are also a necessary condition to achieve post-hoc interpretability. This conclusion is partly based on the shortcomings of the purely functionalist approach to post-hoc interpretability that seems to be predominant in most recent literature.

show abstract

“…Recently, (Fürnkranz et al, 2020) evaluated a selection of cognitive biases in the very specific context of whether minimizing the complexity or length of a rule will also lead to increased interpretability of machine learning models. Kleiger et al (Kliegr et al, 2018) review twenty different cognitive biases that can distort interpretation of inductively learned rules in ML models. This work analyses effect of cognitive biases on human understanding of symbolic ML models and associated debiasing techniques.…”

Section: Related Workmentioning

confidence: 99%

Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making

Rastogi¹,

Zhang²,

Wei³

et al. 2020

Preprint

View full text Add to dashboard Cite

Several strands of research have aimed to bridge the gap between artificial intelligence (AI) and human decisionmakers in AI-assisted decision-making, where humans are the consumers of AI model predictions and the ultimate decision-makers in high-stakes applications. However, people's perception and understanding is often distorted by their cognitive biases, like confirmation bias, anchoring bias, availability bias, to name a few. In this work, we use knowledge from the field of cognitive science to account for cognitive biases in the human-AI collaborative decision-making system and mitigate their negative effects. To this end, we mathematically model cognitive biases and provide a general framework through which researchers and practitioners can understand the interplay between cognitive biases and human-AI accuracy. We then focus on anchoring bias, a bias commonly witnessed in human-AI partnerships. We devise a cognitive science-driven, time-based approach to de-anchoring. A user experiment shows the effectiveness of this approach in human-AI collaborative decision-making. Using the results from this first experiment, we design a time allocation strategy for a resource constrained setting so as to achieve optimal human-AI collaboration under some assumptions. A second user study shows that our time allocation strategy can effectively debias the human when the AI model has low confidence and is incorrect.

show abstract

A review of possible effects of cognitive biases on the interpretation of rule-based machine learning models

Cited by 6 publications

References 124 publications

On cognitive preferences and the plausibility of rule-based models

On cognitive preferences and the plausibility of rule-based models

The Pragmatic Turn in Explainable Artificial Intelligence (XAI)

Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making

Contact Info

Product

Resources

About