Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance

Bansal, Gagan; Wu, Tongshuang; Zhou, Jasmine; Fok, Raymond; Nushi, Besmira; Kamar, Ece; Ribeiro, Marco Túlio; Weld, Daniel S.

doi:10.1145/3411764.3445717

Cited by 298 publications

(303 citation statements)

References 62 publications

Supporting

Mentioning

247

Contrasting

Order By: Relevance

“…It has long been reported that algorithms interacting with humans can negatively impact the human. 9 In our case, the concern might be that users can develop an over-reliance on Polyjuice (Bansal et al, 2021) and hastily accept its generations. Not only can this decrease users' creativity (Green et al, 2014), but it may bias their analysis process: as discussed in §7, Polyjuice generation is not exhaustive, and may favor some perturbation patterns over others in unpredictable ways.…”

Section: Ethical Considerationsmentioning

confidence: 99%

Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models

Wu¹,

Ribeiro²,

Heer³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

Self Cite

View full text Add to dashboard Cite

While counterfactual examples are useful for analysis and training of NLP models, current generation methods either rely on manual labor to create very few counterfactuals, or only instantiate limited types of perturbations such as paraphrases or word substitutions. We present Polyjuice, a general-purpose counterfactual generator that allows for control over perturbation types and locations, trained by finetuning GPT-2 on multiple datasets of paired sentences. We show that Polyjuice produces diverse sets of realistic counterfactuals, which in turn are useful in various distinct applications: improving training and evaluation on three different tasks (with around 70% less annotation effort than manual generation), augmenting state-of-the-art explanation techniques, and supporting systematic counterfactual error analysis by revealing behaviors easily missed by human experts.

show abstract

Section: Ethical Considerationsmentioning

confidence: 99%

Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models

Wu¹,

Ribeiro²,

Heer³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

Self Cite

View full text Add to dashboard Cite

show abstract

“…These models are black boxes and do not offer interpretable explanations for their predictions; there is no notion of step-by-step deduction. This limitation prevents users from understanding and accommodating models' affordances (Hase and Bansal, 2020;Bansal et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

Natural Language Deduction through Search over Statement Compositions

Bostrom¹,

Sprague²,

Chaudhuri³

et al. 2022

Preprint

View full text Add to dashboard Cite

In settings from fact-checking to question answering, we frequently want to know whether a collection of evidence entails a hypothesis. Existing methods primarily focus on end-toend discriminative versions of this task, but less work has treated the generative version in which a model searches over the space of entailed statements to derive the hypothesis. We propose a system for natural language deduction that decomposes the task into separate steps coordinated by best-first search, producing a tree of intermediate conclusions that faithfully reflects the system's reasoning process. Our experiments demonstrate that the proposed system can better distinguish verifiable hypotheses from unverifiable ones and produce natural language explanations that are more internally consistent than those produced by an end-to-end T5 model.

show abstract

“…By combining two types of intelligence, these emerging sociotechnical systems (i.e., human+AI teams) were expected to perform better than either people or AIs alone [35,36]. Recent studies, however, show that although human+AI teams typically outperform people working alone, their performance is usually inferior to the AI's [2,7,9,26,30,41,59]. There is evidence that instead of combining their own insights with suggestions generated by the computational models, people frequently overrely on the AI, following its suggestions even when those suggestions are wrong and the person would have made a better choice on their own [9,30,41].…”

Section: Introductionmentioning

confidence: 99%

“…Explainable AI (XAI), an approach where AI's recommendations are accompanied by explanations or rationales, was intended to address the problem of overreliance: By giving people an insight into how the machine arrived at its recommendations, the explanations were supposed to help them identify the situations in which AI's reasoning was incorrect and the suggestion should be rejected. However, evidence suggests that explainable systems, also, have not had substantial success in reducing human overreliance on the AIs: when the AI suggests incorrect or suboptimal solutions, people still on average make poorer final decisions than they would have without AI's assistance [2,7,30,40,67].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making

Buçinca,

Malaya,

Gajos

2021

Preprint

View full text Add to dashboard Cite

People supported by AI-powered decision support tools frequently overrely on the AI: they accept an AI's suggestion even when that suggestion is wrong. Adding explanations to the AI decisions does not appear to reduce the overreliance and some studies suggest that it might even increase it. Informed by the dual-process theory of cognition, we posit that people rarely engage analytically with each individual AI recommendation and explanation, and instead develop general heuristics about whether and when to follow the AI suggestions. Building on prior research on medical decision-making, we designed three cognitive forcing interventions to compel people to engage more thoughtfully with the AI-generated explanations. We conducted an experiment (N=199), in which we compared our three cognitive forcing designs to two simple explainable AI approaches and to a no-AI baseline. The results demonstrate that cognitive forcing significantly reduced overreliance compared to the simple explainable AI approaches. However, there was a trade-off: people assigned the least favorable subjective ratings to the designs that reduced the overreliance the most. To audit our work for intervention-generated inequalities, we investigated whether our interventions benefited equally people with different levels of Need for Cognition (i.e., motivation to engage in effortful mental activities). Our results show that, on average, cognitive forcing interventions benefited participants higher in Need for Cognition more. Our research suggests that human cognitive motivation moderates the effectiveness of explainable AI solutions.CCS Concepts: • Human-centered computing → Interaction design.

show abstract

Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance

Cited by 298 publications

References 62 publications

Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models

Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models

Natural Language Deduction through Search over Statement Compositions

To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making

Contact Info

Product

Resources

About