Explainable machine learning in deployment

Bhatt, Umang; Xiang, Alice; Sharma, Shubham; Weller, Adrian; Taly, Ankur; Jia, Yunhan Jack; Ghosh, Joydeep; Puri, Ruchir; Moura, J.M.F.; Eckersley, Peter

doi:10.1145/3351095.3375624

Cited by 424 publications

(321 citation statements)

References 34 publications

(25 reference statements)

Supporting

Mentioning

283

Contrasting

Unclassified

Order By: Relevance

“…For instance, there are works on developing algorithms and novel DL architectures in XAI to add explainability to the models [42][43][44][45][46]. In comparison, there is also work that considers user experience and user requirements for XAI [7][8][9][10]47], and evaluates algorithms and models with user studies [48]. However, analyzing and categorizing XAI algorithms is not the focus of this paper.…”

Section: Classifying Hcml Researchmentioning

confidence: 99%

“…Apart from studies to understand aspects of specific applications, there have been attempts to understand user concerns, such as explainability and fairness, that later translate into ML models' features. One comprehensive study [48] has been conducted to explore current practices of explainable-AI in the industry to understand the desirable techniques for real-world usage. Another study [58] attempts to identify gaps between current XAI algorithmic work and practices towards user-centered XAI.…”

Section: User Studiesmentioning

confidence: 99%

“…For instance, the focus may be on the end-user, the developer, or the investor. One could focus on a certain user-aspect when developing a product or service [83,160]; another could be determining design principles for a particular ML system optimizing usability and adoptability [48,65,196]. The multidimensionality of what is considered Human within HCML contributes to the complexities within the field.…”

Section: The 'Human' In Hcmlmentioning

confidence: 99%

“…Examples for these include people who need assistance [2,80,[86][87][88][89]96,117,147], medical professionals [57,67,110,192,193], international travelers [50], Amazon Mechanical Turk [60,99], drivers [161,162], musicians [102], teachers [124], students [128], children [72,125], UX designers [65,115,206,209], UI designers [103,111,173], data analysts [97], video creators [84], and game designers [70,165,174,211]. Apart from focusing on a specific user group, some have tried to understand multiple user-perspectives from ML engineers to the end-user [48]. Some of the prior works that target the developer as the human focus on novice ML engineers to help them develop ML systems faster [62,197].…”

Section: The 'Human' In Hcmlmentioning

confidence: 99%

See 3 more Smart Citations

A Review of Recent Deep Learning Approaches in Human-Centered Machine Learning

Kaluarachchi

Reis

Nanayakkara

2021

Sensors

View full text Add to dashboard Cite

After Deep Learning (DL) regained popularity recently, the Artificial Intelligence (AI) or Machine Learning (ML) field is undergoing rapid growth concerning research and real-world application development. Deep Learning has generated complexities in algorithms, and researchers and users have raised concerns regarding the usability and adoptability of Deep Learning systems. These concerns, coupled with the increasing human-AI interactions, have created the emerging field that is Human-Centered Machine Learning (HCML). We present this review paper as an overview and analysis of existing work in HCML related to DL. Firstly, we collaborated with field domain experts to develop a working definition for HCML. Secondly, through a systematic literature review, we analyze and classify 162 publications that fall within HCML. Our classification is based on aspects including contribution type, application area, and focused human categories. Finally, we analyze the topology of the HCML landscape by identifying research gaps, highlighting conflicting interpretations, addressing current challenges, and presenting future HCML research opportunities.

show abstract

Section: Classifying Hcml Researchmentioning

confidence: 99%

Section: User Studiesmentioning

confidence: 99%

Section: The 'Human' In Hcmlmentioning

confidence: 99%

Section: The 'Human' In Hcmlmentioning

confidence: 99%

See 2 more Smart Citations

A Review of Recent Deep Learning Approaches in Human-Centered Machine Learning

Kaluarachchi

Reis

Nanayakkara

2021

Sensors

View full text Add to dashboard Cite

show abstract

“…While such proposals and mechanisms have their place, many proposals-whether intended to inform the general public, system developers, regulators, or others-address models themselves, rather than broader decision-making processes of which models are but one element. Though explanations can assist some specific concerns, like model engineering [7], explanations or other approaches focused on how the model itself works or has reached a particular outcome may miss much of what is important [75].…”

Section: Limitations Of Model-focused Mechanismsmentioning

confidence: 99%

Reviewable Automated Decision-Making

Cobbe

Lee

Singh

2021

Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency

View full text Add to dashboard Cite

This paper introduces reviewability as a framework for improving the accountability of automated and algorithmic decisionmaking (ADM) involving machine learning. We draw on an understanding of ADM as a socio-technical process involving both human and technical elements, beginning before a decision is made and extending beyond the decision itself. While explanations and other model-centric mechanisms may assist some accountability concerns, they often provide insufficient information of these broader ADM processes for regulatory oversight and assessments of legal compliance. Reviewability involves breaking down the ADM process into technical and organisational elements to provide a systematic framework for determining the contextually appropriate record-keeping mechanisms to facilitate meaningful review-both of individual decisions and of the process as a whole. We argue that a reviewability framework, drawing on administrative law's approach to reviewing human decision-making, offers a practical way forward towards more a more holistic and legally-relevant form of accountability for ADM.

show abstract

Measuring the Impact of AI in the Diagnosis of Hospitalized Patients

Jabbour,

Fouhey,

Shepard

et al. 2023

JAMA

View full text Add to dashboard Cite

ImportanceArtificial intelligence (AI) could support clinicians when diagnosing hospitalized patients; however, systematic bias in AI models could worsen clinician diagnostic accuracy. Recent regulatory guidance has called for AI models to include explanations to mitigate errors made by models, but the effectiveness of this strategy has not been established.ObjectivesTo evaluate the impact of systematically biased AI on clinician diagnostic accuracy and to determine if image-based AI model explanations can mitigate model errors.Design, Setting, and ParticipantsRandomized clinical vignette survey study administered between April 2022 and January 2023 across 13 US states involving hospitalist physicians, nurse practitioners, and physician assistants.InterventionsClinicians were shown 9 clinical vignettes of patients hospitalized with acute respiratory failure, including their presenting symptoms, physical examination, laboratory results, and chest radiographs. Clinicians were then asked to determine the likelihood of pneumonia, heart failure, or chronic obstructive pulmonary disease as the underlying cause(s) of each patient’s acute respiratory failure. To establish baseline diagnostic accuracy, clinicians were shown 2 vignettes without AI model input. Clinicians were then randomized to see 6 vignettes with AI model input with or without AI model explanations. Among these 6 vignettes, 3 vignettes included standard-model predictions, and 3 vignettes included systematically biased model predictions.Main Outcomes and MeasuresClinician diagnostic accuracy for pneumonia, heart failure, and chronic obstructive pulmonary disease.ResultsMedian participant age was 34 years (IQR, 31-39) and 241 (57.7%) were female. Four hundred fifty-seven clinicians were randomized and completed at least 1 vignette, with 231 randomized to AI model predictions without explanations, and 226 randomized to AI model predictions with explanations. Clinicians’ baseline diagnostic accuracy was 73.0% (95% CI, 68.3% to 77.8%) for the 3 diagnoses. When shown a standard AI model without explanations, clinician accuracy increased over baseline by 2.9 percentage points (95% CI, 0.5 to 5.2) and by 4.4 percentage points (95% CI, 2.0 to 6.9) when clinicians were also shown AI model explanations. Systematically biased AI model predictions decreased clinician accuracy by 11.3 percentage points (95% CI, 7.2 to 15.5) compared with baseline and providing biased AI model predictions with explanations decreased clinician accuracy by 9.1 percentage points (95% CI, 4.9 to 13.2) compared with baseline, representing a nonsignificant improvement of 2.3 percentage points (95% CI, −2.7 to 7.2) compared with the systematically biased AI model.Conclusions and RelevanceAlthough standard AI models improve diagnostic accuracy, systematically biased AI models reduced diagnostic accuracy, and commonly used image-based AI model explanations did not mitigate this harmful effect.Trial RegistrationClinicalTrials.gov Identifier: NCT06098950

show abstract

Explainable machine learning in deployment

Cited by 424 publications

References 34 publications

A Review of Recent Deep Learning Approaches in Human-Centered Machine Learning

A Review of Recent Deep Learning Approaches in Human-Centered Machine Learning

Reviewable Automated Decision-Making

Measuring the Impact of AI in the Diagnosis of Hospitalized Patients

Contact Info

Product

Resources

About