Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs

Suresh, Harini; Gomez, Steven R.; Nam, Kevin; Satyanarayan, Arvind

doi:10.1145/3411764.3445088

Cited by 77 publications

(60 citation statements)

References 121 publications

(267 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nevertheless, given the potentially severe consequences of an incorrect prediction, we consider it essential to present additional context for predictions from all AA methods. While the exact context needs to be tailored to the intended application and user (Suresh et al, 2021), authors should include calculated metrics or predictors and any easily obtained fixed or learned thresholds.…”

Section: Which Sample Of Assessments Is Most Effective For Training and Evaluating Aa Methods?mentioning

confidence: 99%

Evidence-based guidelines for automated conservation assessments of plant species

Walker¹,

Leão²,

Bachman³

et al. 2021

Preprint

View full text Add to dashboard Cite

Assessing species’ risk of extinction is a vital first step in setting conservation priorities. However, assessment endeavours like the IUCN Red List of Threatened Species still have significant gaps in their coverage of some taxonomic groups. Automated assessment (AA) methods are gaining popularity to rapidly fill these gaps, taking advantage of improvements in computing and digitally available information. However, implicit choices made when developing and reporting automated assessment methods could prevent their successful adoption or, even worse, their predictions could lead to poor allocation of conservation resources.We systematically explored how the choice of data cleaning, taxonomic group, training sample, and automation method affected predicted threat status. We used occurrence records from GBIF to generate assessments for three distinct taxonomic groups using four different automated assessment methods. We measured each method’s performance and coverage after applying increasingly stringent cleaning to the input occurrence data. We used these results to build evidence-based guidelines for developing and reporting automated assessments.Automatically cleaned data from GBIF resulted in comparable performance to occurrence records cleaned manually by an expert. However, all types of data cleaning removed species and limited the coverage of automated assessments. This limitation was more severe for some groups of species than others. Overall, machine learning-based methods performed well on all taxonomic groups, even with minimal data cleaning.We recommend using a machine learning-based method on minimally cleaned data to get the best compromise between performance and species coverage. However, our results demonstrate that the optimal data cleaning, training sample, and automation method depends on the focal group of species. Therefore, we recommend evaluating new AA methods across multiple groups and providing additional context with extinction risk predictions for users to make informed decisions.

show abstract

Section: Which Sample Of Assessments Is Most Effective For Training and Evaluating Aa Methods?mentioning

confidence: 99%

Evidence-based guidelines for automated conservation assessments of plant species

Walker¹,

Leão²,

Bachman³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The formative study has two parts: we first thoroughly surveyed relevant literature [5,29,31,35,37,38,42,55] and industrial standard documentations [1,2,18,36] to understand how a presentation deck looks like, and what contents are usually included. Based on these insights, we drafted an presentation outline to represent a common presentation structure.…”

Section: Formative Studymentioning

confidence: 99%

“…Some prior works reported that DS workers might spend hours or even days to prepare the presentation slide after they finish their core technical modeling works [38]. The work is tedious and time-consuming due to multiple reasons: a) data scientists have to meticulously locate and distill essential information from the complex, messy, and sometimes fragmented experimental codes [29,39]; b) they need to organize these information to construct a story narrative; and, c) they also need to consider the specific domain context and audience backgrounds, and often add additional information (e.g., visualizations, explanations, and examples) to customize their presentation so that it can better engage the target audience and gain their trust [8,29,42].…”

Section: Introductionmentioning

confidence: 99%

Telling Stories from Computational Notebooks: AI-Assisted Presentation Slides Creation for Presenting Data Science Work

Zheng,

Wang,

Wang

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…It is of particular importance to investigate end user characteristics specifically in the clinical context of the chosen task. This is because, depending on the task, stakeholders have varied interest, prior knowledge, responsibilities, and requirements [92].…”

Section: Intrprt Guidelinementioning

confidence: 99%

“…More specialized guidelines are required for high-risk domains. Similarly, previous attempts to guide the design of effective transparency mechanisms acknowledge that real stakeholders involved should be considered and understood [92,53,101]. Starting from the identification of diverse design goals according to users' needs and their level of expertise on AI technology, and a categorization of evaluation measures for Explainable Artificial Intelligence (XAI) systems, [67] addressed the multidisciplinary efforts needed to build such systems.…”

Section: Increasing Demand For Guidelines To Build ML Systemsmentioning

confidence: 99%

INTRPRT: A Systematic Review of and Guidelines for Designing and Validating Transparent AI in Medical Image Analysis

Chen¹,

Gómez²,

Huang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Transparency in Machine Learning (ML), including interpretable or explainable ML, attempts to reveal the working mechanisms of complex models, including deep neural networks. Transparent ML promises to advance human factors engineering goals of human-centered AI, such as increasing trust or reducing automation bias, in the target users. However, a core requirement in achieving these aims is that the method is indeed transparent to the users. From a human-centered design perspective, transparency is not a property of the ML model but an affordance, i. e., a relationship between algorithm and user; as a result, iterative prototyping and evaluation with users is critical to attaining adequate solutions that afford transparency. However, following human-centered design principles in highly specialized and high stakes domains, such as healthcare and medical image analysis, is challenging due to the limited availability of and access to end users, such as domain experts, providers, or patients. This dilemma is further exacerbated by the high knowledge imbalance between ML designers and those end users, which implies an even greater need for iterated development.To investigate the state of transparent ML in medical image analysis, we conducted and now present a systematic review of the literature. Our review reveals multiple severe shortcomings in the design and validation of transparent ML for medical image analysis applications. We find that most studies to date approach transparency as a property of the model itself, similar to task performance, without considering end users during neither development nor evaluation. Despite the considerable difference between the roles and knowledge of ML developers and clinical stakeholders, no study reported formative user research to inform the design and development of transparent ML models; moreover, only a few studies validated transparency claims through empirical user evaluations. The oversight of considering ML transparency as a relationship with end users, the lack of user research, and the sporadic validation of transparency claims put contemporary research on transparent ML for medical image analysis at risk of being incomprehensible to users, and thus, clinically irrelevant. To alleviate these shortcomings in forthcoming research while acknowledging the challenges of human-centered design in healthcare, we introduce the INTRPRT guideline, a systematic design directive for transparent ML systems in medical image analysis. To bridge the disconnect between ML system designers and end users and avoid misspending costly development efforts on models that are finally validated as non-transparent to end users, the INTRPRT guideline suggests formative user research as the first step of transparent model design to understand user needs and domain requirements. Following this process produces evidence to support * Joint first authors.Preprint. Under review.

show abstract

Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs

Cited by 77 publications

References 121 publications

Evidence-based guidelines for automated conservation assessments of plant species

Evidence-based guidelines for automated conservation assessments of plant species

Telling Stories from Computational Notebooks: AI-Assisted Presentation Slides Creation for Presenting Data Science Work

INTRPRT: A Systematic Review of and Guidelines for Designing and Validating Transparent AI in Medical Image Analysis

Contact Info

Product

Resources

About