The Stakes of Uncertainty: Developing and Integrating Machine Learning in Clinical Care

Elish, Madeleine Clare

doi:10.1111/1559-8918.2018.01213

Cited by 40 publications

(55 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For example, if a model was able to achieve perfect performance on a prediction task or when it is obvious whether the model correctly predicts the outcome (e.g., image classifications that can be verified by visual inspection), explanations might be considered unnecessary. Often, the argument for explanations centers around instilling user trust in the model; however, Elish [10] argues that trust is not predicated on model interpretability, and can instead be developed by involving stakeholders throughout the model development process. Even if explanations are not required to verify model accuracy or instill trust in a model, they may still prove valuable by providing actionable information.…”

Section: Discussionmentioning

confidence: 99%

A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare

Barda

Horvat

Hochheiser

2020

BMC Med Inform Decis Mak

View full text Add to dashboard Cite

Background There is an increasing interest in clinical prediction tools that can achieve high prediction accuracy and provide explanations of the factors leading to increased risk of adverse outcomes. However, approaches to explaining complex machine learning (ML) models are rarely informed by end-user needs and user evaluations of model interpretability are lacking in the healthcare domain. We used extended revisions of previously-published theoretical frameworks to propose a framework for the design of user-centered displays of explanations. This new framework served as the basis for qualitative inquiries and design review sessions with critical care nurses and physicians that informed the design of a user-centered explanation display for an ML-based prediction tool. Methods We used our framework to propose explanation displays for predictions from a pediatric intensive care unit (PICU) in-hospital mortality risk model. Proposed displays were based on a model-agnostic, instance-level explanation approach based on feature influence, as determined by Shapley values. Focus group sessions solicited critical care provider feedback on the proposed displays, which were then revised accordingly. Results The proposed displays were perceived as useful tools in assessing model predictions. However, specific explanation goals and information needs varied by clinical role and level of predictive modeling knowledge. Providers preferred explanation displays that required less information processing effort and could support the information needs of a variety of users. Providing supporting information to assist in interpretation was seen as critical for fostering provider understanding and acceptance of the predictions and explanations. The user-centered explanation display for the PICU in-hospital mortality risk model incorporated elements from the initial displays along with enhancements suggested by providers. Conclusions We proposed a framework for the design of user-centered displays of explanations for ML models. We used the proposed framework to motivate the design of a user-centered display of an explanation for predictions from a PICU in-hospital mortality risk model. Positive feedback from focus group participants provides preliminary support for the use of model-agnostic, instance-level explanations of feature influence as an approach to understand ML model predictions in healthcare and advances the discussion on how to effectively communicate ML model information to healthcare providers.

show abstract

Section: Discussionmentioning

confidence: 99%

A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare

Barda

Horvat

Hochheiser

2020

BMC Med Inform Decis Mak

View full text Add to dashboard Cite

show abstract

“…The ensuing decades saw the salad days of what the analytic philosopher John Haugeland names as "Good Old Fashioned AI" (GOFAI). Within the paradigm GOFAI, artificial intelligence essentially referred to "procedural, logic-based reasoning and the capacity to manipulate abstract symbolic representations" [4]. For instance, the commercial "expert systems" of the 1970s and 1980s are typically understood as exemplifying GOFAI.…”

Section: Defining Ai and Machine Learning 21 Defining Ai Systems Andmentioning

confidence: 99%

“…Critical to understanding AI's consequences for epistemology and social practice, anthropologist of technology M.C. Elish stresses that "the datasets and models used in these systems are not objective representations of reality" as systems that utilize machine learning techniques "can only be thought to 'know' something in the sense that it can correlate certain relevant variables accurately" [4].…”

Section: Defining Ai and Machine Learning 21 Defining Ai Systems Andmentioning

confidence: 99%

See 1 more Smart Citation

Toward the Clinic: Understanding Patient Perspectives on AI and Data-Sharing for AI-Driven Oncology Drug Development

Dousa¹

2020

Artificial Intelligence in Oncology Drug Discovery and Development

View full text Add to dashboard Cite

The increasing application of AI-led systems for oncology drug development and patient care holds the potential to usher pronounced impacts for patients' well-being. Beyond technical innovations and infrastructural adjustments, research suggests that realizing this potential also hinges upon patients' trust and understanding. With the promise of precision oncology predicated on a data-driven approach, public and private survey studies indicate patients view the lack of clarity surrounding data privacy, security, and ownership as a growing concern. Assuming an in-depth, semi-structured interview protocol, this qualitative study examines cancer patients' perceptions of the burgeoning development of AI-led systems for oncology as well as their perspectives on sharing health data (including genetic data) for drug development. This article seeks to provide greater insight into the legal and ethical challenges that surround the application of these tools and to explore patient-centered approaches to building the frameworks of trust and accountability crucial to transferring these advances to the clinic.

show abstract

“…This kind of thinking can reveal how and why complex social problems cannot be addressed with new metrics or algorithms alone. If the "smartness" of AI lies, as Clare Elish writes, in its power to process patterns and numbers with statistics (Elish 2018), then anthropologists need to play a role in the creation and deployment of statistical systems. Anthropologists must widen their horizons to focus not just on users and designs, but also on the machine learning algorithms, data architectures, and institutional hierarchies that make up data-driven organizations.…”

Section: Conclusion: Ethnographic Empowerment In Data-intensive Envirmentioning

confidence: 99%

Ethnographic Agency in a Data Driven World

Levin

2019

Ethnographic Praxis

View full text Add to dashboard Cite

This paper argues that ethnographers can gain increased agency in data-driven corporate environments by increasing their quantitative literacy: their ability to create, understand, and strategically use quantitative data to shape organizations. Drawing on the author's experience conducting strategic user research at a technology company, the paper explores how the ability to engage with quantitative data can increase ethnographers' independence and autonomy within organizations, and can also up-level the role and value of qualitative research. The paper also explores how a deep familiarity with quantitative data can enable ethnographers to imbue quantitative data itself with new forms of agency, and can ultimately give ethnographers the tools to change institutions from within. With a greater understanding of how quantitative data is made and used, ethnographers can ensure that data is collected in representative ways, point out the limitations of existing metrics, and argue for new ways of measuring and understanding social life.

show abstract

The Stakes of Uncertainty: Developing and Integrating Machine Learning in Clinical Care

Cited by 40 publications

References 24 publications

A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare

A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare

Toward the Clinic: Understanding Patient Perspectives on AI and Data-Sharing for AI-Driven Oncology Drug Development

Ethnographic Agency in a Data Driven World

Contact Info

Product

Resources

About