Eight human factors and ergonomics principles for healthcare artificial intelligence

Sujan, Mark-Alexander; Pool, Rachel; Salmon, Paul M.

doi:10.1136/bmjhci-2021-100516

Cited by 23 publications

(21 citation statements)

References 28 publications

(40 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It aims to establish traceable links between the system-level hazards, risks and the safety requirements that have to be satisfied by the machine learning components. It also complements current initiatives and studies that focus on the human and organisational aspects of clinical risk management 4 5. See online supplemental appendix A for more detail on AMLAS.…”

Section: Introductionmentioning

confidence: 83%

See 1 more Smart Citation

Assuring the safety of AI-based clinical decision support systems: a case study of the AI Clinician for sepsis treatment

Festor

Jia

Gordon

et al. 2022

BMJ Health Care Inform

View full text Add to dashboard Cite

ObjectivesEstablishing confidence in the safety of Artificial Intelligence (AI)-based clinical decision support systems is important prior to clinical deployment and regulatory approval for systems with increasing autonomy. Here, we undertook safety assurance of the AI Clinician, a previously published reinforcement learning-based treatment recommendation system for sepsis.MethodsAs part of the safety assurance, we defined four clinical hazards in sepsis resuscitation based on clinical expert opinion and the existing literature. We then identified a set of unsafe scenarios, intended to limit the action space of the AI agent with the goal of reducing the likelihood of hazardous decisions.ResultsUsing a subset of the Medical Information Mart for Intensive Care (MIMIC-III) database, we demonstrated that our previously published ‘AI clinician’ recommended fewer hazardous decisions than human clinicians in three out of our four predefined clinical scenarios, while the difference was not statistically significant in the fourth scenario. Then, we modified the reward function to satisfy our safety constraints and trained a new AI Clinician agent. The retrained model shows enhanced safety, without negatively impacting model performance.DiscussionWhile some contextual patient information absent from the data may have pushed human clinicians to take hazardous actions, the data were curated to limit the impact of this confounder.ConclusionThese advances provide a use case for the systematic safety assurance of AI-based clinical systems towards the generation of explicit safety evidence, which could be replicated for other AI applications or other clinical contexts, and inform medical device regulatory bodies.

show abstract

Section: Introductionmentioning

confidence: 83%

“…It also complements current initiatives and studies that focus on the human and organisational aspects of clinical risk management. 4 5 See online supplemental appendix A for more detail on AMLAS. AMLAS is used here for its modular and iterative approach to safety assessment of a product over its whole lifecycle.…”

Section: Introductionmentioning

confidence: 99%

Assuring the safety of AI-based clinical decision support systems: a case study of the AI Clinician for sepsis treatment

Festor

Jia

Gordon

et al. 2022

BMJ Health Care Inform

View full text Add to dashboard Cite

show abstract

“…For example, FRAM has been used to study IV infusion practices in ICU to highlight performance variability, which can inform design requirements for AI technology that supports rather than erodes the adaptive capacity within this system (Furniss et al, 2020). Systems frameworks and systems analysis methods are essential for ensuring that AI is integrated meaningfully and safely into health systems (Sujan et al, 2022).…”

Section: Discussionmentioning

confidence: 99%

Stakeholder perceptions of the safety and assurance of artificial intelligence in healthcare

Sujan¹,

White

Habli

et al. 2022

Safety Science

View full text Add to dashboard Cite

“…The current focus of development and evaluation of healthcare AI tends to be predominantly technology-centric, with an emphasis on technical issues such as data quality and the potential for bias in the data (Challen et al, 2019). This is reflected in the multitude of retrospective evaluation studies, which focus on evaluation of AI models on previously collected and suitably pre-processed data, but do not consider adequately the safety and assurance of the service within which the AI is going to be used (Sujan et al, 2019).…”

Section: Introductionmentioning

confidence: 99%

“…The development of trustworthy healthcare AI needs to be based on prospective and ergonomics studies that enable iterative and incremental assessment of what happens when AI is introduced into the wider socio-technical system (Sujan et al, 2022a;Vasey et al, 2022).…”

Section: Introductionmentioning

confidence: 99%

Information governance as a socio-technical process in the development of trustworthy healthcare AI

Rees

Holding

Sujan³

2023

Front. Comput. Sci.

Self Cite

View full text Add to dashboard Cite

In this paper we describe our experiences of managing information governance (IG) processes for the assurance of healthcare AI, using the example of an out-of-hospital-cardiac-arrest recognition software within the context of the Welsh Ambulance Service. We frame IG as a socio-technical process. IG processes for the development of trustworthy healthcare AI rely on information governance work, which entails dialogue, negotiation, and trade-offs around the legal basis for data sharing, data requirements and data control. Information governance work should start early in the design life cycle and will likely continue throughout. This includes a focus on establishing and building relationships, as well as a focus on organizational readiness and deeper understanding of both AI technologies as well as their safety assurance requirements.

show abstract

Eight human factors and ergonomics principles for healthcare artificial intelligence

Cited by 23 publications

References 28 publications

Assuring the safety of AI-based clinical decision support systems: a case study of the AI Clinician for sepsis treatment

Assuring the safety of AI-based clinical decision support systems: a case study of the AI Clinician for sepsis treatment

Stakeholder perceptions of the safety and assurance of artificial intelligence in healthcare

Information governance as a socio-technical process in the development of trustworthy healthcare AI

Contact Info

Product

Resources

About