he prospect of improved clinical outcomes and more efficient health systems has fueled a rapid rise in the development and evaluation of AI systems over the last decade. Because most AI systems within healthcare are complex interventions designed as clinical decision support systems, rather than autonomous agents, the interactions among the AI systems, their users and the implementation environments are defining components of the AI interventions' overall potential effectiveness. Therefore, bringing AI systems from mathematical performance to clinical utility needs an adapted, stepwise implementation and evaluation pathway, addressing the complexity of this collaboration between two independent forms of intelligence, beyond measures of effectiveness alone 1 . Despite indications that some AI-based algorithms now match the accuracy of human experts within preclinical in silico studies 2 , there

Systems contradictions present challenges that need to be effectively managed, e.g. due to conflicting rules and advice, goal conflicts, and mismatches between demand and capacity. We apply FRAM (Functional Resonance Analysis Method) to intravenous infusion practices in an intensive care unit (ICU) to explore how tensions and contradictions are managed by people. A multi-disciplinary team including individuals from nursing, medical, pharmacy, safety, IT and human factors backgrounds contributed to this analysis. A FRAM model investigation resulting in seven functional areas are described. A tabular analysis highlights significant areas of performance variability, e.g. administering medication before a prescription, prioritising drugs, different degrees of double checking and using sites showing early signs of infection for intravenous access. Our FRAM analysis has been non-normative: performance variability is not necessarily wanted or unwanted, it is merely necessary where system contradictions cannot be easily resolved and so adaptive capacity is required to cope.

No abstract

Objective Health IT (HIT) systems are increasingly becoming a core infrastructural technology in healthcare. However, failures of these systems, under certain conditions, can lead to patient harm and as such the safety case for HIT has to be explicitly made. This study focuses on safety assurance practices of HIT in England and investigates how clinicians and engineers currently analyse, control and justify HIT safety risks. Methods Three workshops were organised, involving 34 clinical and engineering stakeholders, and centred on predefined risk-based questions. This was followed by a detailed review of the Clinical Safety Case Reports for 20 different national and local systems. The data generated was analysed thematically, considering the clinical, engineering and organisational factors, and was used to examine the often implicit safety argument for HIT. Results Two areas of strength were identified: establishment of a systematic approach to risk management and close engagement by clinicians; and two areas for improvement: greater depth and clarity in hazard analysis practices and greater organisational support for assuring safety. Overall, the dynamic characteristics of healthcare combined with insufficient funding have made it challenging to generate and explain the safety evidence to the required level of detail and rigour. Conclusion Improvements in the form of practical HIT-specific safety guidelines and tools are needed. The lack of publicly available examples of credible HIT safety cases is a major deficit. The availability of these examples can help clarify the significance of the HIT risk analysis evidence and identify the necessary expertise and organisational commitments.

No abstract