The imperative for regulatory oversight of large language models (or generative AI) in healthcare

Abstract

The rapid advancements in artificial intelligence (AI) have led to the development of sophisticated large language models (LLMs) such as GPT-4 and Bard. The potential implementation of LLMs in healthcare settings has already garnered considerable attention because of their diverse applications that include facilitating clinical documentation, obtaining insurance pre-authorization, summarizing research papers, or working as a chatbot to answer questions for patients about their specific data and concerns. While offering transformative potential, LLMs warrant a very cautious approach since these models are trained differently from AI-based medical technologies that are regulated already, especially within the critical context of caring for patients. The newest version, GPT-4, that was released in March, 2023, brings the potentials of this technology to support multiple medical tasks; and risks from mishandling results it provides to varying reliability to a new level. Besides being an advanced LLM, it will be able to read texts on images and analyze the context of those images. The regulation of GPT-4 and generative AI in medicine and healthcare without damaging their exciting and transformative potential is a timely and critical challenge to ensure safety, maintain ethical standards, and protect patient privacy. We argue that regulatory oversight should assure medical professionals and patients can use LLMs without causing harm or compromising their data or privacy. This paper summarizes our practical recommendations for what we can expect from regulators to bring this vision to reality.

Large Language Models in Neurology Research and Future Practice

“…Many areas of regulation lie outside the scope of this article, although there are several regulatory issues that are particularly important with respect to neurology and more broadly, medicine, that pertain to the technical and ethical challenges we raise. 19,47,48…”

“…Many areas of regulation lie outside the scope of this article, although there are several regulatory issues that are particularly important with respect to neurology and more broadly, medicine, that pertain to the technical and ethical challenges we raise. 19,47,48 Software as a medical device (SaMD) is defined by the International Medical Device Regulators Forum (IMDRF), as software that is not embedded within hardware and which performs medical tasks. Therefore, many medical LLMs would fall under this umbrella.…”

Will Generative Artificial Intelligence Deliver on Its Promise in Health Care?

“…Given that even genAI developers frequently do not completely understand how their own tools come up with certain outputs, it is possible that a patient-facing output of a genAI tool may change in a harmful way that escapes notice. The US Food and Drug Administration is currently grappling with the question of how to regulate genAI tools in health care settings, mindful that their output may change from one minute to the next …”

Strengthening ethical guardrails for emerging technology businesses

“…While the need to wait for clear and sufficient evidence of harm cannot be bypassed, regulatory agencies can ensure that there is no additional delay on their part through continuous monitoring so that necessary regulations are introduced in time as soon as they are warranted – that is, as soon as the necessary information is available (Meskó and Topol, 2023). They can also issue preliminary best practices and seek voluntary compliance while they collect the necessary information (Mandel, 2009).…”