Unintended Consequences of Machine Learning in Medicine

Cabitza, Federico; Rasoini, Raffaele; Gensini, Gian Franco

doi:10.1001/jama.2017.7797

Cited by 630 publications

(426 citation statements)

References 10 publications

Supporting

Mentioning

414

Contrasting

Unclassified

Order By: Relevance

“…The consequences of automation on human performance can pose serious safety concerns, with risks depending on the level of automation and the type of automated function 10 , 56 , 57 . Therefore, the use of conversational agents with unconstrained natural language input capabilities and other artificial intelligence applications in healthcare needs to be carefully monitored 57 , 58 …”

Section: Discussionmentioning

confidence: 99%

Conversational agents in healthcare: a systematic review

Laranjo

Dunn

Tong

et al. 2018

Journal of the American Medical Informatics Association

711

492

View full text Add to dashboard Cite

ObjectiveOur objective was to review the characteristics, current applications, and evaluation measures of conversational agents with unconstrained natural language input capabilities used for health-related purposes.MethodsWe searched PubMed, Embase, CINAHL, PsycInfo, and ACM Digital using a predefined search strategy. Studies were included if they focused on consumers or healthcare professionals; involved a conversational agent using any unconstrained natural language input; and reported evaluation measures resulting from user interaction with the system. Studies were screened by independent reviewers and Cohen’s kappa measured inter-coder agreement.ResultsThe database search retrieved 1513 citations; 17 articles (14 different conversational agents) met the inclusion criteria. Dialogue management strategies were mostly finite-state and frame-based (6 and 7 conversational agents, respectively); agent-based strategies were present in one type of system. Two studies were randomized controlled trials (RCTs), 1 was cross-sectional, and the remaining were quasi-experimental. Half of the conversational agents supported consumers with health tasks such as self-care. The only RCT evaluating the efficacy of a conversational agent found a significant effect in reducing depression symptoms (effect size d = 0.44, p = .04). Patient safety was rarely evaluated in the included studies.ConclusionsThe use of conversational agents with unconstrained natural language input capabilities for health-related purposes is an emerging field of research, where the few published studies were mainly quasi-experimental, and rarely evaluated efficacy or safety. Future studies would benefit from more robust experimental designs and standardized reporting.Protocol RegistrationThe protocol for this systematic review is registered at PROSPERO with the number CRD42017065917.

show abstract

Section: Discussionmentioning

confidence: 99%

Conversational agents in healthcare: a systematic review

Laranjo

Dunn

Tong

et al. 2018

Journal of the American Medical Informatics Association

711

492

View full text Add to dashboard Cite

show abstract

“…The paradox is that these methods, despite their advantages, are far from universal acceptance in medical practice. Arguably, one of the reasons is precisely (lack of) interpretability, expressed as "the need to open the machine learning black box" [13]. As already mentioned, DL-based technologies can worsen the problem, despite having already found their way into biomedicine and healthcare [14,15].…”

Section: Interpretability and Explainabilitymentioning

confidence: 99%

Societal Issues Concerning the Application of Artificial Intelligence in Medicine

Vellido

2018

Kidney Dis

View full text Add to dashboard Cite

Background: Medicine is becoming an increasingly data-centred discipline and, beyond classical statistical approaches, artificial intelligence (AI) and, in particular, machine learning (ML) are attracting much interest for the analysis of medical data. It has been argued that AI is experiencing a fast process of commodification. This characterization correctly reflects the current process of industrialization of AI and its reach into society. Therefore, societal issues related to the use of AI and ML should not be ignored any longer and certainly not in the medical domain. These societal issues may take many forms, but they all entail the design of models from a human-centred perspective, incorporating human-relevant requirements and constraints. In this brief paper, we discuss a number of specific issues affecting the use of AI and ML in medicine, such as fairness, privacy and anonymity, explainability and interpretability, but also some broader societal issues, such as ethics and legislation. We reckon that all of these are relevant aspects to consider in order to achieve the objective of fostering acceptance of AI- and ML-based technologies, as well as to comply with an evolving legislation concerning the impact of digital technologies on ethically and privacy sensitive matters. Our specific goal here is to reflect on how all these topics affect medical applications of AI and ML. This paper includes some of the contents of the “2nd Meeting of Science and Dialysis: Artificial Intelligence,” organized in the Bellvitge University Hospital, Barcelona, Spain. Summary and Key Messages: AI and ML are attracting much interest from the medical community as key approaches to knowledge extraction from data. These approaches are increasingly colonizing ambits of social impact, such as medicine and healthcare. Issues of social relevance with an impact on medicine and healthcare include (although they are not limited to) fairness, explainability, privacy, ethics and legislation.

show abstract

“…It has been argued, though, that this application might lead to a reduction of skills among medical experts. This negative consequence of the use of ML methods in medicine has been described as ML methods' undue "focus on text and the demise of context" [12]. The second example involves the implementation of a European Union directive for general data protection regulation that will be enforced in 2018 and mandates a right to explanation of all decisions made by automated or AI algorithmic systems [13].…”

Section: Present Artificial Intelligence In Medicine: Challenges For mentioning

confidence: 99%

“…Three main challenges for the application of ML in medicine have recently been listed [12], and one of them is precisely interpretability, expressed as "the need to open the machine learning black box." Not that this is a new challenge for ML, because the black box syndrome was already on the table decades ago [16].…”

Section: Present Artificial Intelligence In Medicine: Challenges For mentioning

confidence: 99%

Artificial Intelligence for the Artificial Kidney: Pointers to the Future of a Personalized Hemodialysis Therapy

et al. 2018

View full text Add to dashboard Cite

Background: Current dialysis devices are not able to react when unexpected changes occur during dialysis treatment or to learn about experience for therapy personalization. Furthermore, great efforts are dedicated to develop miniaturized artificial kidneys to achieve a continuous and personalized dialysis therapy, in order to improve the patient’s quality of life. These innovative dialysis devices will require a real-time monitoring of equipment alarms, dialysis parameters, and patient-related data to ensure patient safety and to allow instantaneous changes of the dialysis prescription for the assessment of their adequacy. The analysis and evaluation of the resulting large-scale data sets enters the realm of “big data” and will require real-time predictive models. These may come from the fields of machine learning and computational intelligence, both included in artificial intelligence, a branch of engineering involved with the creation of devices that simulate intelligent behavior. The incorporation of artificial intelligence should provide a fully new approach to data analysis, enabling future advances in personalized dialysis therapies. With the purpose to learn about the present and potential future impact on medicine from experts in artificial intelligence and machine learning, a scientific meeting was organized in the Hospital Universitari Bellvitge (L’Hospitalet, Barcelona). As an outcome of that meeting, the aim of this review is to investigate artificial intel ligence experiences on dialysis, with a focus on potential barriers, challenges, and prospects for future applications of these technologies. Summary and Key Messages: Artificial intelligence research on dialysis is still in an early stage, and the main challenge relies on interpretability and/or comprehensibility of data models when applied to decision making. Artificial neural networks and medical decision support systems have been used to make predictions about anemia, total body water, or intradialysis hypotension and are promising approaches for the prescription and monitoring of hemodialysis therapy. Current dialysis machines are continuously improving due to innovative technological developments, but patient safety is still a key challenge. Real-time monitoring systems, coupled with automatic instantaneous biofeedback, will allow changing dialysis prescriptions continuously. The integration of vital sign monitoring with dialysis parameters will produce large data sets that will require the use of data analysis techniques, possibly from the area of machine learning, in order to make better decisions and increase the safety of patients.

show abstract

Unintended Consequences of Machine Learning in Medicine

Cited by 630 publications

References 10 publications

Conversational agents in healthcare: a systematic review

Conversational agents in healthcare: a systematic review

Societal Issues Concerning the Application of Artificial Intelligence in Medicine

Artificial Intelligence for the Artificial Kidney: Pointers to the Future of a Personalized Hemodialysis Therapy

Contact Info

Product

Resources

About