Using Machine Learning to Assess Physician Competence: A Systematic Review

Dias, Roger D.; Gupta, Avni; Yule, Steven

doi:10.1097/acm.0000000000002414

Cited by 48 publications

(49 citation statements)

References 82 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The second review was systematic and addressed assessment of physician competence. 8 Of 69 articles, only six focused on some interpersonal and communication skills. Two of these analysed written performance reports, one analysed the quality of clinical notes in health records, and two provided automated feedback to medical students after interacting with SPs-but only for non-verbal communication skills such as body language and tone.…”

Section: To Address Shortcomings Bias and Costs Ryan Et Al And Butowmentioning

confidence: 99%

Machine learning to extract communication and history‐taking skills in OSCE transcripts

Jani

Jones

et al. 2020

Medical Education

View full text Add to dashboard Cite

Objectives: Observed Structured Clinical Exams (OSCEs) allow assessment of, and provide feedback to, medical students. Clinical examiners and standardised patients (SP) typically complete itemised checklists and global scoring scales, which have known shortcomings. In this study, we applied machine learning (ML) to label some communication skills and interview content information in OSCE transcripts and to compare several ML methodologies by performance and transferability. Methods: One-hundred and twenty-one transcripts of two OSCE scenarios were manually annotated per utterance across 19 communication skills and content areas. Utterances were converted to two types of numeric sentence vector representations and were paired with three types of ML algorithms. First, ML models (MLMs) were evaluated using a five K-fold cross-validation technique on all transcripts in one scenario to generate precision and recall, and their harmonic mean, F1 scores. Second, ML models were trained on all 101 transcripts from scenario 1 and tested for transferability on 20 scenario 2 transcripts. Results: Performance testing in the K-fold cross-validation demonstrated relatively high mean F1 scores: median 0.87 and range 0.53-0.98 across all 19 labels. Transferability testing demonstrated success: F1 median 0.76 and range 0.46-0.97. The combination of a bi-directional long short-term memory neural network (biL-STM) algorithm with GenSen numeric sentence vector representations was associated with greater F1 scores across both performance and transferability (P < .005). Conclusions: We report the first application of ML in the context of student-SP OSCEs. We demonstrated that several MLMs automatically labelled OSCE transcripts for a range of interview content and some clinical communications skills. Some MLMs achieved greater performance and transferability. Optimised MLMs could provide automated and accurate assessment of OSCEs with potential to track student progress and identify areas for further practice.

show abstract

Section: To Address Shortcomings Bias and Costs Ryan Et Al And Butowmentioning

confidence: 99%

Machine learning to extract communication and history‐taking skills in OSCE transcripts

Jani

Jones

et al. 2020

Medical Education

View full text Add to dashboard Cite

show abstract

“…More generally, as with previous generations of information and computing technology, the introduction of AI into hospitals and health care settings is likely to lead to a shift in power and authority away from frontline practitioners to those who manage and design the IT systems 14 . Finally, research is already being directed toward using AI to monitor physician performance, 15 suggesting that physician surveillance will be one of the first uses of AI in the health sector. Physicians who are demoralized, disempowered, concerned for their jobs, and feel themselves to be under surveillance are ill placed to win political victories.…”

Section: Essaymentioning

confidence: 99%

High Hopes for “Deep Medicine”? AI, Economics, and the Future of Care

Sparrow¹,

Hatherley²

2020

Hastings Center Report

View full text Add to dashboard Cite

In the much‐celebrated book Deep Medicine, Eric Topol argues that the development of artificial intelligence for health care will lead to a dramatic shift in the culture and practice of medicine. In the next several decades, he suggests, AI will become sophisticated enough that many of the everyday tasks of physicians could be delegated to it. Topol is perhaps the most articulate advocate of the benefits of AI in medicine, but he is hardly alone in spruiking its potential to allow physicians to dedicate more of their time and attention to providing empathetic care for their patients in the future. Unfortunately, several factors suggest a radically different picture for the future of health care. Far from facilitating a return to a time of closer doctor‐patient relationships, the use of medical AI seems likely to further erode therapeutic relationships and threaten professional and patient satisfaction.

show abstract

“…The algorithms, namely Trees (e.g., random forest), Support vector Machine, and neural networks, are modeling approaches that can be used with different types of learning tasks, including supervised, unsupervised, semisupervised, and reinforcement learning. We searched project titles, project abstracts, and project terms using the following keywords: 10,11 "artificial intelligence," "Bayesian learning," "boosting," "gradient boosting," "computational intelligence," "computer reasoning," "deep learning," "machine intelligence," "machine learning," "naive Bayes," "neural network," "neural networks," "networks analysis," "natural language processing," "support vector machines," "random forest," "computer vision systems," and "deep networks." Alternative versions of these keywords have been tested to ascertain if abstracts could be identified, and those found useful have been included in the final list of keywords.…”

Section: Study Sample and Search Strategymentioning

confidence: 99%

The National Institutes of Health funding for clinical research applying machine learning techniques in 2017

et al. 2020

View full text Add to dashboard Cite

Machine learning (ML) techniques have become ubiquitous and indispensable for solving intricate problems in most disciplines. To determine the extent of funding for clinical research projects applying ML techniques by the National Institutes of Health (NIH) in 2017, we searched the NIH Research Portfolio Online Reporting Tools Expenditures and Results (RePORTER) system using relevant keywords. We identified 535 projects, which together received a total of $264 million, accounting for 2% of the NIH extramural budget for clinical research.npj Digital Medicine (2020) 3:13 ; https://doi.

show abstract

Using Machine Learning to Assess Physician Competence: A Systematic Review

Cited by 48 publications

References 82 publications

Machine learning to extract communication and history‐taking skills in OSCE transcripts

Machine learning to extract communication and history‐taking skills in OSCE transcripts

High Hopes for “Deep Medicine”? AI, Economics, and the Future of Care

The National Institutes of Health funding for clinical research applying machine learning techniques in 2017

Contact Info

Product

Resources

About