Can artificial intelligence pass the Fellowship of the Royal College of Radiologists examination? Multi-reader diagnostic accuracy study

Abstract

Objective To determine whether an artificial intelligence candidate could pass the rapid (radiographic) reporting component of the Fellowship of the Royal College of Radiologists (FRCR) examination. Design Prospective multi-reader diagnostic accuracy study. Setting United Kingdom. Participants One artificial intelligence candidate (Smarturgences, Milvue) and 26 radiologists who had passed the FRCR examination in the preceding 12 months. Main outcome measures Accuracy and pass rate of the artificial intelligence compared with radiologists across 10 mock FRCR rapid reporting examinations (each examination containing 30 radiographs, requiring 90% accuracy rate to pass). Results When non-interpretable images were excluded from the analysis, the artificial intelligence candidate achieved an average overall accuracy of 79.5% (95% confidence interval 74.1% to 84.3%) and passed two of 10 mock FRCR examinations. The average radiologist achieved an average accuracy of 84.8% (76.1-91.9%) and passed four of 10 mock examinations. The sensitivity for the artificial intelligence was 83.6% (95% confidence interval 76.2% to 89.4%) and the specificity was 75.2% (66.7% to 82.5%), compared with summary estimates across all radiologists of 84.1% (81.0% to 87.0%) and 87.3% (85.0% to 89.3%). Across 148/300 radiographs that were correctly interpreted by >90% of radiologists, the artificial intelligence candidate was incorrect in 14/148 (9%). In 20/300 radiographs that most (>50%) radiologists interpreted incorrectly, the artificial intelligence candidate was correct in 10/20 (50%). Most imaging pitfalls related to interpretation of musculoskeletal rather than chest radiographs. Conclusions When special dispensation for the artificial intelligence candidate was provided (that is, exclusion of non-interpretable images), the artificial intelligence candidate was able to pass two of 10 mock examinations. Potential exists for the artificial intelligence candidate to improve its radiographic interpretation skills by focusing on musculoskeletal cases and learning to interpret radiographs of the axial skeleton and abdomen that are currently considered “non-interpretable.”

Analysis of large-language model versus human performance for genetics questions

“…7 Overall, approaches using LLMs have shown that they can perform relatively well in a variety of medical testing situations. 4,5 These scores are sure to improve, perhaps rapidly, and tools like ChatGPT may become ubiquitous, much the way various forms of AI are currently used in clinical and research genomic analyses every day. 8,9 As one of many examples, patients and research subjects may increasingly turn to ChatGPT and other such interfaces to learn about conditions that may affect them, including the suggested work-up and treatment.…”

“…[1][2][3] Recently, large-language models (LLMs) like ChatGPT (https://chat.openai.com/chat) have been receiving attention in many venues, including via demonstration of medical knowledge. 4,5 By way of background, LLMs use a specific type of DL called a transformer. By training the model on a large dataset of text, it learns to predict the next word in a sentence or set of words following a prompt such as a question.…”

Efficacy of Artificial Intelligence in the Categorisation of Paediatric Pneumonia on Chest Radiographs: A Systematic Review

“…This algorithm can detect seven different pathologies (fracture, pleural effusion, lung opacification, joint effusion, lung nodules, pneumothorax, and joint dislocation). A recent study which compared this AI algorithm’s ability to human radiologists showed that the algorithm failed to pass the Fellowship of Royal College of Radiologists Rapid Reporting examinations [ 57 ]. This exam is normally taken between radiology speciality training years four and five in the UK [ 58 ].…”

Robots, radiologists, and results

“…Now, Susan Shelmerdine and UK based co-authors have asked whether an artificial intelligence candidate could pass the Fellowship of the Royal College of Radiologists (FRCR) examination, which allows radiology candidates to obtain the diploma for the licence to practise (doi:10.1136/bmj-2022-072826). 9…”