Can large language models reason about medical questions?

The potential of ChatGPT in medicine: an example analysis of nephrology specialty exams in Poland

“…Similar results were obtained in the first input data, as in the work of Liévin et al . (2023), with a 46% accuracy for the algorithm, with zero suggestions, as well as in neurology exam answers [ 7 , 28 ]. This is also a similar result to a recent paper published by Suwała et al .…”

ChatGPT: The End of Online Exam Integrity?

“…However, while there are valid concerns regarding LLMs' current limitations in handling complex reasoning tasks, there is also accumulating evidence of their improving capabilities [38,56,57]. Within an academic context, Liévin et al [58] conclude that LLMs can effectively answer and reason about medical questions, while a recent survey Chang et al [30] indicates that LLMs perform well in tasks like arithmetic reasoning and demonstrate marked competence in logical reasoning tasks too; though, they do encounter significant challenges with abstract and multi-hop reasoning, struggling particularly with tasks requiring complex, novel, or counterfactual thinking. The ability to self-critique is necessary for advanced reasoning that supports rational decision-making and problem-solving, and Luo et al [59] demonstrate the difficulty of achieving this within LLMs; however, they show how an improvement in LLM's performances on reasoning tasks can be elicited through advanced prompting techniques involving self-critique.…”

Comparative Analysis of Artificial Intelligence Virtual Assistant and Large Language Models in Post-Operative Care

“…That said, there is a substantial body of research focused on the application of LLMs and AIVAs in specialized patient education tasks. These studies evaluate the feasibility, accuracy, and suitability of these technologies for responding to inquiries across various medical fields [9,15,[20][21][22][23]. Some studies have enhanced chatbots using LLMs, successfully simulating the patient-physician dynamic.…”