Challenging ChatGPT 3.5 in Senology—An Assessment of Concordance with Breast Cancer Tumor Board Decision Making

Abstract

With the recent diffusion of access to publicly available large language models (LLMs), common interest in generative artificial-intelligence-based applications for medical purposes has skyrocketed. The increased use of these models by tech-savvy patients for personal health issues calls for a scientific evaluation of whether LLMs provide a satisfactory level of accuracy for treatment decisions. This observational study compares the concordance of treatment recommendations from the popular LLM ChatGPT 3.5 with those of a multidisciplinary tumor board for breast cancer (MTB). The study design builds on previous findings by combining an extended input model with patient profiles reflecting patho- and immunomorphological diversity of primary breast cancer, including primary metastasis and precancerous tumor stages. Overall concordance between the LLM and MTB is reached for half of the patient profiles, including precancerous lesions. In the assessment of invasive breast cancer profiles, the concordance amounts to 58.8%. Nevertheless, as the LLM makes considerably fraudulent decisions at times, we do not identify the current development status of publicly available LLMs to be adequate as a support tool for tumor boards. Gynecological oncologists should familiarize themselves with the capabilities of LLMs in order to understand and utilize their potential while keeping in mind potential risks and limitations.

Utilizing large language models in breast cancer management: systematic review

“…Three studies (50.0%) evaluated the performance of ChatGPT on actual patient data (Sorin et al 2023a , b , c ; Choi et al 2023 ; Lukac et al 2023 ), as opposed to two studies that used data from the internet (Rao et al 2023 ; Haver et al 2023 ). One study crafted fictitious patient profiles by the head investigator (Griewing et al 2023 ).…”

“…Rao et al and Haver et al evaluated LLMs for breast imaging recommendations (Rao et al 2023 ; Haver et al 2023 ). Sorin et al, Lukac et al and Griewing et al evaluated LLMs as supportive decision-making tools in multidisciplinary tumor boards (Sorin et al 2023a , b , c ; Lukac et al 2023 ; Griewing et al 2023 ). Choi et al used LLM for information extraction from ultrasound and pathology reports (Choi et al 2023 ) (Fig.…”

“…Best performance rates were achieved for information extraction and question-answering, with correct responses ranging from 88 to 98% (Choi et al 2023 ; Rao et al 2023 ) (Table 3 ). The lower performance was for clinical decision support in breast tumor boards, ranging between 50 and 70% (Sorin et al 2023a , b , c ; Lukac et al 2023 ; Griewing et al 2023 ). The range in performance on this task was wide between studies.…”

“…The range in performance on this task was wide between studies. However, the methods of the three studies also varied significantly (Sorin et al 2023a , b , c ; Lukac et al 2023 ; Griewing et al 2023 ). Sorin et al and Lukac et al used authentic patient data and compared ChatGPT-3.5 to the retrospective decisions in breast tumor board (Sorin et al 2023a , b , c ; Lukac et al 2023 ).…”

“…In both studies, the authors used reviewers that scored ChatGPT-3.5 responses (Sorin et al 2023a , b , c ; Lukac et al 2023 ). (Griewing et al 2023 ) crafted 20 fictitious patient files that were then discussed by a multidisciplinary tumor board. Their assessment was based on binary evaluation of various treatment approaches, including surgery, endocrine, chemotherapy, and radiation therapy.…”