Length of Stay Prediction in Neurosurgery with Russian GPT-3 Language Model Compared to Human Expectations

Szostek, Małgorzata; Kotik, Konstantin; Shevchenko, E V; Usachev, Dmitriy; Shifrin, Michael; Strunina, Y. V.; Tsukanova, Tatyana; Ishankulov, Timur; Лукшин, В А; Потапов, А. А.

doi:10.3233/shti210882

Cited by 6 publications

(7 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Burdenko National Medical Research Center of Neurosurgery for the period between 2000 and 2017. Data preprocessing and modeling details were thoroughly presented in our previous work (2). We fine-tuned the Generative Pre-trained Transformer 3 model previously trained with large (600+ GB) corpus in the Russian language (ruGPT3) in two stages of 22 epochs totally, using the batch size of top level = 256, overall batch size = 16, learning rate = 1e-5 with L1LOSS loss function, Adam optimizer.…”

Section: Methodsmentioning

confidence: 99%

“…The evidence-based prognosis of hospital stay might be valuable for patients, relatives, doctors, and healthcare providers in high-tech surgery (1). We have recently evaluated the performance of the GPT3 neural network model pre-trained on a massive amount of narrative texts in the Russian language and fine-tuned on the neurosurgical dataset from Electronic Health Records (EHR) to predict the inpatient length of stay (LOS) in neurosurgery (2). We have earlier demonstrated the feasibility of using unstructured medical texts in such a prediction task (3,4).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Predicting the Length of Stay in Neurosurgery with RuGPT-3 Language Model

Danilov

Kotik

Shevchenko

et al. 2022

Studies in Health Technology and Informatics

Self Cite

View full text Add to dashboard Cite

In this study, we update the evaluation of the Russian GPT3 model presented in our previous paper in predicting the length of stay (LOS) in neurosurgery. We aimed to assess the performance the Russian GPT-3 (ruGPT-3) language model in LOS prediction using narrative medical records in neurosurgery compared to doctors’ and patients’ expectations. Doctors appeared to have the most realistic LOS expectations (MAE = 2.54), while the model’s predictions (MAE = 3.53) were closest to the patients’ (MAE = 3.47) but inferior to them (p = 0.011). A detailed analysis showed a solid quality of ruGPT-3 performance based on narrative clinical texts. Considering our previous findings obtained with recurrent neural networks and FastText vector representation, we estimate the new result as important but probably improveable.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Predicting the Length of Stay in Neurosurgery with RuGPT-3 Language Model

Danilov

Kotik

Shevchenko

et al. 2022

Studies in Health Technology and Informatics

Self Cite

View full text Add to dashboard Cite

show abstract

“…1,2 They have been used for a wide range of applications in health care, including predicting length of postsurgical hospital stay, captioning medical images, summarizing radiology reports, and named entity recognition of electronic health record notes. [3][4][5][6] Among these models, ChatGPT (OpenAI) has emerged as a particularly powerful tool based on GPT-3.5 that was designed specifically for the task of generating natural and contextually appropriate responses in a conversational setting. Building on the GPT-3 model, GPT-3.5 was trained on a larger corpus of textual data and with additional training techniques like Reinforcement Learning from Human Feedback (RLHF), which incorporates human knowledge and expertise into the model.…”

Section: Introductionmentioning

confidence: 99%

“…These models, including bidirectional encoder representations from transformers (BERT) and generative pretrained transformer 3 (GPT-3), are trained on massive amounts of text data and excel at natural language processing tasks such as text summarization or responding to queries . They have been used for a wide range of applications in health care, including predicting length of postsurgical hospital stay, captioning medical images, summarizing radiology reports, and named entity recognition of electronic health record notes …”

Section: Introductionmentioning

confidence: 99%

Comparison of Ophthalmologist and Large Language Model Chatbot Responses to Online Patient Eye Care Questions

Bernstein,

Zhang,

Govil

et al. 2023

JAMA Netw Open

View full text Add to dashboard Cite

ImportanceLarge language models (LLMs) like ChatGPT appear capable of performing a variety of tasks, including answering patient eye care questions, but have not yet been evaluated in direct comparison with ophthalmologists. It remains unclear whether LLM-generated advice is accurate, appropriate, and safe for eye patients.ObjectiveTo evaluate the quality of ophthalmology advice generated by an LLM chatbot in comparison with ophthalmologist-written advice.Design, Setting, and ParticipantsThis cross-sectional study used deidentified data from an online medical forum, in which patient questions received responses written by American Academy of Ophthalmology (AAO)–affiliated ophthalmologists. A masked panel of 8 board-certified ophthalmologists were asked to distinguish between answers generated by the ChatGPT chatbot and human answers. Posts were dated between 2007 and 2016; data were accessed January 2023 and analysis was performed between March and May 2023.Main Outcomes and MeasuresIdentification of chatbot and human answers on a 4-point scale (likely or definitely artificial intelligence [AI] vs likely or definitely human) and evaluation of responses for presence of incorrect information, alignment with perceived consensus in the medical community, likelihood to cause harm, and extent of harm.ResultsA total of 200 pairs of user questions and answers by AAO-affiliated ophthalmologists were evaluated. The mean (SD) accuracy for distinguishing between AI and human responses was 61.3% (9.7%). Of 800 evaluations of chatbot-written answers, 168 answers (21.0%) were marked as human-written, while 517 of 800 human-written answers (64.6%) were marked as AI-written. Compared with human answers, chatbot answers were more frequently rated as probably or definitely written by AI (prevalence ratio [PR], 1.72; 95% CI, 1.52-1.93). The likelihood of chatbot answers containing incorrect or inappropriate material was comparable with human answers (PR, 0.92; 95% CI, 0.77-1.10), and did not differ from human answers in terms of likelihood of harm (PR, 0.84; 95% CI, 0.67-1.07) nor extent of harm (PR, 0.99; 95% CI, 0.80-1.22).Conclusions and RelevanceIn this cross-sectional study of human-written and AI-generated responses to 200 eye care questions from an online advice forum, a chatbot appeared capable of responding to long user-written eye health posts and largely generated appropriate responses that did not differ significantly from ophthalmologist-written responses in terms of incorrect information, likelihood of harm, extent of harm, or deviation from ophthalmologist community standards. Additional research is needed to assess patient attitudes toward LLM-augmented ophthalmologists vs fully autonomous AI content generation, to evaluate clarity and acceptability of LLM-generated answers from the patient perspective, to test the performance of LLMs in a greater variety of clinical contexts, and to determine an optimal manner of utilizing LLMs that is ethical and minimizes harm.

show abstract

“…One notable example is the Generative Pretrained Transformer 3 (GPT-3), which has been extensively pre-trained on massive amounts of text data, allowing it to analyze and generate text in various healthcare domains. These LLMs have been found wide applications in the healthcare field through pre-training on a vast amount of textual data and abstract analysis of texts, such as predicting postoperative hospitalization time and generating electronic health records [1,2]. This indicates the potential applications of LLMs in clinical [3,4,5,6,7], educational [8,9,10], and research settings [11,12].…”

Section: Introductionmentioning

confidence: 99%

Infectious Hospital Area, the First Affiliated Hospital of University of Science and Technology of China, Hefei, China

Zhao¹,

Hu²,

Xue³

et al. 2021

BEBM

View full text Add to dashboard Cite

With advances in deep learning, neural network based speech enhancement (SE) has developed rapidly in the last decade. Meanwhile, the self-supervised pre-trained model and vector quantization (VQ) have achieved excellent performance on many speech-related tasks, while they are less explored on SE. As it was shown in our previous work that utilizing a VQ module to discretize noisy speech representations is beneficial for speech denoising, in this work we therefore study the impact of using VQ at different layers with different number of codebooks. Different VQ modules indeed enable to extract multiplegranularity speech features. Following an attention mechanism, the contextual features extracted by a pre-trained model are fused with the local features extracted by the encoder, such that both global and local information are preserved to reconstruct the enhanced speech. Experimental results on the Valentini dataset show that the proposed model can improve the SE performance, where the impact of choosing pre-trained models is also revealed.

show abstract

Length of Stay Prediction in Neurosurgery with Russian GPT-3 Language Model Compared to Human Expectations

Cited by 6 publications

References 3 publications

Predicting the Length of Stay in Neurosurgery with RuGPT-3 Language Model

Predicting the Length of Stay in Neurosurgery with RuGPT-3 Language Model

Comparison of Ophthalmologist and Large Language Model Chatbot Responses to Online Patient Eye Care Questions

Infectious Hospital Area, the First Affiliated Hospital of University of Science and Technology of China, Hefei, China

Contact Info

Product

Resources

About