Re-Evaluating GPT-4's Bar Exam Performance

Martínez, Eric J. Alfaro

doi:10.2139/ssrn.4441311

Cited by 6 publications

(4 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this article, the authors assess 2 of the world's most prominent large language models (LLMs) on the Self-Assessment Neurosurgery (SANS) Exams for the American Board of Neurological Surgery (ABNS) written exam and find that the LLMs perform better than the average human test taker. These findings are consistent with the performance of LLMs on other exams, 1a although GPT-4's superhuman results on the Bar Exam 2a have recently been called into question. The challenge of evaluating claims around these models, which themselves have not been reported in any scientific publications due to “competitive concerns,” raises many questions and underscores the importance of independent evaluations such as this project.…”

Section: Commentssupporting

confidence: 72%

Performance of ChatGPT and GPT-4 on Neurosurgery Written Board Examinations

et al. 2023

View full text Add to dashboard Cite

BACKGROUND AND OBJECTIVES: Interest surrounding generative large language models (LLMs) has rapidly grown. Although ChatGPT (GPT-3.5), a general LLM, has shown near-passing performance on medical student board examinations, the performance of ChatGPT or its successor GPT-4 on specialized examinations and the factors affecting accuracy remain unclear. This study aims to assess the performance of ChatGPT and GPT-4 on a 500-question mock neurosurgical written board examination. METHODS: The Self-Assessment Neurosurgery Examinations (SANS) American Board of Neurological Surgery Self-Assessment Examination 1 was used to evaluate ChatGPT and GPT-4. Questions were in single best answer, multiple-choice format. χ2, Fisher exact, and univariable logistic regression tests were used to assess performance differences in relation to question characteristics. RESULTS: ChatGPT (GPT-3.5) and GPT-4 achieved scores of 73.4% (95% CI: 69.3%-77.2%) and 83.4% (95% CI: 79.8%-86.5%), respectively, relative to the user average of 72.8% (95% CI: 68.6%-76.6%). Both LLMs exceeded last year's passing threshold of 69%. Although scores between ChatGPT and question bank users were equivalent (P = .963), GPT-4 outperformed both (both P < .001). GPT-4 answered every question answered correctly by ChatGPT and 37.6% (50/133) of remaining incorrect questions correctly. Among 12 question categories, GPT-4 significantly outperformed users in each but performed comparably with ChatGPT in 3 (functional, other general, and spine) and outperformed both users and ChatGPT for tumor questions. Increased word count (odds ratio = 0.89 of answering a question correctly per +10 words) and higher-order problem-solving (odds ratio = 0.40, P = .009) were associated with lower accuracy for ChatGPT, but not for GPT-4 (both P > .005). Multimodal input was not available at the time of this study; hence, on questions with image content, ChatGPT and GPT-4 answered 49.5% and 56.8% of questions correctly based on contextual context clues alone. CONCLUSION: LLMs achieved passing scores on a mock 500-question neurosurgical written board examination, with GPT-4 significantly outperforming ChatGPT.

show abstract

Section: Commentssupporting

confidence: 72%

Performance of ChatGPT and GPT-4 on Neurosurgery Written Board Examinations

et al. 2023

View full text Add to dashboard Cite

show abstract

“…GPT 4, as per the latest research, demonstrates pro ciency comparable to human performance across a variety of domains, including medicine (Nori et al, 2023), law (Martínez, 2023), and cognitive psychology (Dhingra et al, 2023). This signi cant advancement is presumably attributable to incorporating Reinforcement Learning from Human Feedback (RLHF) during its training phase, coupled with a more voluminous training data corpus.…”

Section: Discussionmentioning

confidence: 96%

Identification and Description of Emotions by Current Large Language Models

Patel,

Fan

2024

Preprint

View full text Add to dashboard Cite

The assertion that artificial intelligence (AI) cannot grasp the complexities of human emotions has been a long-standing debate. However, recent advancements in large language models (LLMs) challenge this notion by demonstrating an increased capacity for understanding and generating human-like text. In this study, we evaluated the empathy levels and the identification and description of emotions by three current language models: Bard, GPT 3.5, and GPT 4. We used the Toronto Alexithymia Scale (TAS-20) and the 60-question Empathy Quotient (EQ-60) questions to prompt these models and score the responses. The models' performance was contrasted with human benchmarks of neurotypical controls and clinical populations. We found that the less sophisticated models (Bard and GPT 3.5) performed inferiorly on TAS-20, aligning close to alexithymia, a condition with significant difficulties in recognizing, expressing, and describing one's or others' experienced emotions. However, GPT 4 achieved performance close to the human level. These results demonstrated that LLMs are comparable in their ability to identify and describe emotions and may be able to surpass humans in their capacity for emotional intelligence. Our novel insights provide alignment research benchmarks and a methodology for aligning AI with human values, leading toward an empathetic AI that mitigates risk.

show abstract

“…However, using a percentile chart from a February 2018 exam administration (which is generally available online), ChatGPT would receive a score below the 10th percentile of test-takers while GPT-4 would receive a combined score approaching the 90th percentile of test-takers. However, it should be noted that this chart might not be the best approach to the estimation given the skew towards 'retakers' in the February exam administration[78]. While we are not fully convinced of the methodological approach taken in some subsequent analysis[78], we do agree that it would be better to consider the raw 297 UBE as falling within a range between 68th and 90th percentile (depending on the precise state and timing of the exam administration).…”

mentioning

confidence: 83%

“…However, it should be noted that this chart might not be the best approach to the estimation given the skew towards 'retakers' in the February exam administration[78]. While we are not fully convinced of the methodological approach taken in some subsequent analysis[78], we do agree that it would be better to consider the raw 297 UBE as falling within a range between 68th and 90th percentile (depending on the precise state and timing of the exam administration). See table 8, electronic supplementary material, for additional information.…”

mentioning

confidence: 83%

GPT-4 passes the bar exam

Katz,

Bommarito,

Gao

et al. 2024

Phil. Trans. R. Soc. A.

View full text Add to dashboard Cite

In this paper, we experimentally evaluate the zero-shot performance of GPT-4 against prior generations of GPT on the entire uniform bar examination (UBE), including not only the multiple-choice multistate bar examination (MBE), but also the open-ended multistate essay exam (MEE) and multistate performance test (MPT) components. On the MBE, GPT-4 significantly outperforms both human test-takers and prior models, demonstrating a 26% increase over ChatGPT and beating humans in five of seven subject areas. On the MEE and MPT, which have not previously been evaluated by scholars, GPT-4 scores an average of 4.2/6.0 when compared with much lower scores for ChatGPT. Graded across the UBE components, in the manner in which a human test-taker would be, GPT-4 scores approximately 297 points, significantly in excess of the passing threshold for all UBE jurisdictions. These findings document not just the rapid and remarkable advance of large language model performance generally, but also the potential for such models to support the delivery of legal services in society. This article is part of the theme issue ‘A complexity science approach to law and governance’.

show abstract

Re-Evaluating GPT-4's Bar Exam Performance

Cited by 6 publications

References 13 publications

Performance of ChatGPT and GPT-4 on Neurosurgery Written Board Examinations

Performance of ChatGPT and GPT-4 on Neurosurgery Written Board Examinations

Identification and Description of Emotions by Current Large Language Models

GPT-4 passes the bar exam

Contact Info

Product

Resources

About