Evaluation of ChatGPT and Gemini Large Language Models for Pharmacometrics with NONMEM

Abstract

Purpose To assess the ChatGPT 4.0 (ChatGPT) and Gemini Ultra 1.0 (Gemini) large language models on tasks relevant to NONMEM coding in pharmacometrics and clinical pharmacology settings. Methods ChatGPT and Gemini performance on tasks mimicking real-world applications of NONMEM was assessed. The tasks ranged from providing a curriculum for learning NONMEM and an overview of NONMEM code structure to generating code. Prompts to elicit NONMEM code for a linear pharmacokinetic (PK) model with oral administration and a more complex one-compartment model with two parallel first-order absorption mechanisms were investigated. The prompts for all tasks were presented in lay language. The code was carefully reviewed for errors by two experienced NONMEM experts, and the revisions needed to run the code successfully were identified. Results ChatGPT and Gemini provided useful NONMEM curriculum structures combining foundational knowledge with advanced concepts (e.g., covariate modeling and Bayesian approaches) and practical skills, including NONMEM code structure and syntax. Large language models (LLMs) provided an informative summary of the NONMEM control stream structure and outlined the key NM-TRAN records needed. ChatGPT and Gemini were able to generate applicable code blocks for the NONMEM control stream from the lay language prompts for the three coding tasks. The control streams contained focal structural and NONMEM syntax errors that required revision before they could be executed without errors and warnings. Conclusions LLMs may be useful in pharmacometrics for efficiently generating an initial coding template for modeling projects. However, the output can contain errors that require correction.