The Performance of Large Language Models on Quantitative and Verbal Ability Tests: Initial Evidence and Implications for Unproctored High-stakes Testing

Abstract

Abstract. Unproctored assessments are widely used in pre-employment assessment. However, the recent emergence of widely accessible large language models (LLMs) poses challenges for unproctored personnel assessments, given that applicants may use them to artificially inflate their scores beyond their true abilities. This may be particularly concerning in cognitive ability testing, which is widely used and is less fakeable by humans than personality tests. Thus, this study compares the performance of LLMs on two common types of cognitive tests: quantitative ability and verbal ability. The particular tests investigated are used in real-world, high-stakes selection. We also examine the performance of the LLMs across different test formats (i.e., open-ended vs. multiple choice). Further, we contrast the performance of two LLMs (GPT 3.5 and GPT 4) across multiple prompt approaches and temperature settings. We find that the LLMs score much better, in terms of percentile scores, on the verbal ability test than the quantitative ability test, even when accounting for the test format. GPT 4 outperforms GPT 3.5 across both types of tests. Notably, although prompt approaches and temperature settings do affect LLM test performance, the effects are minor relative to differences across tests and language models. We provide recommendations for securing pre-employment testing against LLM influences. Additionally, we call for rigorous research investigating the prevalence of LLM usage in pre-employment testing as well as on how LLM usage influences selection test validity.