The ChatGPT Artificial Intelligence Chatbot: How Well Does It Answer Accounting Assessment Questions?

Abstract

ChatGPT, a language-learning model chatbot, has garnered considerable attention for its ability to respond to users’ questions. Using data from 14 countries and 186 institutions, we compare ChatGPT and student performance for 28,085 questions from accounting assessments and textbook test banks. As of January 2023, ChatGPT provides correct answers for 56.5 percent of questions and partially correct answers for an additional 9.4 percent of questions. When considering point values for questions, students significantly outperform ChatGPT with a 76.7 percent average on assessments compared to 47.5 percent for ChatGPT if no partial credit is awarded and 56.5 percent if partial credit is awarded. Still, ChatGPT performs better than the student average for 15.8 percent of assessments when we include partial credit. We provide evidence of how ChatGPT performs on different question types, accounting topics, class levels, open/closed assessments, and test bank questions. We also discuss implications for accounting education and research.

Using large language models to write theses and dissertations

“…However, these systems provide more active support, beyond queries, such as providing help associated with programming and coding. Systems such as ChatGPT also provide homework assistance and answers to both students and faculty to class questions (Wood et al, 2023). In addition, these systems can summarize documents.…”

Evaluating the Performance of ChatGPT in Accounting and Auditing Exams: An Experimental Study in North Macedonia

“…Frey and Osborne (2017) assess the vulnerability of job roles to automation, highlighting a significant likelihood of AI taking over accounting positions. In this context, Wood et al (2023) suggest that the AI chatbots' accuracy in responding to accounting assessment questions may have implications for how AI's capacity to substitute accountants in practical applications is perceived. Due to the extensive incorporation of contemporary information technologies (IT) in the accounting, including big data, data analytics, AI, blockchain, cloud computing, and others, there has been a shift in market demands for accounting proficiency.…”

“…In a survey carried out by Constantz (2023), approximately 30% of the participants reported attempting to utilize ChatGPT in their workplace. Wood et al (2023) present valuable insights into ChatGPT's performance in accounting education, utilizing a vast dataset contributed by a diverse team of 327 coauthors from 186 educational institutions worldwide. Their study encompassed an impressive 25,817 questions drawn from 869 distinct class assessments, along with an additional 2,268 questions sourced from textbook test banks, spanning topics like accounting information systems, auditing, financial accounting, management accounting, and tax.…”

“…ChatGPT is a large language-learning model that employs machine learning to produce human-like text in natural language (Susnjak 2022). It represents a "General Purpose Technology", as typically defined in broader economic or technological contexts (Wood et al, 2023). When asked "What are you and how are you designed", the answer was as follows: "I am ChatGPT, a computer program developed by OpenAI.…”

“…Our findings have the potential to revolutionize how students approach and excel in these challenging subjects, making a significant contribution to the ongoing discussion about AI's role in education. Such experimental research should raise debates about the use of AI tools in higher education and develop a larger level discussion on the following questions, raised by Wood et al, (2023): In what ways should students be granted access to AI? Which material should be memorized versus referenced?…”