Towards an AI Tutor for Undergraduate Geotechnical Engineering: A Comparative Study of Evaluating the Efficiency of Large Language Model Application Programming Interfaces

Abstract

This study investigates the efficiency of Large Language Model (LLM) Application Programming Interfaces (APIs)—specifically GPT-4 and Llama-3—as AI tutors for undergraduate Geotechnical Engineering education. As educational needs in specialised fields like Geotechnical Engineering become increasingly complex, innovative teaching tools that provide personalised learning experiences are essential. This research evaluates the capabilities of GPT-4’s and Llama-3’s APIs in integrating and applying Geotechnical Engineering formulas, offering accurate problem-solving and explanatory responses, and adapting to varied educational requirements. Using comparative analysis, the study employs a formula integration approach known as Retrieval-Augmented Generation (RAG) with two widely used LLM models, GPT-4 and Llama-3. A set of 20 challenging questions, previously identified as problematic for zero-shot solutions for GPT-4, served as the evaluation basis. The models were assessed on accuracy, formula integration, clarity of explanation, and problem-solving adaptability. Results indicate that GPT-4 and Llama-3 have significant potential as AI tutors in Geotechnical Engineering. GPT-4, utilising RAG, demonstrated superior performance, correctly answering 95% of the questions at a temperature setting of 0.1, 82.5% at 0.5, and 60% at 1. In contrast, Llama-3 correctly answered 25% of the questions in zero-shot tasks and 45% with API by setting a temperature of 0.1. The study underscores the need for advanced formula integration techniques and domain-specific training to enhance the educational utility of LLM APIs. Future research should focus on refining formula integration methods, expanding domain-specific knowledge bases, and assessing long-term learning outcomes. This work contributes to the ongoing dialogue on AI in education, providing insights into deploying LLMs as personalised, effective teaching aids in engineering disciplines.