Potentiality of generative AI tools in higher education: Evaluating ChatGPT's viability as a teaching assistant for introductory programming courses

Abstract

<abstract> <p>With the advent of large language models like ChatGPT, there is interest in leveraging these tools as teaching assistants in higher education. However, important questions remain regarding the effectiveness and appropriateness of AI systems in educational settings. This study evaluated ChatGPT's potential as a teaching assistant for an introductory programming course. We conducted an experimental study where ChatGPT was prompted in response to common student questions and misconceptions from a first-year programming course. This study was conducted over a period of 2 weeks with 20 undergraduate students and 5 faculty members from the department of computer science. ChatGPT's responses were evaluated along several dimensions—accuracy, completeness, pedagogical soundness, and the ability to resolve student confusion by five course faculties through a survey. Additionally, another survey was administered to students in the course to assess their perception of ChatGPT's usefulness after interacting with the tool. The findings suggested that while ChatGPT demonstrated strengths in explaining introductory programming concepts accurately and completely, it showed weaknesses in resolving complex student confusion, adapting responses to individual needs, and providing tailored debugging assistance. This study highlighted key areas needing improvement and provided a basis to develop responsible integration strategies that harness AI to enrich rather than replace human instruction in technical courses. The results, based on the limited sample size and study duration, indicated that ChatGPT has potential as a supplemental teaching aid for core concepts, but also highlighted areas where human instruction may be particularly valuable, such as providing advanced support. Further research with larger samples and longer study periods is needed to assess the generalizability of these findings.</p> </abstract>