Impersonating Chatbots in a Code Review Exercise to Teach Software Engineering Best Practices

Abstract

Over the past decade, the use of chatbots for educational purposes has gained considerable traction. A similar trend has been observed in social coding platforms, where automated agents support software developers with tasks such as performing code reviews. While incorporating code reviews and social coding platforms into software engineering education has been found to be beneficial, challenges such as steep learning curves and privacy considerations are barriers to their adoption. Furthermore, no study has addressed the role chatbots play in supporting code reviews as a pedagogical tool. To help address this gap, we developed an online learning application that simulates the code review features available on social coding platforms and allows instructors to interact with students using chatbot identities. We then embedded this application within a lesson on software engineering best practices and conducted a controlled in-class experiment. This experiment examined the effect that explaining content via chatbot identities had on three aspects: (i) students' perceived usability of the lesson, (ii) their engagement with the code review process, and (iii) their learning gains. While our findings show that it is feasible to simulate the code review process within an online learning platform and achieve good usability, our quantitative analysis did not yield significant differences across treatment conditions for any of the aspects considered. Nevertheless, our qualitative results suggest that students expect explicit feedback when performing this type of exercise and could thus benefit from automated replies provided by an interactive chatbot. We propose to build on our current findings to further explore this line of research in future work.

A Blueprint for Integrating Task-Oriented Conversational Agents in Education

“…As chatbots have been particularly useful in computer-assisted language learning (CALL) [2,7,31], it would be particularly pertinent to implement chatbots aimed at supporting tasks in CALL (e.g., providing conversational practice, checking grammar, interfacing dictionaries). These implementations will be validated following experimental designs used in previous exploratory studies [11,12,14] comprising controlled experiments measuring learning gains, engagement, and usability using both standard (e.g., the User Experience Questionnaire [24] and the System Usability Scale [3]) and bespoke (e.g., scores calculated using the results of pre/post-tests) instruments. Third, the need to support instructors in the configuration and deployment of chatbots built with our blueprint was not addressed.…”

Harnessing Rule-Based Chatbots to Support Teaching Python Programming Best Practices

“…However, very few studies have focused on the use of chatbots to support the code review process in formal education settings. We build on a previous Wizard of Oz [6] experiment [10] and a pilot study [8] to explore the impact that rule-based chatbots supporting code review exercises could have on the learning experience. In the following section, we present our guiding research question and the methodology we followed for our evaluation.…”

ChatGPT: Challenges and Benefits in Software Programming for Higher Education

“…In the realm of software education, chatbots have emerged as a valuable asset in teaching coding programs [26,27] or disseminating best practices in software engineering [28]. Students engage with ChatGPT, receiving immediate responses, thereby enabling self-paced learning and fostering a deeper grasp of intricate coding concepts.…”