Students’ Experience of a ChatGPT Enabled Final Exam in a Non-Majors Chemistry Course

Abstract

In the field of education, ChatGPT has become a topic of debate for its usefulness as a learning tool. This article focuses on non-science majors' (n = 29) perceptions of a ChatGPT enabled final exam, where, prior to the exam, students wrote papers on science and sustainability and, during the final exam, students were asked to compare their paper to one produced on the same topic by ChatGPT. Thus, the underlying chemistry, its broader impacts, and connection to sustainability and writing styles were compared. Students' perceptions were analyzed through a developed coding framework that enabled the visualization of emerging themes. The most common themes revealed that students believed the ChatGPT essay did not read as "human-like", used more intricate words, and often did not include enough science to support its arguments. Students also noted that their essays provided more chemistry details and were easier to read as they focused on connecting chemistry concepts to their essay topic as well as sustainable policies and practices. Students were impressed, however, by ChatGPT's ability to discuss various sustainability solutions, policies, and practices. The final exam inspired self-reflection for the students to improve not only their writing but also their analysis of sustainability responses. Overall, students rated the comparative activity as a final exam to be favorable and remarked on the importance of analyzing AI generated work for the future of learning.

ChatGPT Writing Assistance and Evaluation Assignments Across the Chemistry Curriculum

“…Sometimes it added adjectives, and it tended to break up long and convoluted sentences. These changes are consistent with findings reported by other researchers. , …”

“…One student noted getting similar results from using Grammarly. However, not all students were impressedone noted that although it can sometimes be helpful, “it really just changes a couple of words and broke up the paragraphs.” These findings are consistent with ChatGPT’s revisions of Biochem students’ work, and with previous reports. ,, …”

“…The most common content suggestion was that the abstract should include specific data and conclusions tied to the lab. These observed weaknesses are to be expected of AI-generated content and agree with previous reports. ,, …”

“…The ACS Approval Program guidelines expect that departments will produce graduates who write “clearly and concisely, in a scientifically appropriate style.” Instructors use many different strategies to teach writing in our discipline, including peer review, , specifications grading, and stepwise instruction in writing different lab report sections . Generative AI can be used by instructors to rapidly generate products for students to critique and imitate; several publications in this journal have described instructors using outputs of generative AI as a starting point for essays, class discussions, exams, and critical thinking exercises . Evaluation is the second-highest level skill of Bloom’s taxonomy, immediately below creation.…”

“…A large study found that argumentative essays generated by ChatGPT were more formulaic but higher in quality and more “scientific” in language than those written by high schoolers . Similarly, students asked to compare an essay they had written about sustainability to one written on the same subject by ChatGPT found it used more formal and less variable structure, and “more intricate words.” Another study found that when students were given open-ended permission to use ChatGPT to write an essay, there was little difference in student grades and the length of time it took them to write it . In a study where students used ChatGPT to help them write an essay about a General Chemistry topic, students became more confident in their own critical thinking skills, but they did not always evaluate ChatGPT outputs correctly, and they did not find ChatGPT useful for generating creative or truly new information …”