Design and Evaluation of a Natural Language Tutor for Force and Motion

Abstract

Abstract:We report on the design and pilot evaluation of a simple natural language computer tutor that targets student difficulties with the concepts of force and motion. The tutor prompts students to respond in free-response natural language to questions that address the relationships between the directions of net force, velocity, and acceleration. To examine the effectiveness of the natural language format, we compared student performance on a previously validated force and motion assessment after tutoring via natural language and multiple choice formats. Natural language training with feedback, multiple choice training with feedback, and natural language training without feedback formats resulted in effect sizes of d = 0.60 (p = 0.07), d = 0.46 (p = 0.13), and d = 0.09 (p = 0.97) respectively versus a no-training control. In addition, a median split on course grades showed no significant aptitude-treatment interaction across training conditions. However, accounting for time spent on training, the multiple choice training was significantly more efficient. For the natural language format, an analysis of performance (62% identification of an initial student response), false positives, and typical student answer patterns suggest room for improvement and subsequent study.