BEETLE II: Deep Natural Language Understanding and Automatic Feedback Generation for Intelligent Tutoring in Basic Electricity and Electronics

Abstract

Within STEM domains, physics is considered to be one of the most difficult topics to master, in part because many of the underlying principles are counter-intuitive. Effective teaching methods rely on engaging the student in active experimentation and encouraging deep reasoning, often through the use of selfexplanation. Supporting such instructional approaches poses a challenge for developers of Intelligent Tutoring Systems. We describe a system that addresses this challenge by teaching conceptual knowledge about basic electronics and electricity through guided experimentation with a circuit simulator and reflective dialogue to encourage effective self-explanation. The Basic Electricity and Electronics Tutorial Learning Environment (BEETLE II) advances the state of the art in dynamic adaptive feedback generation and natural language processing (NLP) by extending symbolic NLP techniques to support unrestricted student natural language input in the context of a dynamically changing simulation environment in a moderately complex domain. This allows contextually-appropriate feedback to be generated "on the fly" without requiring curriculum designers to anticipate possible student answers and manually author multiple feedback messages. We present the results of a system evaluation. Our curriculum is highly effective, achieving effect sizes of 1.72 when comparing pre-to post-test learning gains from our system to those of a no-training control group. However, we are unable to demonstrate that dynamically generated feedback is superior to a non-NLP feedback condition. Evaluation of interpretation quality demonstrates its link with instructional effectiveness, and provides directions for future research and development.

Beetle-Grow

“…Instead, the system simply states the expected answer and leaves the student to decide if their own answer was a close enough match. This behavior corresponds to the tell condition reported in [5], which demonstrated that asking students for an explanation and then immediately giving away the correct answer was highly effective, despite the obvious potential to 'game the system' by typing minimal or nonsense answers. This approach also avoids the negative effect on student satisfaction and engagement seen in Beetle II evaluations due to the limitations of the natural language understanding and feedback selection modules used there.…”

“…Beetle-Grow builds on the success of Beetle II, a natural-language ITS which has been shown to be highly effective in teaching basic electricity and electronics, demonstrating effect sizes of d=1.72 [5]. Beetle II combines natural language dialogue, guided experimentation, and adaptive feedback to implement the conceptual change [4] tutoring approach.…”

Sistema Inteligente de Tutoría para entrenamiento procedimental apoyado en un diálogo adaptado al contexto

“…Beetle II (Dzikovska et al 2014), es un sistema desarrollado para enseñar la construcción de un circuito electrónico. Permite entradas en lenguaje natural sin restricciones y la aplicación de diferentes estrategias de diálogo para realizar una retroalimentación adaptativa dinámica.…”

Assistentes de Aprendizagem em Simuladores de Voo

“…Os assistentes são procedimentos baseados em passo a passo, de modo a orientar o aprendiz a concluir um passo para que o próximo seja apresentado. Além disso, existem projetos que abordam outros ambientes de simulação (diferente da prática de voo), como os ambientes descritos por Dzikovska et al (2014) e Zachary et al (1999), mas que possuem abordagens pedagógicas com foco na melhoria do desenvolvimento dos aprendizes e podem ser considerados na análise dos ambientes computacionais para assistência de aprendizagem em simuladores. A seguir, são descritos estes ambientes computacionais.…”

“…Segundo Dzikovska et al (2014), o Ambiente de Aprendizagem Tutorial de Eletricidade e Eletrônica Básica (BEETLE II) é um Sistema Tutorial Inteligente (ITS), focado no desenvolvimento de um currículo efetivo para os subtópicos de física, eletricidade e circuitos elétricos, no domínio da Ciência, Tecnologia, Engenharia e Matemática (STEM). As lições no BEETLE II aplicam uma abordagem instrucional, implementada com base na análise do comportamento de um tutor humano para o mesmo currículo, dentro do contexto de uma bancada de circuito simulada.…”