Analysis of Student Behaviour in Habitable Worlds Using Continuous Representation Visualization

Abstract

We introduce a novel approach to visualizing temporal clickstream behaviour in the context of a degree-satisfying online course, Habitable Worlds, offered through Arizona State University. The current practice for visualizing behaviour within a digital learning environment has been to generate plots based on hand engineered or coded features using domain knowledge. While this approach has been effective in relating behaviour to known phenomena, features crafted from domain knowledge are not likely well suited to make unfamiliar phenomena salient and thus can preclude discovery. We introduce a methodology for organically surfacing behavioural regularities from clickstream data, conducting an expert in-the-loop hyperparameter search, and identifying anticipated as well as newly discovered patterns of behaviour. While these visualization techniques have been used before in the broader machine learning community to better understand neural networks and relationships between word vectors, we apply them to online behavioural learner data and go a step further; exploring the impact of the parameters of the model on producing tangible, non-trivial observations of behaviour that are suggestive of pedagogical improvement to the course designers and instructors. The methodology introduced in this paper led to an improved understanding of passing and non-passing student behaviour in the course and is widely applicable to other datasets of clickstream activity where investigators and stakeholders wish to organically surface principal patterns of behaviour. NOTES FOR PRACTICE• Continuous representation visualization can produce a high-level view of emergent student behavior online without the need for defining features or tagging • Differential visualization of passing and non-passing student course behaviors can help identify deep and shallow learning strategies and provide instructors with essential information for modifying the curricula to discourage strategies associated with failure • Involving instructors in the tuning of the visualization and model parameters produces analyses with a desirable mixture of expected and unexpected, but explainable, patterns • Layering on additional data, such when students create a discussion post, further contextualizes insight into student learning strategies from visualizations

Moments of Confusion in Simulation-Based Learning Environments

“…Each "training" module introduced a new concept. These modules were mostly linear in structure, with occasional pathway adaptivity for remediation of learners with prior misconceptions (Pardos & Horodyskyj, 2019). Students could not proceed in a training module unless they correctly completed the current task.…”

Analyzing Student Strategies In Blended Courses Using Clickstream Data

“…Other lines of research rely on more traditional data mining techniques [17,34], or extraction of n-grams, i.e. sub-sequences of n consecutive actions [8,32,44,36]. The authors of [32] use a multi-step procedure to extract frequent n-grams that are subsequently used to identify different strategies in a collaborative interactive tabletop game.…”

“…The authors of [44] use topical n-gram models to automatically extract 'topics' in the form of frequent patterns from clickstreams. In [36], the authors train a skip-gram neural network to receive a structure preserving vector embedding of the types of clicks student can make. After standard dimensionality reduction, the researchers are able to provide a new kind of visualization of students' trajectories through the course.…”

“…In some sense, LDA is similar to the ideas proposed by [44] but requires less training data which renders it particularly useful for blended courses. In addition, we use an adapted form of the skip-gram model proposed by [36] in order to explore the context of student actions in our data. To the best of our knowledge, this is one of the first works to employ NLP methods for analysis of blended courses.…”

Student Testing and Monitoring System (Stms) Using Nlp

“…Bloom's taxonomy-based questions help in assessing the learning capability of the user. In Document processing tree tagger tool and stemming process is done to eliminate the human process [16]. Data characterization takes a rundown of keyword created by Data Processing and finds the Bloom's classification of those words, via looking through suitable activity verb in store which fits with the given keyword.…”