Aligning learning design and learning analytics through instructor involvement: a MOOC case study

Abstract

With the emergence of MOOCs, there is a growing interest in prediction research. Most existing predictive models do not consider the context for which they are intended, thus resulting in limited impact. Learning design (LD) can provide a contextual understanding for the design of predictive models in collaboration with the instructors, maximizing their potential for supporting learning. This paper presents the findings of a mixed-methods research that explored the potentials emerging from aligning LD and LA during the design of a predictive analytics solution and from involving the instructors in the design process. The context was a past MOOC, where the learner data and the instructors were accessible for posterior analysis and additional data collection. Through a close collaboration with the instructors, the details of the prediction task were identified, including the learning activity to focus on, the target variable to predict, and the practical constraints to consider. Later, two predictive models were built for the prediction task identified: LD-specific model, in which the features were based on the LD and pedagogical intentions, and a generic model, which was based on cumulative features, not informed by the LD. Although the LD-specific predictive model did not outperform the generic one, some features derived from the LD and pedagogical intentions were predictive. The quantity and the power of such features were associated with the degree to which the students acted as guided by the LD and pedagogical intentions. The leading instructor's opinion about the importance of the learning activities in the LD was compared with the results of the feature importance analysis. This comparison helped identify the parts of the LD that need improvement. That is, the results of the LA informed back the LD, where the instructor was a mediator. The implications for improving the LD are discussed.

Theory-based learning analytics to explore student engagement patterns in a peer review activity

“…But, more importantly, the results from this research work also suggest that certain learning activities, within peer review learning designs, are key for generating the data needed to obtain learning analytics indicators about significant regulatory processes. This implication reinforces the importance of aligning learning design and learning analytics [4]. An important implication for research is that the findings of this research suggest theory-based indicators to measure engagement not only in SRL but also in SSRL and Co-RL.…”

Generating actionable predictions regarding MOOC learners’ engagement in peer reviews

“…Similarly, number of discussion counts did not matter in the first course, whereas they were predictive in several models in the third course. We argue that this inconsistency is associated with the different learning design applied in each course [49]. That is, for example, the way discussion forums are used pedagogically and connected with the peer-reviewed activity may differ from one course to another, which may create a weaker/stronger association between student engagement in discussions and the peer reviews.…”

“…Additionally, when involved in the process, instructors may be inclined to use the actionable models to improve their teaching practice. Several research highlighted the benefits emerging from involving instructors in the loop of designing predictive models [35], [49].…”

Insights from Learning Analytics for Hands-On Cloud Computing Labs in AWS

“…Indeed, the work by Ruipérez-Valiente et al [20] introduced a learning analytics tool implemented for Open edX, called ANALYSE, which provides useful visualizations for teachers' feedback backed by pedagogical foundations. The work by Er et al [21] uses learning analytics to design a predictive analytics solution to involve instructors in the design process of an MOOC. Finally, the work by Tabaa and Medouri [22] focuses on creating a learning analytics system for massive open online courses based on big data techniques, such as Hadoop, in order to target "at-risk" students.…”