Discovering Insights in Learning Analytics Through a Mixed-Methods Framework: Application to Computer Programming Education

Abstract

Aim/Purpose: This article proposes a framework based on a sequential explanatory mixed-methods design in the learning analytics domain to enhance the models used to support the success of the learning process and the learner. The framework consists of three main phases: (1) quantitative data analysis; (2) qualitative data analysis; and (3) integration and discussion of results. Furthermore, we illustrated the application of this framework by examining the relationships between learning process metrics and academic performance in the subject of Computer Programming coupled with content analysis of the responses to a students’ perception questionnaire of their learning experiences in this subject. Background: There is a prevalence of quantitative research designs in learning analytics, which limits the understanding of students’ learning processes. This is due to the abundance and ease of collection of quantitative data in virtual environments and learning management systems compared to qualitative data. Methodology: This study uses a mixed-methods, non-experimental, research design. The quantitative phase of the framework aims to analyze the data to identify behaviors, trends, and relationships between measures using correlation or regression analysis. On the other hand, the qualitative phase of the framework focuses on conducting a content analysis of the qualitative data. This framework was applied to historical quantitative and qualitative data from students’ use of an automated feedback and evaluation platform for programming exercises in a programming course at the National University of Colombia during 2019 and 2020. The research question of this study is: How can mixed-methods research applied to learning analytics generate a better understanding of the relationships between the variables generated throughout the learning process and the academic performance of students in the subject of Computer Programming? Contribution: The main contribution of this work is the proposal of a mixed-methods learning analytics framework applicable to computer programming courses, which allows for complementing, corroborating, or refuting quantitatively evidenced results with qualitative data and generating hypotheses about possible causes or explanations for student behavior. In addition, the results provide a better understanding of the learning processes in the Computer Programming course at the National University of Colombia. Findings: A framework based on sequential explanatory mixed-methods design in the field of learning analytics has been proposed to improve the models used to support the success of the learning process and the learner. The answer to the research question posed corresponds to that the mixed methods effectively complement quantitative and qualitative data. From the analysis of the data of the application of the framework, it appears that the qualitative data, representing the perceptions of the students, generally supported and extended the quantitative data. The consistency between the two phases allowed us to generate hypotheses about the possible causes of student behavior and provide a better understanding of the learning processes in the course. Recommendations for Practitioners: We suggest implementing the proposed mixed-methods learning analytics framework in various educational contexts and populations. By doing so, practitioners can gather more diverse data and insights, which can lead to a better understanding of learning processes in different settings and with different groups of learners. Recommendation for Researchers: Researchers can use the proposed approach in their learning analytics projects, usually based exclusively on quantitative data analysis, to complement their results, find explanations for their students’ behaviors, and understand learning processes in depth thanks to the information provided by the complementary analysis of qualitative data. Impact on Society: The prevalence of exclusively quantitative research designs in learning analytics can limit our understanding of students’ learning processes. Instead, the mixed-methods approach we propose suggests a more comprehensive approach to learning analytics that includes qualitative data, which can provide deeper insight into students’ learning experiences and processes. Ultimately, this can lead to more effective interventions and improvements in teaching and learning practices. Future Research: Potential lines of research to continue the work on mixed-method learning analytics methodology include the following: first, implementing the framework on a different population sample, such as students from other universities or other knowledge areas; second, using techniques to correct unbalanced data sets in learning analytics studies; third, analyzing student interactions with the automated grading platform and their academic activities in relation with their activity grades; last, using the findings to design interventions that positively impact academic performance and evaluating the impact statistically through experimental study designs. In the context of introductory programming education, AI/large language models have the potential to revolutionize teaching by enhancing the learning experience, providing personalized support, and enabling more efficient assessment and feedback mechanisms. Future research in this area is to implement the proposed framework on data from an introductory programming course using these models.