Is critical thinking happening? Testing content analysis schemes applied to MOOC discussion forums

Abstract

Learners' progress within computer-supported collaborative learning environments is typically measured via analysis and interpretation of quantitative web interaction measures. However, the usefulness of these "proxies for learning" is questioned as they do not necessarily reflect critical thinking-an essential component of collaborative learning. Research indicates that pedagogical content analysis schemes have value in measuring critical discourse in small scale, formal, online learning environments, but research using these methods on high volume, informal, Massive Open Online Course (MOOC) forums is less common. The challenge in this setting is to develop valid and reliable indicators that operate successfully at scale. In this study, we test two established coding schemes used for the pedagogical content analysis of online discussions in a large-scale review of MOOC comment data. Pedagogical Scores are derived from manual ratings applied to comments by raters and correlated with automatically derived linguistic and interaction indicators. Results show that the content analysis methods are reliable, and are very strongly correlated with each other, suggesting that their specific format is not significant in this setting. In addition, the methods are strongly associated with the relevant linguistic indicators of higher levels of learning and have weaker correlations with other linguistic and interaction metrics. This suggests promise for further research using Machine Learning techniques, with the goal of providing realistic feedback to instructors, learners, and learning designers.

The Use of MOOCS in Online Engineering Education

“…However, the usefulness of these learning proxies has been questioned as they do not necessarily reflect critical thinking -an important component of collaborative learning. Research shows that pedagogical content analysis schemes are valuable for measuring critical discourse in small formal online learning environments [49]. The integration of MOOCs into a traditional curriculum requires fine-tuning of learning outcomes, assessment methods, and student workload.…”

A Systematic Review of the Effects of Automatic Scoring and Automatic Feedback in Educational Settings

“…Students can solve problems without learning the underlying concepts. Using the feedback as a trial-and-error exercise or accessing answers using the internet can cause a very detrimental effect on learning [119]. Similarly, where solutions are not easily defined or grouped, subjects will have a more challenging time implementing these technologies as they are less developed in those areas.…”

An Observation Protocol for Scaffolding Community of Inquiry and Its Exemplary Practices in Language MOOCs

“…The first is a shift from students' self-reported perspectives to raters' observational ones. The second is an expansion from items that focused on CoIs in discussion forums only, to include ones that focus on CoIs in all-around online course delivery (Caskurlu et al, 2021;Fiock, 2020), EFL teaching (Mo & Lee, 2017;Yang et al, 2019), and teaching in MOOC contexts (Cohen & Holstein, 2018;Goshtasbpour et al, 2020;Kaul et al, 2018;O'Riordan et al, 2020;Zou et al, 2021). The third is the addition of concrete examples from EFL that correspond to different dimensions within each CoI presence.…”