Learnersourcing Quality Assessment of Explanations for Peer Instruction

IEEE Trans. Learning Technol.

Rahimi

et al. 2023

Engaging students in creating learning resources has demonstrated pedagogical benefits. However, to effectively utilise a repository of student-generated content (SGC), a selection process is needed to separate high-from low-quality resources as some of the resources created by students can be ineffective, inappropriate, or incorrect. A common and scalable approach is to use a peer review process where students are asked to assess the quality of resources authored by their peers. Given that judgements of students, as experts-in-training, cannot wholly be relied upon, a redundancy-based method is widely employed where the same assessment task is given to multiple students. However, this approach introduces a new challenge, referred to as the consensus problem: How can we assign a final quality to a resource given ratings by multiple students? To address this challenge, we investigate the predictive performance of 18 inference models across five well-established categories of consensus approaches for inferring the quality of SGC at scale. The analysis is based on the engagement of 2,141 undergraduate students across five courses in creating 12,803 resources and 77,297 peer reviews. Results indicate that the quality of reviews is quite diverse, and students tend to overrate. Consequently, simple statistics such as mean and median fail to identify poor quality resources. Findings further suggest that incorporating advanced probabilistic and text analysis methods to infer the reviewers' reliability and reviews' quality improves performance; however, there is still an evident need for instructor oversight and training of students to write compelling and reliable reviews.

Section: A Learnersourcingmentioning

confidence: 99%

Assessing the Quality of Student-Generated Content at Scale: A Comparative Analysis of Peer-Review Models

Darvishi

IEEE Trans. Learning Technol.

Rahimi

et al. 2023

“…Their findings provide strong evidence that students can create high‐quality learning resources that meet rigorous judgemental and statistical criteria (Bates et al, 2014; Galloway & Burns, 2015) and that engaging in learnersourcing can enhance student learning (Khosravi et al, 2019). Other examples focusing on creating content include creating repositories of learning content such as multiple‐choice questions (Khosravi et al, 2019), knowledge components (Moore et al, 2020), explanations for programming misconceptions (Guo et al, 2020), solutions to open‐ended questions (Wang et al, 2019; Williams et al, 2016), explanations for peer instruction (Bhatnagar et al, 2020), summaries of steps in how‐to videos (Weir et al, 2015) and personalized hints (Glassman et al, 2016). These learning repositories can be utilized to develop adaptive and intelligent systems to support the personalisation of education (Khosravi et al, 2019; Williams et al, 2016).…”

Section: Background and Related Workmentioning

confidence: 99%

Analytics of learning tactics and strategies in an online learnersourcing environment

Lahza

Computer Assisted Learning

Demartini

2022

Background The use of crowdsourcing in a pedagogically supported form to partner with learners in developing novel content is emerging as a viable approach for engaging students in higher‐order learning at scale. However, how students behave in this form of crowdsourcing, referred to as learnersourcing, is still insufficiently explored. Objectives To contribute to filling this gap, this study explores how students engage with learnersourcing tasks across a range of course and assessment designs. Methods We conducted an exploratory study on trace data of 1279 students across three courses, originating from the use of a learnersourcing environment under different assessment designs. We employed a new methodology from the learning analytics (LA) field that aims to represent students' behaviour through two theoretically‐derived latent constructs: learning tactics and the learning strategies built upon them. Results The study's results demonstrate students use different tactics and strategies, highlight the association of learnersourcing contexts with the identified learning tactics and strategies, indicate a significant association between the strategies and performance and contribute to the employed method's generalisability by applying it to a new context. Implications This study provides an example of how learning analytics methods can be employed towards the development of effective learnersourcing systems and, more broadly, technological educational solutions that support learner‐centred and data‐driven learning at scale. Findings should inform best practices for integrating learnersourcing activities into course design and shed light on the relevance of tactics and strategies to support teachers in making informed pedagogical decisions.

“…Previous studies have demonstrated examples of how learnersourcing can be used towards facilitating a learner-centred approach based on the aforementioned three points. Examples of employing learnersourcing to engage learners in higher-order learning tasks includes enabling students to create and evaluate knowledge components [47], multiple-choice questions [17,34], personalised hints [25], summaries of steps in how-to videos [64], explanations for peer instruction [7], solutions to open-ended questions [63], explanations for programming misconceptions [26], and comparing and contrasting pairs of similar learning artefacts [19]. Learnersourcing is also commonly used to support the delivery of feedback using peer assessment and grading systems (e.g., [52,67]), which have been demonstrated to help learners develop evaluative judgement, the capacity to make accurate evaluations about the quality of their work and that of others [33,56].…”

Section: Introductionmentioning

confidence: 99%

Charting the Design and Analytics Agenda of Learnersourcing Systems

LAK21: 11th International Learning Analytics and Knowledge Conference

Demartini

Sadiq

et al. 2021

Learnersourcing is emerging as a viable learner-centred and pedagogically justified approach for harnessing the creativity and evaluation power of learners as experts-in-training. Despite the increasing adoption of learnersourcing in higher education, understanding students' behaviour while engaged in learnersourcing and best practices for the design and development of learnersourcing systems are still largely under-researched. This paper offers data-driven reflections and lessons learned from the development and deployment of a learnersourcing adaptive educational system called RiPPLE, which to date, has been used in more than 50-course offerings with over 12,000 students. Our reflections are categorised into examples and best practices on (1) assessing the quality of students' contributions using accurate, explainable and fair approaches to data analysis, (2) incentivising students to develop high-quality contributions and (3) empowering instructors with actionable and explainable insights to guide student learning. We discuss the implications of these findings and how they may contribute to the growing literature on the development of effective learnersourcing systems and more broadly technological educational solutions that support learner-centred learning at scale. CCS CONCEPTS• Applied computing → Computer-assisted instruction; Interactive learning environments; • Human-centered computing → Collaborative and social computing systems and tools.