Evaluating the Quality of Learning Resources: A Learnersourcing Approach

Abdi, Solmaz; Khosravi, Hassan; Sadiq, Shazia; Demartini, Gianluca

doi:10.1109/tlt.2021.3058644

Cited by 29 publications

(16 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Prior studies assessing such qualitative measures, however, have involved significant manual effort by experts. In learnersourcing contexts, aggregating student ratings to assess the quality of questions is scalable and agrees well with expert ratings of quality (Abdi et al 2021;Darvishi, Khosravi, and Sadiq 2021;McQueen et al 2014). In our own work, we use averaged student ratings as groundtruth labels when training our AQQR models.…”

Section: Related Workmentioning

confidence: 74%

“…A more scalable solution is to have students review and evaluate the content themselves (Darvishi, Khosravi, and Sadiq 2021). Prior research has shown that students can make similar quality judgments to experts, especially when the assessments provided by multiple students are aggregated (Abdi et al 2021). However, a sufficient number of students must view and evaluate each artefact before a valid assessment can be produced, which is inefficient.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

DeepQR: Neural-Based Quality Ratings for Learnersourced Multiple-Choice Questions

Qiming

et al. 2022

AAAI

View full text Add to dashboard Cite

Automated question quality rating (AQQR) aims to evaluate question quality through computational means, thereby addressing emerging challenges in online learnersourced question repositories. Existing methods for AQQR rely solely on explicitly-defined criteria such as readability and word count, while not fully utilising the power of state-of-the-art deep-learning techniques. We propose DeepQR, a novel neural-network model for AQQR that is trained using multiple-choice-question (MCQ) datasets collected from PeerWise, a widely-used learnersourcing platform. Along with designing DeepQR, we investigate models based on explicitly-defined features, or semantic features, or both. We also introduce a self-attention mechanism to capture semantic correlations between MCQ components, and a contrastive-learning approach to acquire question representations using quality ratings. Extensive experiments on datasets collected from eight university-level courses illustrate that DeepQR has superior performance over six comparative models.

show abstract

Section: Related Workmentioning

confidence: 74%

Section: Introductionmentioning

confidence: 99%

DeepQR: Neural-Based Quality Ratings for Learnersourced Multiple-Choice Questions

Qiming

et al. 2022

AAAI

View full text Add to dashboard Cite

show abstract

“…As students are developing their expertise, it is likely that some of the learning resources created are ineffective, inappropriate or incorrect. Hence, RiPPLE utilises an evaluation process that again partners with students as moderators to judge the quality of their peers' work (Abdi et al, 2021). Figure 1 illustrates the evaluation interfaces used by the platform.…”

Section: Overview Of Ripple Platformmentioning

confidence: 99%

Repositioning students as co-creators of curriculum for online learning resources

McDonald

McGowan

Naylor

et al. 2021

AJET

Self Cite

View full text Add to dashboard Cite

Amid increasing calls for universities to transition to online learning, there is a need to explore how platforms and technology can provide positive student experiences and support learning. In this paper, we discuss the implementation of an online peer learning and recommender platform in a large, multi-campus, first-year health subject (n = 2095). The Recommendation in Personalised Peer Learning Environments (RiPPLE) platform supports student’s co-creation of learning resources and allows for students to provide feedback and rate their peers’ submissions. Our results indicated that both student engagement and academic performance were positively impacted for users by the introduction of the RiPPLE platform, but that academic preparedness, in the form of students’ ATAR scores, strongly influenced their engagement and the benefits received. Implications for practice or policy: We explored if students were willing to co-create learning resources online. Our study piloted an online platform known as Recommendation in Personalised Peer Learning Environments (RiPPLE). Critical analysis provides insights into fostering online engagement and peer learning. We further offer recommendations for future practice on how to embed online student co-creation of curriculum.

show abstract

“…Here we provide a brief overview of the study. A fuller account is available in Abdi, Khosravi, Sadiq, and Demartini (2021).…”

Section: Case Study 1: An Observational Study To Investigate Students' Evaluative Judgmentmentioning

confidence: 99%

Bridging the Gap Between Theory and Empirical Research in Evaluative Judgment

Khosravi

Gyamfi

Hanna

et al. 2021

JLA

Self Cite

View full text Add to dashboard Cite

The value of students developing the capacity to accurately judge the quality of their work and that of others has been widely studied and recognized in higher education literature. To date, much of the research and commentary on evaluative judgment has been theoretical and speculative in nature, focusing on perceived benefits and proposing strategies seen to hold the potential to foster evaluative judgment. The efficacy of the strategies remains largely untested. The rise of educational tools and technologies that generate data on learning activities at an unprecedented scale, alongside insights from the learning sciences and learning analytics communities, provides new opportunities for fostering and supporting empirical research on evaluative judgment. Accordingly, this paper offers a conceptual framework and an instantiation of that framework in the form of an educational tool called RiPPLE for data-driven approaches to investigating the enhancement of evaluative judgment. Two case studies, demonstrating how RiPPLE can foster and support empirical research on evaluative judgment, are presented.

show abstract

Evaluating the Quality of Learning Resources: A Learnersourcing Approach

Cited by 29 publications

References 50 publications

DeepQR: Neural-Based Quality Ratings for Learnersourced Multiple-Choice Questions

DeepQR: Neural-Based Quality Ratings for Learnersourced Multiple-Choice Questions

Repositioning students as co-creators of curriculum for online learning resources

Bridging the Gap Between Theory and Empirical Research in Evaluative Judgment

Contact Info

Product

Resources

About