Juxtapeer

Abstract

Figure 1. The review interface presents two peer submissions side-by-side, and aligns corresponding submission parts so reviewers can identify similarities and differences. To scaffold feedback, a rubric guides the reviewer through a structured comparison.

Transforming Everyday Information into Practical Analytics with Crowdsourced Assessment Tasks

“…A review of 51 studies [21] identified the main domains that have leveraged crowdsourcing: to generate content, provide training experiences, exchange knowledge, and gather feedback from other experts and novices [10,12,20]. Researchers have developed systems to crowdsource grades, provide feedback on student work, and increase student knowledge attainment as they assess their own or peers' work [6,10,13,33]. However, most of these cases occur in self-contained courses where students assess each other, often in higher education settings, and where students can be expected to have some domain knowledge.…”

“…A key issue in educational crowdsourcing is how to increase the overall quality of the work [6,10,14,26]. Educational crowdsourcing tasks often require specialized knowledge (e.g., teaching and learning knowledge, or skills to assess a given assignment) that cannot be solved effectively by unskilled crowdworkers [24].…”

“…Educational crowdsourcing tasks often require specialized knowledge (e.g., teaching and learning knowledge, or skills to assess a given assignment) that cannot be solved effectively by unskilled crowdworkers [24]. Researchers have sought to overcome this knowledge barrier by recruiting workers from specific pools with relevant knowledge or experience, such as submitters of the same homework assignment, who can then grade others [6,10] or the potential end-users of a system [13]. Another strategy is to screen workers [9].…”

“…Crowdsourcing can be seen as outsourcing tasks to humans that artificial intelligence cannot yet accomplish [28]. Of particular interest to our work are studies that employ crowdsourced assessment to reduce the workload for educators and provide diverse perspectives to students about their learning progress [3,6]. However, additional design studies are needed to explore how to create crowdsourcing systems that can process the diverse forms of data found in K-12 contexts.…”