Crowd-scale interactive formal reasoning and analytics

Fast, Ethan; Lee, Colleen; Aiken, Alex; Bernstein, Michael S.; Koller, Daphne; Smith, Eric

doi:10.1145/2501988.2502028

Cited by 12 publications

(7 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…But more interestingly, these ideas may also apply to indexing formal logic proofs [4] and equations [13], or even natural language mathematical proofs, which can be useful either for researchers or educators. Looking even further, it is tantalizing to speculate about the role that effective search engines for student content might play in the MOOC ecosystem beyond STEM (science, technology, engineering and mathematics) courses.…”

Section: Discussionmentioning

confidence: 99%

Codewebs

Nguyễn

Piech

Huang

et al. 2014

Proceedings of the 23rd International Conference on World Wide Web

View full text Add to dashboard Cite

Massive open online courses (MOOCs), one of the latest internet revolutions have engendered hope that constant iterative improvement and economies of scale may cure the "cost disease" of higher education. While scalable in many ways, providing feedback for homework submissions (particularly open-ended ones) remains a challenge in the online classroom. In courses where the student-teacher ratio can be ten thousand to one or worse, it is impossible for instructors to personally give feedback to students or to understand the multitude of student approaches and pitfalls. Organizing and making sense of massive collections of homework solutions is thus a critical web problem. Despite the challenges, the dense solution space sampling in highly structured homeworks for some MOOCs suggests an elegant solution to providing quality feedback to students on a massive scale.We outline a method for decomposing online homework submissions into a vocabulary of "code phrases", and based on this vocabulary, we architect a queryable index that allows for fast searches into the massive dataset of student homework submissions. To demonstrate the utility of our homework search engine we index over a million code submissions from users worldwide in Stanford's Machine Learning MOOC and (a) semi-automatically learn shared structure amongst homework submissions and (b) generate specific feedback for student mistakes.Codewebs is a tool that leverages the redundancy of densely sampled, highly structured homeworks in order to forcemultiply teacher effort. Giving articulate, instant feedback is a crucial component of the online learning process and thus by building a homework search engine we hope to take a step towards higher quality free education.

show abstract

Section: Discussionmentioning

confidence: 99%

Codewebs

Nguyễn

Piech

Huang

et al. 2014

Proceedings of the 23rd International Conference on World Wide Web

View full text Add to dashboard Cite

show abstract

“…In contrast, most prior work has capitalized on scale only after all assignments are submitted. For instance, DeduceIt uses the semantic similarity between student solutions to provide automatic hinting and to check solution correctness [14], while other systems cluster solutions to help teachers provide feedback quickly [6].…”

Section: Related Workmentioning

confidence: 99%

PeerStudio

Kulkarni

Bernstein

Klemmer³

2015

Proceedings of the Second (2015) ACM Conference on Learning @ Scale

Self Cite

142

View full text Add to dashboard Cite

Rapid feedback is a core component of mastery learning, but feedback on open-ended work requires days or weeks in most classes today. This paper introduces PeerStudio, an assessment platform that leverages the large number of students' peers in online classes to enable rapid feedback on in-progress work. Students submit their draft, give rubricbased feedback on two peers' drafts, and then receive peer feedback. Students can integrate the feedback and repeat this process as often as they desire. In MOOC deployments, the median student received feedback in just twenty minutes. Rapid feedback on in-progress work improves course outcomes: in a controlled experiment, students' final grades improved when feedback was delivered quickly, but not if delayed by 24 hours. More than 3,600 students have used PeerStudio in eight classes, both massive and inperson. This research demonstrates how large classes can leverage their scale to encourage mastery through rapid feedback and revision.

show abstract

“…The large scale of online classes enables machines to effectively improve the educational experience [9]. By lessening grading burden, machines can focus peers on providing more detailed feedback.…”

Section: Future Work and Conclusionmentioning

confidence: 99%

Scaling short-answer grading by combining peer assessment with algorithmic scoring

Kulkarni

Socher

Bernstein

et al. 2014

Proceedings of the First ACM Conference on Learning @ Scale Conference

Self Cite

View full text Add to dashboard Cite

Peer assessment helps students reflect and exposes them to different ideas. It scales assessment and allows large online classes to use open-ended assignments. However, it requires students to spend significant time grading. How can we lower this grading burden while maintaining quality? This paper integrates peer and machine grading to preserve the robustness of peer assessment and lower grading burden. In the identify-verify pattern, a grading algorithm first predicts a student grade and estimates confidence, which is used to estimate the number of peer raters required. Peers then identify key features of the answer using a rubric. Finally, other peers verify whether these feature labels were accurately applied. This pattern adjusts the number of peers that evaluate an answer based on algorithmic confidence and peer agreement. We evaluated this pattern with 1370 students in a large, online design class. With only 54% of the student grading time, the identify-verify pattern yields 80-90% of the accuracy obtained by taking the median of three peer scores, and provides more detailed feedback. A second experiment found that verification dramatically improves accuracy with more raters, with a 20% gain over the peer-median with four raters. However, verification also leads to lower initial trust in the grading system. The identify-verify pattern provides an example of how peer work and machine learning can combine to improve the learning experience.

show abstract

Crowd-scale interactive formal reasoning and analytics

Cited by 12 publications

References 32 publications

Codewebs

Codewebs

PeerStudio

Scaling short-answer grading by combining peer assessment with algorithmic scoring

Contact Info

Product

Resources

About