Wise Crowd Content Assessment and Educational Rubrics

Passonneau, Rebecca J.; Poddar, Ananya; Gite, Gaurav B.; Krivokapic, Alisa; Yang, Qian; Perin, Dolores

doi:10.1007/s40593-016-0128-6

Cited by 14 publications

(20 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To annotate the summary content, we use Pyramid annotation (Nenkova et al, 2007), a summary content annotation that has been shown to correlate with a main ideas rubric used in an educational intervention with community college students (Passonneau et al, 2018). As in that study, we collect five reference summaries written by more advanced students, referred to as a wise crowd.…”

Section: Annotation Of Essay Contentmentioning

confidence: 99%

“…To follow the principles of pyramid annotation applied to education (Passonneau et al, 2018), we collected wise crowd essays written by sophomores who took the academic skills course in the previous year and by the trained raters on the project (advanced undergraduates), to constitute 2 Currently in submission to another venue. five references per topic.…”

Section: Content Annotation Workflowmentioning

confidence: 99%

See 1 more Smart Citation

Rubric Reliability and Annotation of Content and Argument in Source-Based Argument Essays

Gao¹,

Driban²,

McManus³

et al. 2019

Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications

Self Cite

View full text Add to dashboard Cite

We present a unique dataset of student sourcebased argument essays to facilitate research on the relations between content, argumentation skills, and assessment. Two classroom writing assignments were given to college students in a STEM major, accompanied by a carefully designed rubric. The paper presents a reliability study of the rubric, showing it to be highly reliable, and initial annotation on content and argumentation annotation of the essays.

show abstract

Section: Annotation Of Essay Contentmentioning

confidence: 99%

Section: Content Annotation Workflowmentioning

confidence: 99%

Rubric Reliability and Annotation of Content and Argument in Source-Based Argument Essays

Gao¹,

Driban²,

McManus³

et al. 2019

Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications

Self Cite

View full text Add to dashboard Cite

show abstract

“…PyrEval differs from other automated pyramid tools in its focus on accurately isolating and weighting the distinct SCUs in the reference summaries. Three previous semi-automated pyramid tools used dynamic programming to score summaries, given a manual pyramid (Harnly et al, 2005;Passonneau et al, 2013Passonneau et al, , 2018. The first of these used unigram overlap to compare summaries to a pyramid.…”

Section: Related Workmentioning

confidence: 99%

“…A subsequent extension that used cosine similarity of latent vector representations of ngrams and SCUs, based on (Guo and Diab, 2012), had much better performance (Passonneau et al, 2013). This was extended further through use of a weighted set cover algorithm for scoring (Passonneau et al, 2018). PEAK was the first fully automated approach to construct a pyramid and score summaries (Yang et al, 2016).…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Automated Pyramid Summarization Evaluation

Gao

Sun

Passonneau³

2019

Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)

Self Cite

View full text Add to dashboard Cite

Pyramid evaluation was developed to assess the content of paragraph length summaries of source texts. A pyramid lists the distinct units of content found in several reference summaries, weights content units by how many reference summaries they occur in, and produces three scores based on the weighted content of new summaries. We present an automated method that is more efficient, more transparent, and more complete than previous automated pyramid methods. It is tested on a new dataset of student summaries, and historical NIST data from extractive summarizers.

show abstract

Text mining in education

Mello

Andre

Pinheiro

et al. 2019

WIREs Data Min & Knowl

138

View full text Add to dashboard Cite

The explosive growth of online education environments is generating a massive volume of data, specially in text format from forums, chats, social networks, assessments, essays, among others. It produces exciting challenges on how to mine text data in order to find useful knowledge for educational stakeholders. Despite the increasing number of educational applications of text mining published recently, we have not found any paper surveying them. In this line, this work presents a systematic overview of the current status of the Educational Text Mining field. Our final goal is to answer three main research questions: Which are the text mining techniques most used in educational environments? Which are the most used educational resources? And which are the main applications or educational goals? Finally, we outline the conclusions and the more interesting future trends. This article is categorized under: Application Areas > Education and Learning Ensemble Methods > Text Mining

show abstract

Wise Crowd Content Assessment and Educational Rubrics

Cited by 14 publications

References 53 publications

Rubric Reliability and Annotation of Content and Argument in Source-Based Argument Essays

Rubric Reliability and Annotation of Content and Argument in Source-Based Argument Essays

Automated Pyramid Summarization Evaluation

Text mining in education

Contact Info

Product

Resources

About