Comparing holistic and analytic scoring methods: issues of validity and reliability

Harsch, Claudia; Martin, Guido

doi:10.1080/0969594x.2012.742422

Cited by 43 publications

(52 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…After training, annotators reached an inter-annotator agreement between 52 and 100% ModAgree (cf. Harsch and Martin (2012, p. 228-250) and Harsch and Martin (2013)) for the different descriptor variables. Percentage ModAgree is computed by measuring per essay and variable what percentage of all ratings assigned by the different annotators for this essay agrees with the mode, i.e.…”

Section: A More Challenging Essay Datasetmentioning

confidence: 96%

Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications

Tetreault

Burstein

Leacock³

et al. 2017

View full text Add to dashboard Cite

Since the first workshop in 1997, BEA has become the leading venue for sharing and publishing innovative work that uses NLP to develop educational applications. The consistent interest and growth of the workshop has clear ties to challenges in education. The research presented at the workshop highlights advances in the technology and the maturity of the field of NLP in education. The capabilities serve as a response to educational challenges and are poised to support the needs of a variety of stakeholders, including educators, learners, parents, and administrators.NLP capabilities now support an array of learning domains, including writing, speaking, reading, and mathematics. In the writing and speech domains, automated writing evaluation (AWE) and speech assessment applications, respectively, are commercially deployed in high-stakes assessment and instructional settings, including Massive Open Online Courses (MOOCs). We also see widelyused commercial applications for plagiarism detection and peer review and explosive growth of mobile applications for game-based applications for instruction and assessment. The current educational and assessment landscape continues to foster a strong interest and high demand that pushes the state of the art in AWE capabilities to expand the analysis of written responses to writing genres other than those traditionally found in standardized assessments, especially writing tasks requiring use of sources and argumentative discourse.Steady growth in the development of NLP-based applications for education has prompted an increased number of workshops that typically focus on a single subfield. In BEA, we make an effort to have papers from many subfields, for example, tools for automated scoring, automated test-item generation, curriculum development, evaluation of text, dialogue, evaluation of genres beyond essays, feedback studies, and grammatical error correction.This year we received a record 62 submissions, and accepted 9 papers as oral presentations and 25 as poster presentation and/or demos, for an overall acceptance rate of 55 percent. Each paper was reviewed by three members of the Program Committee who were believed to be most appropriate for each paper. We continue to have a very strong policy to deal with conflicts of interest. First, we made a concerted effort to not assign papers to reviewers to evaluate if the paper had an author from their institution. Second, with respect to the organizing committee, authors of papers for which there was a conflict of interest recused themselves from the discussions.While the field is growing, we do recognize that there is a core group of institutions and researchers who work in this area. With a higher acceptance rate, we were able to include papers from a wider variety of topics and institutions. The papers accepted were selected on the basis of several factors, including the relevance to a core educational problem space, the novelty of the approach or domain, and the strength of the research. The accepted papers were highly diverse -...

show abstract

Section: A More Challenging Essay Datasetmentioning

confidence: 96%

Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications

Tetreault

Burstein

Leacock³

et al. 2017

View full text Add to dashboard Cite

show abstract

“…This scoring form is exactly the same for all assessments, independent of their content, and combines holistic with analytic scoring (e.g. Harsch and Martin 2012). It starts with a holistic judgement: 'is the student competent in performing the professional whole task?'…”

Section: Professional Expertisementioning

confidence: 99%

“…Theoretical rationales underlying the new assessment are (1) the assessment assesses the integration of knowledge, skills and attitudes at the does-level of Millers pyramid, which (2) requires an authentic assessment of real-life, complex practical performance at the level of pre-VET students in which (3) generic competencies should be given a concrete meaning. Additionally, (4) it should combine at least two assessment methods of which one focuses on Millers' does/show how level and one on Millers' knows/knows how level; (5) various actions should be put in place in the assessment design to help assessors to evaluate students' competencies as objective as possible, while at the same time, not trivialising the assessment into ticking off analytic criteria (Harsch and Martin 2012), and (6) the assessments should be embedded in a competence-based teaching and learning process.…”

Section: Implications Of Theory For the Assessment Designmentioning

confidence: 99%

An assessment innovation as flywheel for changing teaching and learning

Gulikers

Runhaar

Mulder

2017

Journal of Vocational Education & Training

View full text Add to dashboard Cite

This contribution shows an example of how an assessment innovation can serve as a flywheel for changing teaching and learning. This article first explains the development of an authentic, competence-based performance assessment in pre-vocational secondary education (pre-VET). Using these assessments in pre-VET, that is preparatory to VET, is important for motivating students for a future in VET. Second, in an action research 34 teacher teams of 11 pre-VET institutions implement the new assessment, supported by researchers and teacher coaches, to illuminate how it motivates students, prepares them for VET, and influences teaching and learning. Data from 76 teachers, 68 students, 24 teacher coaches and 3 researchers showed how implementing these assessments raised questions about changing educational content and pedagogy to fit a competence-based approach. Increasing teachers' expectations of students, changing studentteacher interaction patterns, and developing teachers' practical understanding of competencies and how they can be observed and discussed in assessments are some of the intriguing challenges that require further support for proper implementation of competence-based assessments and developing an aligned competence-based teaching and learning process.

show abstract

“…Technical adequacy of quality measures. Numerous studies have analyzed the differences between scores obtained using both holistic and analytic scoring methods (Barkaoui, 2011;Harsch & Martin, 2013). These studies found that holistic scoring led to higher levels of rater agreement.…”

Section: Scoring Approachesmentioning

confidence: 99%

“…This is difficult to capture in a single holistic score; therefore, the holistic score only captures the low skill levels. Holistic scores may also mask deviances in how descriptors are applied (Harsch & Martin, 2013). With a holistic score, it is not possible to examine how raters applied the different parts of the scale in order to form their holistic criterion scores or overall scores.…”

Section: Scoring Approachesmentioning

confidence: 99%

Combining quality and curriculum-based measurement

Ganzeveld¹

View full text Add to dashboard Cite

Comparing holistic and analytic scoring methods: issues of validity and reliability

Cited by 43 publications

References 33 publications

Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications

Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications

An assessment innovation as flywheel for changing teaching and learning

Combining quality and curriculum-based measurement

Contact Info

Product

Resources

About