An Examination of the Reliabilities of Two Choral Festival Adjudication Forms

Abstract

The purpose of this study was to compare the reliability of a common school choral festival adjudication form with that of a second form that is a more descriptive extension of the first. Specific research questions compare the interrater reliabilities of each form, the differences in mean scores of all dimensions between the forms, and the concurrent validity of the forms. Analysis of correlations between all possible pairs of four judges determined that the interrater reliability of the second form was stronger than that of the traditional form. Moderate correlations between the two forms further support the notion that the two forms measured the dimensions in somewhat different ways, suggesting the second form offered more specific direction in the evaluation of the choral performances. The authors suggest continued development of language and descriptors within a rubric that might result in increased levels of interrater reliability and validity in each dimension.

Reliability and Perceived Pedagogical Utility of a Weighted Music Performance Assessment Rubric

“…Conclusions, although limited to this study, indicate that this rubric demonstrated good internal consistency and was perceived by directors and adjudicators to offer improved pedagogical utility when compared to other, nonrubric forms. Therefore, our conclusions generally support the findings of previous researchers (e.g., Asmus, 1999;Ciorba & Smith, 2009;Gordon, 2002;Norris & Borst, 2007;Whitcomb, 1999) who also advocated the use of performance assessment rubrics with dimensions and descriptors.…”

“…This finding led the researchers to conclude that multidimensional rubrics can provide useful means of tracking student achievement. Norris and Borst (2007) compared a traditional choral music festival assessment form to a rubric-style version of the form. Both forms incorporated tone quality, diction, blend, intonation, rhythm, balance, and interpretation as dimensions.…”

Large-Group Contest Ratings and Music Teacher Evaluation: Issues and Recommendations

“…Norris and Borst (2007) found that interrater reliability was stronger for individual categories (e.g., tone, intonation, technique) and final scores when judges evaluated choral ensembles using a rubric with detailed descriptions for each possible score (1-5) in all categories as opposed to a traditional form with no category descriptions. Nonetheless, reliability for categories at largegroup festivals in Kansas varied from .47 (other) to .77 (tone), even with descriptors for each level of achievement (Latimer, Bergee, and Cohen 2010).…”

“…Furthermore, individual category ratings often are unreliable (e.g., Burnsed, Hinkle, and King 1985) but improve when adjudicators use a rubric that describes achievement at each level (Norris and Borst 2007). In addition, the interrater reliability of final ratings is generally acceptable but can vary widely from one adjudication panel to the next (e.g., Hash 2012).…”

“…These data will help evaluators make valid interpretations of ratings by providing norms with which to compare individual results (MSMTE 2011). • Contest organizations should continue to examine ways of increasing the reliability of large-group festivals by revising adjudication forms (e.g., Norris and Borst 2007) and using trained, experienced judges who are familiar with the medium under evaluation (e.g., Brakel 2006). • Teachers, administrators, and researchers should work together to determine how best to use contest data for calculating student growth.…”