Faculty Interjudge Reliability of Music Performance Evaluation

Abstract

Assessment of music performance in authentic contexts remains an underinvestigated area of research. This study is an examination of one such context, the inter-judge reliability of faculty evaluation of end-of-semester applied music performances. Brass (n = 4), percussion (n = 2), woodwind (n = 5), voice (n = 5), piano (n = 3), and string (n = 5) instructors evaluating a recent semester's applied music juries at a large university participated in the study. Each evaluator completed a criterion-specific rating scale for each performer and assigned each performance a global letter grade not shared with other evaluators or with the performer. Interjudge reliability was determined for each group's rating scale total scores, subscale scores, and the letter-grade assessment. All possible permutations of two, three, and four were examined for interjudge reliability, and averaged correlations, standard deviations, and ranges were determined. Full-panel interjudge reliability was consistently good regardless of panel size. All total score reliability coefficients were statistically significant, as were all coefficients for the global letter-grade assessment. All subscale reliabilities for all groups except Percussion (which, with an n of 2, had a stringent significance criterion) were statistically significant, with the exception of the Suitability subscale in Voice. For larger panels (ns of 4 and 5), rating scale total score reliability was consistently but not greatly higher than reliability for the letter-grade assessment. There was no decrease of average reliability as group size incrementally decreased. Permutations of two and three evaluators, however, tended on average to exhibit more variability, greater range, and less uniformity than did groups of four and five. No differences in reliability were noted among levels of experience or between teaching assistants and faculty members. Use of a minimum of five adjudicators for performance evaluation in this context was recommended.

Effects of Excerpt Tempo and Duration on Musicians' Ratings of High-Level Piano Performances

“…Research in musical adjudication has spawned a number of facetfactorial studies, elegantly summarized in Bergee (2003), as well as additional research. Studies have almost always dealt with intra-and interjudge reliability and issues of validity, as well as with correlations between test items.…”

Sight over sound in the judgment of music performance

“…With the general consensus on the importance of sound in the domain of music, as "an art of sound" (40), it follows that experts and key decision makers would privilege auditory-related rating in professional evaluation and assessment, even when such items show insufficient reliability (41)(42)(43)(44)(45). However, despite all that is invested in the auditory domain, low interrater correlations suggest that such basis of evaluation is an unreliable process.…”

An Examination of the Reliabilities of Two Choral Festival Adjudication Forms

“…Each dimension on Form B was rated lower than its counterpart on Form A (since the number &dquo;1&dquo; is considered the &dquo;best&dquo; score, lower scores or ratings are indicated by higher numbers). Paired-samples t-tests revealed significant differences between forms at the .05 level or lower in the following dimensions: tone ( t = -2.27, p = .027), diction ( t = -2.40, p=.02), blend ( t = -3.36, p = .001 ) , intonation ( t = -2.34, p = .023), rhythm ( t = -2.80, p = .007), balance ( = -4.09, p < .001 ) , total score ( = -3.94, p < .001), and rating ( (Garman, Barry, & DeCarbo, 1991;Bergee, 1988Bergee, , 1989Bergee, , 1993Bergee, , 1997Bergee, , 2003 The additional analysis of the means of dimensions, total score, and overall ratings corroborates the above comments (see Table 1 and above t-test results). Form B yielded significantly different ratings in every dimension except interpretation, suggesting that the adjudicators in this setting rated the choirs more severely when using Form B.…”