The analysis of the impact of individual weighting factor on individual scores

Abstract

In this study, category-based self and peer assessment were applied twice in a semester in an Elementary Science Teaching Methods course in order to assess individual contributions of group members to group projects as well as to analyze the impact of Individual Weighting Factors (IWF) on individual scores and individual grades. IWF were calculated by applying Conway et al.'s (1993) formula. The resulting individual scores were analyzed to find out the impact of IWF on individual scores and individual grades. It was found that the differentiation of individual scores was very large in both applications and caused dramatic changes in some of the students' grades. For this reason, the IWF had to be scaled down to decrease its effect on individual grades. After scaling down the IWF, more acceptable distribution of student grades was obtained.

Peer assessment accounting for student agreement

“…His approach is based upon tweaking the slope parameter based upon interpreting an ANOVA-like statistic, which he denotes by A, which itself is a function of the f-statistic. His approach is solidly grounded in statistical theory but requires considerable care in application, which like Kilic and Cakan's (2006) approach might be labour-and time-intensive. In addition, many undergraduates (and some non-technically minded instructors) might find his method difficult to understand.…”

“…Kilic and Cakan (2006), for example, used a scaled weighting factor (SWF) provided in Conway et al (1993) to adjust the impact of the IWF method on individual grades. The SWF is a function of the IWF and scaling parameter, denoting the weighting parameter by p, the SWF is given by, SWF = IWF(1−p) + p. With this scaling, the individual grade is computed as the product of the group grade with the SWF instead of the IWF.…”

“…That being said, many practitioners and researchers do embrace the IWF method (cf. Cheng and Warren 2000;Gatfield 1999;Goldfinch and Summers 1996;Gupta 2004;Kilic and Cakan 2006;Lejk and Wyvill 2001a, 2001b, 2002Sharp 2006).…”

“…For more than 10 years the method of Conway et al (1993) has sparked discussion and research (Cheng and Warren 2000;Gatfield 1999;Gupta 2004;Kilic and Cakan 2006;Lejk and Wyvill 2001a, 2001b, 2002Sharp 2006). This method (which will be subsequently referred to as the individual weighting factor (IWF) method) is based upon calculating an IWF based upon peer assessment scores.…”

“…Unfortunately, applying the IWF method often requires time-consuming exploratory data analysis to choose a weighting parameter for a scaling factor -also proposed in Conway et al (1993) -because of the potential for the IWF method to yield a high spread of grades, including grades over 100% (Kilic and Cakan 2006;Sharp 2006). The present study is intended for researchers and practitioners who would be prone to use the IWF method but would consider the exploratory data analysis required to chose a weighting parameter too time-consuming or too arbitrary to be practical.…”