Using generalizability theory to investigate the reliability of peer assessment

Abstract

In this study, the effectiveness of peer assessment, which has an important role in measurement and evaluation, was attempted to be defined. For this purpose, performance task, which is one of the alternative assessment techniques, was evaluated with the help of a scoring rubric prepared by the researchers. As a basic research, the working group was 41 sophomore students and their instructor. Three of 41 students were acted as rater and they rated their 38 peers' performances with the instructor. The analysis of the data was carried out by using fully crossed two-facet design (sxtxr) of generalizability theory in three steps: G-studies for peer and peers-instructor ratings and D-study for peer ratings. According to the results of the G studies, the reliability coefficient obtained from the peer ratings and peer-instructor ratings were quite high (0.86 and 0.82 respectively). According to the result of the D study of peer ratings, just two peer raters are enough for getting high reliability coefficient. With the help of the gained results, it is suggested that peer assessment, which is effective on learning and decision making processes of students, should be used more often in education systems.

Investigation of the Reliability of Teachers, Self and Peer Assessments at Primary School Level with Generalizability Theory

“…the variance being negative may be that the study group is small, or the measurement pattern is not suitable (Taşdelen-Teker et al, 2016). In this study, since there was no problem with the pattern, the finding is thought to be related to the size of the study group.…”

“…When the literature is examined; it is clear that Classical Test Theory (CTT), Generalizability (G) Theory and Item Response Theory (IRT) are employed to identify the reliability of the measurement results (Güler, 2011). Especially when it is focused on the studies that try to determine the reliability between different raters, it is seen that G Theory or IRT-based methods have been preferred more frequently compared to CTT (Atılgan, 2005;Börkan, 2017;Büyükkıdık & Anıl, 2015;Farrokhi, Esfandiari & Dalili, 2011;Farrokhi, Esfandiari & Schaefer, 2012;Karakaya, 2015;Matsuno, 2009;Nalbantoğlu-Yılmaz, 2017;Taşdelen-Teker, Şahin & Baytemir, 2016;Yıldıztekin, 2014). If a comparison is made on the basis of CTT and G-theory, it is seen that only one error source is allowed to be estimated in the reliability determination studies based on the CTT, while all error sources can be included in the analysis in the reliability analysis based on G theory.…”

“…When Turkish lesson is in question, it is seen that the component explaining the total variance is the main effect of the student (s). The fact that the main effect of the student (s) has the largest percentage of explanation is desirable during the assessment process, because it is obtained that the differences between students can be revealed by the assessment process (Atılgan, 2005;Doğan & Anadol, 2017;Taşdelen-Teker et al, 2016). It is seen that the total variance is the second mostly explained component by the residues (src,e) following the main effect of the student (s).…”

“…Alanyazın incelendiğinde; ölçme sonuçlarına ait güvenirliğin belirlenmesinde Klasik Test Kuramı (KTK), Genellenebilirlik (G) Kuramı ve Madde Tepki Kuramının (MTK) işe koşulduğu görülmektedir (Güler, 2011). Özellikle farklı puanlayıcılar arasındaki güvenirliğin belirlenmeye çalışıldığı çalışmalara odaklanıldığında, son zamanlarda G Kuramı ya da MTK'ye dayalı yöntemlerin KTK'ye nazaran daha sık tercih edildiği görülmektedir (Atılgan, 2005;Börkan, 2017;Büyükkıdık & Anıl, 2015;Farrokhi, Esfandiari & Dalili, 2011;Farrokhi, Esfandiari & Schaefer, 2012;Karakaya, 2015;Matsuno, 2009;Nalbantoğlu-Yılmaz, 2017;Taşdelen-Teker, Şahin & Baytemir, 2016;Yıldıztekin, 2014). KTK ve G Kuramı temelinde bir karşılaştırma yapılırsa; KTK temeline dayalı yapılan güvenirlik belirleme çalışmalarında yalnızca bir hata kaynağına bağlı kestirim yapılmasına izin verilirken, G kuramına dayalı yapılan güvenirlik belirleme çalışmalarında tüm hata kaynaklarının analize dahil edilebildiği görülmektedir.…”

Akran değerlendirmede analitik rubrikle ve genel izlenimle yapılan puanlamaların incelenmesi

“…Peer assessment is used often in higher education, as is the case across other levels of education (Amo & Jareno, 2011;van den Berg, Admiraal, & Pilot, 2006). Studies carried out in the academic field set forth the common usage of peer assessment in higher education (Hanrahan & Isaacs, 2001;Macpherson, 1999;Mehrdad, Bigdelib, & Ebrahimia, 2012;Smith, Cooper, & Lancaster, 2002;Şahin, Taşdelen-Teker, & Güler, 2016;Taşdelen-Teker, Şahin, & Baytemir, 2016;Wen & Tsai, 2006). Sluijmans, Brand-Gruwel, van Merrienboer, and Bastiaens (2003) underline the fact that the use of self-assessment, as well as peer assessment, is gradually increasing in teacher-training institutions as both methods comply with contemporary pedagogical approaches related to training teacher candidates.…”