The number of feedbacks needed for reliable evaluation. A multilevel analysis of the reliability, stability and generalisability of students’ evaluation of teaching

Rantanen, Pekka

doi:10.1080/02602938.2011.625471

Cited by 30 publications

(30 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Morley (2014) showed that inter-rater reliabilities were considerably lower than internal consistency indices. Rantanen (2013) calculated mean of inter-rater reliabilities for SET as 0.29, whereas Solomon et al (2007) found a value of 0.22. Thus the assumption that high internal consistency does not guarantee agreement among students on instructors should not be assumed.…”

Section: Inter-rater Reliabilitymentioning

confidence: 94%

“…Cronbach's Alpha does not take errors due to ratings of students into account (Rantanen, 2013). However that component of variance is what constitutes the main error source related to SET (Sun & Valiga, 1997).…”

Section: Inter-rater Reliabilitymentioning

confidence: 99%

“…In their study, they estimated that 30 to 40 feedbacks are required to achieve an inter-rater reliability of 0.92, a value suggested for high-stake decisions. Rantanen (2013) showed that 15 feedbacks are needed, if one wishes to make inference about an instructor on a single course. But if a more comprehensive evaluation on an instructor is to be made, more feedback would be required.…”

Section: Feedbacks Required For Reliable Set Scoresmentioning

confidence: 99%

“…However, before calculating ICC indices, data were transposed so that rows presented items and columns the students, due to a requirement by the algorithm used by SPSS. Two values are selected in the present study: 0.70, a value that can be used for formative purposes and 0.90, for summative assessment including high-stake decisions (Morley 2013;Rantanen, 2013).…”

Section: Description Of the Analyses Conductedmentioning

confidence: 99%

See 3 more Smart Citations

Reliability-Related Issues in the Context of Student Evaluations of Teaching in Higher Education

Kalender

2015

IJHE

View full text Add to dashboard Cite

Student evaluations of teaching (SET) have been the principal instrument to elicit students' opinions in higher education institutions. Many decisions, including high-stake ones, are made based on SET scores reported by students. In this respect, reliability of SET scores is of considerable importance. This paper has an argument that there are some problems in choosing and using of reliability indices in SET context. Three hypotheses were tested: (i) using internal consistency measures is misleading in SET context since the variability is mainly due to disagreement between students' ratings, which requires use of inter-rater reliability coefficients, (ii) number of minimum feedbacks is not achieved in most of the classes, resulting unreliable decisions, and (iii) calculating reliability coefficient assuming a common factor structure across all classes is misleading because a common model may not be tenable for all. Results showed that use of internal consistency only to assess reliability of SET scores may result in wrong decisions. Considerable large numbers of missing feedbacks were observed to achieve acceptable reliability levels. Findings also indicated that factorial model differed across several groups.

show abstract

Section: Inter-rater Reliabilitymentioning

confidence: 94%

Section: Inter-rater Reliabilitymentioning

confidence: 99%

Section: Feedbacks Required For Reliable Set Scoresmentioning

confidence: 99%

Section: Description Of the Analyses Conductedmentioning

confidence: 99%

See 2 more Smart Citations

Reliability-Related Issues in the Context of Student Evaluations of Teaching in Higher Education

Kalender

2015

IJHE

View full text Add to dashboard Cite

show abstract

“…Despite this controversy, such evaluations are seen by many as a valuable and beneficial tool to improve teaching and student learning outcomes (Lattuca, & Domagal-Goldman, 2007;Dommeyer, Baum, Hanna, & Chapman, 2004). To maximize the SET benefits, Rantanen (2013) suggests applying SET surveys to suitable courses for each teacher, while Giles and colleagues (2004) recommend student partnership in designing and implementing evaluations.…”

Section: ‫حقن‬ ‫حسب‬ ‫العالي‬ ‫التعميم‬ ‫يف‬ ‫التدريسية‬ ‫لمفاعمية‬ ‫mentioning

confidence: 99%

Differential Item Functioning in Students Rating of Teaching Effectiveness Surveys in Higher Education According to Academic Disciplines: Data from a Saudi University

Alquraan

Kuwaiti

2017

J. educ. psychol. stud.

View full text Add to dashboard Cite

This study explored academic discipline as a source of differential item functioning (DIF) in students' rating of teaching quality and effectiveness at higher education institutions. Data utilized in this study was collected by Imam Abudalruman Bin Faisal University -KSA. The total number of surveys analyzed for the purpose of this study is 36459 from three colleges: Education, Health, and Engineering. Using Extended Rasch model (Rating Scale Model), the results show that the instrument contains four DIF items. The content of these four items confirm the possibility of considering discipline as a source of DIF items in students evaluation of teaching in higher education. Moreover, the results of the current study show that removing DIF items from the instrument increases construct validity.

show abstract

New Insights on Student Evaluation of Teaching in Italy

Bassi

Grilli

Paccagnella

et al. 2019

New Statistical Developments in Data Science

View full text Add to dashboard Cite

The number of feedbacks needed for reliable evaluation. A multilevel analysis of the reliability, stability and generalisability of students’ evaluation of teaching

Cited by 30 publications

References 23 publications

Reliability-Related Issues in the Context of Student Evaluations of Teaching in Higher Education

Reliability-Related Issues in the Context of Student Evaluations of Teaching in Higher Education

Differential Item Functioning in Students Rating of Teaching Effectiveness Surveys in Higher Education According to Academic Disciplines: Data from a Saudi University

New Insights on Student Evaluation of Teaching in Italy

Contact Info

Product

Resources

About