Reliability formulas for independent decision data when reliability data are matched

Rajaratnam, Nageswari

doi:10.1007/bf02289730

Cited by 43 publications

(21 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Classical estimates of c and u under model (1) may be obtained by performing an analysis of variance for repeated measurements (see, for example, Rajaratnam [7], Bartko [8] and Shrout and Fleiss [9]). From data Y ij , we calculate the following sums of squares:…”

Section: Model and Associated Parametersmentioning

confidence: 99%

Assessing intrarater, interrater and test–retest reliability of continuous measurements

Rousson

Gasser

Seifert

2002

Statistics in Medicine

334

246

View full text Add to dashboard Cite

In this paper we review the problem of defining and estimating intrarater, interrater and test-retest reliability of continuous measurements. We argue that the usual notion of product-moment correlation is well adapted in a test-retest situation, whereas the concept of intraclass correlation should be used for intrarater and interrater reliability. The key difference between these two approaches is the treatment of systematic error, which is often due to a learning effect for test-retest data. We also consider the reliability of a sum and a difference of variables and illustrate the effects on components. Further, we compare these approaches of reliability with the concept of limits of agreement proposed by Bland and Altman (for evaluating the agreement between two methods of clinical measurements) and show how product-moment correlation is related to it. We then propose new kinds of limits of agreement which are related to intraclass correlation. A test battery to study the development of neuro-motor functions in children and adolescents illustrates our purpose throughout the paper.

show abstract

Section: Model and Associated Parametersmentioning

confidence: 99%

Assessing intrarater, interrater and test–retest reliability of continuous measurements

Rousson

Gasser

Seifert

2002

Statistics in Medicine

334

246

View full text Add to dashboard Cite

show abstract

“…Others, such as Cohen's kappa, are limited to two raters and therefore would not have much application to student evaluations of instruction. Rajaratnam's (1960) consensus estimate examined in this study can also be problematic, in that, when raters are considered fixed, and Bartko (1966) points out that it is impossible to derive unconfounded estimates of variance components when a rater by evaluation item interaction is present. In spite of these difficulties, the worst possible state of affairs is the present one, where we do not even attempt to assess the reliability of student ratings in individual classes.…”

Section: Cautions and Limitationsmentioning

confidence: 94%

Assessing the reliability of student evaluations of teaching: choosing the right coefficient

Morley

2013

Assessment & Evaluation in Higher Education

View full text Add to dashboard Cite

Many of the studies used to support the claim that student evaluations of teaching are reliable measures of teaching effectiveness have frequently calculated inappropriate reliability coefficients. This paper points to three coefficients that would be appropriate depending on if student evaluations were used for formative or summative purposes. Results from the present study indicated that students had very low absolute inter-rater reliability, but somewhat higher consistency inter-rater reliability.

show abstract

“…This approach was proposed by Rajaratnam [7] and Rajaratnam [1]. The method is in general biased and the resulting estimates may be negative.…”

Section: )mentioning

confidence: 99%

Modified profile likelihood approach for certain intraclass correlation coefficients

Xiao

Liu

2013

Comput Stat

View full text Add to dashboard Cite

In this paper we consider the problem of constructing confidence intervals and lower bounds for the intraclass correlation coefficient in an interrater reliability study where the raters are randomly selected from a population of raters.The likelihood function of the interrater reliability is derived and simplified, and the profile likelihood based approach is readily available for computing the confidence intervals of the interrater reliability. Unfortunately, the confidence intervals computed by using the profile likelihood function are in general too narrow to have the desired coverage probabilities. From the point view of practice, a conservative approach, if is at least as precise as any existing method, is preferred since it gives the correct results with a probability higher than claimed. Under this rationale, we propose the so-called modified profile likelihoodapproach in this paper. Simulation study shows that, the proposed method in general has better performance than currently used methods. INDEX WORDS:Interrater reliability, Profile likelihood, Confidence intervals, Generalized P-values, Modified large-sample approach MODIFIED PROFILE

show abstract

Reliability formulas for independent decision data when reliability data are matched

Cited by 43 publications

References 7 publications

Assessing intrarater, interrater and test–retest reliability of continuous measurements

Assessing intrarater, interrater and test–retest reliability of continuous measurements

Assessing the reliability of student evaluations of teaching: choosing the right coefficient

Modified profile likelihood approach for certain intraclass correlation coefficients

Contact Info

Product

Resources

About