A review of reliability procedures for measuring observer agreement

Towstopiat, Olga

doi:10.1016/0361-476x(84)90039-0

Cited by 8 publications

(5 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The attractiveness of the present approach lies in the interpretability of the terms of expected and observed disagreement in terms of average multivariate distances between observations, and the simplicity with which the definition of agreement as a function of these terms enables the extension of kappa to the multivariate case. The objection to multivariate extensions of kappa, that they lack an interpretation (e.g., Towstopiat, 1984), no longer seems valid. Our approach, unlike most previous, addresses the case of multivariate nominal data.…”

Section: Discussionmentioning

confidence: 99%

A Measure of Agreement for Interval or Nominal Multivariate Observations by Different Sets of Judges

Janson

Olsson

2004

Educational and Psychological Measurement

View full text Add to dashboard Cite

This article addresses the problem of accounting overall multivariate chance-corrected interobserver agreement when targets have been rated by different sets of judges (not necessarily equal in number). The proposed approach builds on Janson and Olsson’s multivariate generalization of Cohen’s kappa but incorporates weighting for number of judges and applies an expression for expected disagreement suitable for the case with different judges. The authors suggest that the attractiveness of this approach to multivariate agreement measurement lies in the interpretability of the terms of expected and observed disagreement as average distances between observations, and that addressing agreement without regard to the covariance structure among variables has advantages in simplicity and interpretability. Correspondences to earlier approaches are noted, and application of the proposed measure is exemplified using hypothetical data sets.

show abstract

Section: Discussionmentioning

confidence: 99%

A Measure of Agreement for Interval or Nominal Multivariate Observations by Different Sets of Judges

Janson

Olsson

2004

Educational and Psychological Measurement

View full text Add to dashboard Cite

show abstract

“…Just how high kappa needs to be is an open question, although Suen and Lee (1985) suggested 0.60 as a lenient criterion and 0.75 as more stringent. Towstopiat (1984), however, noting that the use of kappa is limited to situations when only two observers are involved, suggested the use of multivariate agreement models.…”

Section: Measuring Agreement Between Observersmentioning

confidence: 97%

The behavioural approach in schools: a time for caution revisited

Harrop

Swinson

2007

Educational Studies

View full text Add to dashboard Cite

This paper takes as its starting point an examination of the current status of some of the concerns that were raised in the mid-1980s about methodological problems faced by educational researchers using the behavioural approach in schools. These concerns included the measurement of agreement between observers, the interpretation of raw data extracted, the potential influences of observers and the inherent properties of research designs. Subsequently, some more wide-ranging concerns are considered, in particular the kinds of behaviour selected for treatment, the lack of analysis of what is involved in teachers' positive responses to pupils' behaviour and the relatively uninvestigated effects of teachers' negative responses. The conclusions are presented as a series of points that are listed, as far as possible, in the order in which they confront the investigator.More than 20 years ago an article was written (Harrop, 1985) discussing some of the methodological problems faced by researchers using the behavioural approach, with their associated research designs, in school classrooms. The areas of concern at that time were centred on the way in which data from investigations were recorded and processed. The 'human observer' was likened to a faulty 'cumulative recorder'. More specifically, problems were identified in:1. measuring agreement between observers;

show abstract

“…Cronbach, Gleser, Nanda, and Rajaratnam (1972) extended the components of variance approach to complex experimental designs in which variability among different raters is one factor influencing &dquo;generalizability.&dquo; However, none of those authors discuss the problem of separating differing individual rater reliability as such. More recent authors have compared other approaches to estimating overall consistency among several raters as distinct from the specific reliabilities of individual raters (James, Demaree, & Wolf, 1984;Jones, Johnson, Butler, & Main, 1983;Towstopiat, 1984). Coefficient alpha (Cronbach, 1951) is another frequently used estimate of the reliability of a composite test or the average reliability of subtests entering into the composite (Conn & Ramanaiah, 1990;Dolan, Lacey, & Evans, 1990).…”

Section: Estimating Individual Rater Reliabilitiesmentioning

confidence: 99%

Estimating Individual Rater Reliabilities

Overall¹,

Magee²

1992

Applied Psychological Measurement

View full text Add to dashboard Cite

Rating scales have no inherent reliability that is independent of the observers who use them. The often reported interrater reliability is an average of perhaps quite different individual rater reliabilities. It is possible to separate out the individual rater reliabilities given a number of independent raters who observe the same sample of ratees. Under certain assumptions, an external measure can replace one of the raters, and individual reliabilities of two independent raters can be estimated. In a somewhat similar fashion, estimates of treatment effects present in ratings by two independent raters can provide the external frame of reference against which differences in their individual reliabilities can be evaluated. Models for estimating individual rater reliabilities are provided for use in selecting, evaluating, and training participants in clinical research.Rating scales provide essential measurements for much clinical research. References to the reliability of a rating instrument are common; however, clinical rating scales do not have inherent reliability that is independent of the skill of the observers who use them. The fact that raters differ in the accuracy of their judgments, and in the consistency with which those judgments are recorded, is a major problem for clinical research. This paper describes several models that provide estimates of the reliability of a rating scale used by a particular individual rater. The use of such models may be helpful in selecting and training raters for participation in clinical research.Concern with interrater consistency has generally focused on the reliability of the rating instrument or procedure. Estimates of the reliability of mean ratings calculated across raters and estimates of the average reliability of the individual raters using an instrument have been provided (Armstrong, 1981). Haggard (1958) and Winer (1962) were among the first to introduce psychologists to the intraclass correlation coefficient as a measure of average interrater reliability, and Shrout and Fleiss (1979) have elaborated the statistical models underlying the intraclass correlation coefficient. Cronbach, Gleser, Nanda, and Rajaratnam (1972) extended the components of variance approach to complex experimental designs in which variability among different raters is one factor influencing &dquo;generalizability.&dquo; However, none of those authors discuss the problem of separating differing individual rater reliability as such. More recent authors have compared other approaches to estimating overall consistency among several raters as distinct from the specific reliabilities of individual raters ). Coefficient alpha can be calculated (for standardized ratings) from the correlation matrix relating ratings made by several raters; in that regard, it is more logically compared with the models for individual rater reliabilities discussed here than is the intraclass coefficient obtained from an analysis of variance model. This comparison emphasizes further the difference between average rater ...

show abstract

A review of reliability procedures for measuring observer agreement

Cited by 8 publications

References 33 publications

A Measure of Agreement for Interval or Nominal Multivariate Observations by Different Sets of Judges

A Measure of Agreement for Interval or Nominal Multivariate Observations by Different Sets of Judges

The behavioural approach in schools: a time for caution revisited

Estimating Individual Rater Reliabilities

Contact Info

Product

Resources

About