Situational Effects on Observer Accuracy: Behavioral Predictability, Prior Experience, and Complexity of Coding Categories

Mash, Eric J.; McElwee, John

doi:10.2307/1127957

Cited by 53 publications

(23 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Few studies have examined methods for conducting initial observer training. Research has shown that recording unpredictable (rather than predictable) events during practice produces better generalization to novel situations (Mash & McElwee, 1974), that it takes longer to acquire competence in observing a larger (compared to a smaller) number of events (Bass, 1987), and that supervised (compared to unsupervised) practice tends to generate the recording of a larger number of events (Wildman, Erickson, & Kent, 1975). These studies provided useful information for structuring the content or supervision of practice sessions; however, they did not evaluate training methods per se.…”

mentioning

confidence: 99%

Observer Training Revisited: A Comparison of in Vivo and Video Instruction

Dempsey

Iwata

Fritz

et al. 2012

J of App Behav Analysis

View full text Add to dashboard Cite

We compared the effects of 2 observer‐training procedures. In vivo training involved practice during actual treatment sessions. Video training involved practice while watching progressively more complex simulations. Fifty‐nine undergraduate students entered 1 of the 2 training conditions sequentially according to an ABABAB design. Results showed that the 2 training methods produced almost identical scores on a posttraining observational test; however, the video method required fewer training sessions to complete.

show abstract

mentioning

confidence: 99%

Observer Training Revisited: A Comparison of in Vivo and Video Instruction

Dempsey

Iwata

Fritz

et al. 2012

J of App Behav Analysis

View full text Add to dashboard Cite

show abstract

“…The quality of observational data is usually judged from interobserver agreement scores because of the difficulty in procuring a criterion against which to measure the observers' actual accuracy. Possible accuracy criteria, however, include mechanical measurements of behavior (e.g., Bechtel, 1967), mechanically generated re-479 1981, 141, 479-489 NUMBER 4 (WINTER 198 1) sponses (e.g., Repp, Roberts, Slack, Repp, & Berkler, 1976), recorded behaviors orchestrated by a predetermined script (e.g., Mash & McElwee, 1974), and consensually validated criterion protocols produced by the observation of multiple observers (e.g., Kent et al, 1974;Foster & Cone, 1980). Although agreement is generally used to evaluate the quality of observational data, agreement and accuracy are not the same (Foster & Cone, 1980;Johnson & Bolstad, 1973;Kazdin, 1977).…”

mentioning

confidence: 99%

The Effects of Instructions and Calculation Procedures on Observers' Accuracy, Agreement, and Calculation Correctness

Boykin

Nelson

1981

J of App Behav Analysis

View full text Add to dashboard Cite

Although the quality of observational data is generally evaluated by observer agreement, measures of both observer agreement and accuracy were available in the present study. Videotapes with a criterion protocol were coded by 16 observers. All observers calculated agreement scores both on their own and their partner's data and on a contrived data set misrepresented as data collected by other observers. Compared with agreement scores calculated by the experimenter, observers erroneously inflated their own agreement scores and deflated the agreement scores on the contrived data. Half of the observers (n = 8) had been given instructions emphasizing the importance of accuracy during observation while the other half had been given instructions emphasizing interobserver agreement. Accuracy exceeded agreement for the former group, whereas agreement exceeded accuracy for the latter group. The implications are that agreement should be calculated by the experimenter and that the accuracy‐agreement relationship can be altered by differential observer instructions.

show abstract

“…Gowland and coworkers (1995) also reported that the complexity of the GMPM scoring system was an explanation for the differences in scoring seen between evaluators. Interobserver reliability has been found to be influenced by the complexity of the coding system (Mash and McElwee 1974). Complexity refers to the number of different response categories of an observational coding system and the number of different behaviors that are scored within a particular observational system (Mash andMcElwee 1974, Kazdin 1977).…”

Section: Reliabilitymentioning

confidence: 99%

“…Interobserver reliability has been found to be influenced by the complexity of the coding system (Mash and McElwee 1974). Complexity refers to the number of different response categories of an observational coding system and the number of different behaviors that are scored within a particular observational system (Mash andMcElwee 1974, Kazdin 1977). Systems with more categories and behaviors are more complex than systems with fewer categories, thus making reliability more difficult.…”

Section: Reliabilitymentioning

confidence: 99%

Interobserver reliability of the Gross Motor Performance Measure: preliminary results

Thomas

Buckon

Phillips³

et al. 2001

Develop Med Child Neuro

View full text Add to dashboard Cite

Although assessment of the quality of movement in children with cerebral palsy (CP) is difficult, the development of the Gross Motor Performance Measure (GMPM) has facilitated this process. In order to determine the interobserver reliability of the GMPM, 36 children with spastic neuromuscular disorders (mean age 7 years, range 4 to 15 years) were evaluated using four of the five dimensions of the GMPM. Percent Agreement, Intraclass Correlations, and Kappas were calculated by both dimension and attribute to determine reliability. In addition, reliability measures were evaluated over time to determine whether reliability improved with continual use of the GMPM. Overall, interobserver reliability was in the ‘fair to good’ category regardless of the reliability measure used in the analysis. Reliability scores improved over time with a greater number of individual item scores moving from the ‘fair to good’ category to the ‘excellent’ category. Results from this study indicate that it is possible to assess reliably the quality of movement in children with CP.

show abstract

Situational Effects on Observer Accuracy: Behavioral Predictability, Prior Experience, and Complexity of Coding Categories

Cited by 53 publications

References 0 publications

Observer Training Revisited: A Comparison of in Vivo and Video Instruction

Observer Training Revisited: A Comparison of in Vivo and Video Instruction

The Effects of Instructions and Calculation Procedures on Observers' Accuracy, Agreement, and Calculation Correctness

Interobserver reliability of the Gross Motor Performance Measure: preliminary results

Contact Info

Product

Resources

About