Effect of merging levels of locomotion scores for dairy cows on intra- and interrater reliability and agreement

Abstract

Locomotion scores are used for lameness detection in dairy cows. In research, locomotion scores with 5 levels are used most often. Analysis of scores, however, is done after transformation of the original 5-level scale into a 4-, 3-, or 2-level scale to improve reliability and agreement. The objective of this study was to evaluate different ways of merging levels to optimize resolution, reliability, and agreement of locomotion scores for dairy cows. Locomotion scoring was done by using a 5-level scale and 10 experienced raters in 2 different scoring sessions from videos from 58 cows. Intra- and interrater reliability and agreement were calculated as weighted kappa coefficient (κw) and percentage of agreement (PA), respectively. Overall intra- and interrater reliability and agreement and specific intra- and interrater agreement were determined for the 5-level scale and after transformation into 4-, 3-, and 2-level scales by merging different combinations of adjacent levels. Intrarater reliability (κw) ranged from 0.63 to 0.86, whereas intrarater agreement (PA) ranged from 60.3 to 82.8% for the 5-level scale. Interrater κw=0.28 to 0.84 and interrater PA=22.6 to 81.8% for the 5-level scale. The specific intrarater agreement was 76.4% for locomotion level 1, 68.5% for level 2, 65% for level 3, 77.2% for level 4, and 80% for level 5. Specific interrater agreement was 64.7% for locomotion level 1, 57.5% for level 2, 50.8% for level 3, 60% for level 4, and 45.2% for level 5. Specific intra- and interrater agreement suggested that levels 2 and 3 were more difficult to score consistently compared with other levels in the 5-level scale. The acceptance threshold for overall intra- and interrater reliability (κw and κ ≥0.6) and agreement (PA ≥75%) and specific intra- and interrater agreement (≥75% for all levels within locomotion score) was exceeded only for the 2-level scale when the 5 levels were merged as (12)(345) or (123)(45). In conclusion, when locomotion scoring is performed by experienced raters without further training together, the lowest specific intra- and interrater agreement was obtained in levels 2 and 3 of the 5-level scale. Acceptance thresholds for overall intra- and interrater reliability and agreement and specific intra- and interrater agreement were exceeded only in the 2-level scale.

Experienced and inexperienced observers achieved relatively high within-observer agreement on video mobility scoring of dairy cows

“…Overall, our results are in the range of observers experienced in locomotion scoring (Schlageter-Tello et al, 2014a), who achieved between 60 and 82% of perfect agreement. Alternatively, kappa statistic could also be used for comparisons because it takes into account the agreement by chance for the answers on replicate 1 and 2 (Cohen, 1968).…”

“…The only practically relevant alternative is to show and assess the same video sample twice. To the best of our knowledge, the largest study assessing within-observer agreement with video included up to 10 experienced observers (Schlageter-Tello et al, 2014a). Therefore, little is known about the within-observer agreement among farmers, students, and veterinarians outside research environments and which factors may influence their performance.…”

“…If farmers and veterinarians disagree with themselves, they obviously cannot agree on a joint criterion for decision making. Observers with considerable cattle experience obtained 60 to 83% within-observer PA while gait scoring on video, yet some pairs of observers encountered a much lower between-observer agreement (Schlageter-Tello et al, 2014a). Lower between-observer agreement than within-observer agreement can be explained by simple probability calculations.…”

Lameness detection in dairy cattle: single predictor v. multivariate analysis of image-based posture processing and behaviour and performance sensing

“…Serving as a reference for model development, Schlageter-Tello et al (2014a) reported that the binary merging level influences the reliability of the locomotion score. The lowest true positive rates were in this study obtained for the lame cows (LS ⩾ 3).…”

“…This procedure combines several gait and posture characteristics such as back arch, gait asymmetry and weight bearing into one overall locomotion score (LS) (Sprecher et al, 1997;Flower and Weary, 2009;Schlageter-Tello et al, 2014b). The procedure is however subjective (Schlageter-Tello et al, 2014a), time-consuming and costly. Due to increased farm size and limited time per animal, visual observation of cow health and welfare is evolving to automated monitoring systems.…”