Experienced and inexperienced observers achieved relatively high within-observer agreement on video mobility scoring of dairy cows

Abstract

Assessment of lameness prevalence and severity requires visual evaluation of thelocomotion of a cow. Welfare schemes including locomotion assessments are increasingly being adopted, and more farmers and their veterinarians might implement a locomotion-scoring routine together. However, high within-observer agreement is a prerequisite for obtaining valid mobility scorings, and within-observer agreement cannot be estimated in a barn, because the gait of cows is dynamic and may change between 2 occasions. The objective of this study was to estimate the within-observer agreement according to the observers' educational background and experience with cattle, based on video recordings with very diverse types of gait. Groups of farmers, bovine veterinarians, first- and fourth-year veterinary students, researchers, and cattle-inexperienced sensory assessors evaluated mobility using a 5-point mobility score system developed specifically for walking cows (n=102 observers). The evaluation sessions were similar for all groups, lasted 75 min, and were organized as follows: introduction, test A, short training session, break, and test B. In total, video recordings of 22 cows were displayed twice in a random order (11 cows in each test × 2 replicates). Data were analyzed applying kappa coefficient, logistic regression, and testing for random effects of observers. The crude estimates of 95% confidence interval for weighted kappa in test A and B ranged, respectively, from 0.76 to 0.80 and 0.70 to 0.75. When adjusting for the fixed effects of video sample and gait scoring preferences, the probability of assigning the same mobility score twice to the same cow varied from 55% (sensory assessors) to 72% (fourth-year veterinary students). The random effect of the individual observers was negligible. That is, in general observers could categorize the mobility characteristics of cows quite well. Observers who preferred to assess the attributes back arch or the overall mobility score (based on uneven gait) had the highest agreement, respectively, 69 or 68%. The training session seemed insufficient to improve agreement. Nonetheless, even novice observers were able to achieve perfect agreement up to 60% of the 22 scorings with merely the experience obtained during the study (introduction and training session). The relatively small differences between groups, together with a high agreement, demonstrate that the new system is easy to follow compared with previously described scoring systems. The mobility score achieves sufficiently high within-observer repeatability to allow between-observer agreement estimates, which are reliable compared with other more-complex scoring systems. Consequently, the new scoring scale seems feasible for on-farm applications as a tool to monitor mobility within and between cows, for communication between farmers and veterinarians with diverse educational background, and for lamenessbenchmarking of herds.

Short communication: An observational study investigating inter-observer agreement for variation over time of body condition score in dairy cows

“…In such case, the expected inter-observer agreement would be 0.64 (0.80 times 0.80) illustrating the fact that a low intra-observer agreement for even just one rater would inevitably lead to a lower inter-observer agreement. Although intra-observer agreement could not be estimated in the current study, the findings of Garcia et al (2015) suggest that it would be greater than the observed inter-observer agreement value. The high inter-observer agreements observed in both exams of the present study suggest higher intra-observer agreement.…”

“…For practical reasons, and because the aim of the current study was inter-observer agreement, repeated BCS measurements within the same observers were not collected and intra-observer agreement could not be computed. As suggested by Garcia et al (2015), before computing inter-observer agreement, the first step in a validation process should be the assessment of the intra-observer agreement to evaluate if observers agree at least with themselves. The latter study gave an example where 2 observers using a categorical scale have intra-observer agreement of 0.80.…”

Lameness in Beef Cattle: UK Farmers' Perceptions, Knowledge, Barriers, and Approaches to Treatment and Control

“…Combining this with the difficulties expressed by farmers in terms of identification of lame animals suggests that training and practice is required in order to enable farmers to improve the prompt detection of lame animals. Although some dairy cattle studies have suggested that training may provide limited improvements in intra- and inter-observer agreement when locomotion scoring (26, 27), inter-observer reliability has been shown to increase with increased time/scoring sessions (28). Also, experienced scorers have been shown to perform better than less experiences scorers when using video footage of cattle (29).…”

A comparison of online and live training of livestock farmers for an on-farm self-assessment of animal welfare

“…It has already been shown that indicators for on-farm assessments can successfully be trained using pictures for body condition scoring (BCS) in dairy cows (Ferguson et al, 2006) and in goats (Vieira et al, 2015). The same applies to lameness in dairy cows by using videos (Garcia et al, 2015). However, information on training for farmers is lacking.…”