Inter-observer reliability of Qualitative Behavioural Assessments of sheep

Abstract

Qualitative Behaviour Assessment (QBA) is whole-animal methodology that assesses the expressive qualities of animal demeanour using descriptors such as 'relaxed', 'anxious' or 'content'. This study aimed to examine the inter-observer reliability of 12 fixed-list QBA descriptors for sheep that had been generated in a previous Free-Choice Profiling study by experienced animal welfare inspectors, based on the same video footage used in the current study. The 12 QBA terms were scored by two different assessor groups consisting of two veterinary science students and four veterinary surgeons (Group 1), and seven farm assurance inspectors (Group 2). The two assessor groups met and received training on different dates, and viewed the same 12 video clips of sheep showing a wide range of behavioural expressions in varying indoor and outdoor situations and housing systems. For each clip assessors scored the sheep expressions on each of the 12 QBA terms using a Visual Analogue Scale. Principal Component Analysis (covariance matrix, no rotation) was used to analyse assessor scores, both for Groups 1 and 2 separately, and together in one all-assessor group. All three analyses identified that over 70% of variance in sheep expression was described by two main components. For the all-assessor analysis the first Principal Component (PC1, 49% of variation) ranged from 'content/relaxed/bright' to 'distressed/dejected/tense', while PC2 (31%) ranged from 'agitated/responsive/anxious' to 'dull/dejected/relaxed'. The 2 Principal Components identified by Groups 1 and 2 when analysed separately, were highly similar. The level of agreement between assessors in each group was evaluated using Kendall's coefficient of concordance (W). This produced W values of 0.83 (PC 1) and 0.84 (PC 2) for the all-assessor group, values of 0.90 (PC 1) and 0.86 (PC 2) for Group 1, and of 0.78 (PC 1) and 0.91 (PC 2) for Group 2. All values were significant at P < 0.001. These results indicate that two trained assessor groups achieved high levels of inter-observer agreement using a list of 12 prefixed QBA terms developed for sheep. This study is the first to investigate QBA as a tool for assessing sheep behavioural expression, and its results support further exploration of the feasibility and validity of applying this method to the assessment of sheep welfare.

Are observer ratings influenced by prescription? A comparison of Free Choice Profiling and Fixed List methods of Qualitative Behavioural Assessment

“…Several prior investigations using the FL method have similarly reported significant interobserver reliability in sheep (Phythian et al, 2013) dairy cattle (Andreasen et al, 2013), pigs (Wemelsfelder and Millard, 2009) and cattle (Sant'Anna and Paranhos da Costa, 2013). Conversely, Bokkers et al (2012) reported low inter-observer reliability between trained and inexperienced observers scoring dairy cattle behaviour.…”

“…This test is used for expressing inter-rater agreement amongst judges who are seeing the same thing, and has been used previously to compare the ratings of observers for other QBA studies (Wemelsfelder and Millard, 2009;Bokkers et al, 2012;Phythian et al, 2013). A Principal Component Analysis (PCA) was carried out for the VAS scores for each individual clip scored by each observer.…”

Navigating the iceberg: reducing the number of parameters within the Welfare Quality® assessment protocol for dairy cows

“…Compared to the free-choice profiling format, where QBA has been associated with physiological welfare indicators in cattle and steers (Stockman et al, 2011 and, and induced emotional states in pigs (Rutherford et al, 2012), QBA as part of the Welfare Quality ® protocol has not been found to be associated with other measures in veal calves (Brscic et al, 2009), or beef cattle (Kirchner et al, 2012). In terms of inter-assessor agreement, the evidence is also mixed for the fixed term format with high levels of inter-observer agreement having been reported in sheep (Phythian et al, 2013), but only 'slight to moderate' agreement for Welfare Quality ® QBA in dairy cows (Bokkers et al, 2012). In addition, QBA has been found to be influenced by time of day (Schwed, 2013), though perhaps this can be understood as relating to changes in the environment, which have been found to be reflected by QBA in veal calves (Brscic et al, 2009) and similarly, by pigs in enriched environments (Mullan et al, 2011).…”

The Relationship Between Coat Color and Aggressive Behaviors in the Domestic Cat

“…However, there is evidence that survey answers are strongly correlated with observed behaviors. For instance, studies of temperament tests in horses, dogs, and sheep in which guardians (and other observers) had previously completed behavior questionnaires showed a high correlation between what the guardians reported and how the dogs or horses responded during the tests (Bennett, Litster, Weng, Walker, & Luescher, 2012;Momozawa et al, 2003Momozawa et al, , 2007Phythiana, Michalopouloua, Duncana, & Wemelsfelder, 2013). Additionally, the questions in this survey were based on the types of observations often performed during feline temperament tests and assessments.…”