Are inspectors’ assessments reliable? Ratings of NHS acute hospital trust services in England

Abstract

The credibility of a regulator could be threatened if stakeholders perceive that assessments of performance made by its inspectors are unreliable. Yet there is little published research on the reliability of inspectors’ assessments of health care organizations’ services.ObjectivesWe investigated the inter-rater reliability of assessments made by inspectors inspecting acute hospitals in England during the piloting of a new regulatory model implemented by the Care Quality Commission (CQC) during 2013 and 2014. Multi-professional teams of inspectors rated service provision on a four-point scale for each of five domains: safety; effectiveness; caring; responsiveness; and leadership.MethodsIn an online survey, we asked individual inspectors to assign a domain and a rating to each of 10 vignettes of service information extracted from CQC inspection reports. We used these data to simulate the ratings that might be produced by teams of inspectors. We also observed inspection teams in action, and interviewed inspectors and staff from hospitals that had been inspected.ResultsLevels of agreement varied substantially from vignette to vignette. Characteristics such as professional background explained only a very small part of the variation. Overall, agreement was higher on ratings than on domains, and for groups of inspectors compared with individual inspectors. A number of potential causes of disagreement were identified, such as differences regarding the weight that should be given to contextual factors and general uncertainty about interpreting the rating and domain categories.ConclusionGroups of inspectors produced more reliable assessments than individual inspectors, and there is evidence to support the utility of appropriate discussions between inspectors in improving reliability. The reliability of domain allocations was lower than for ratings. It is important to define categories and rating levels clearly, and to train inspectors in their use. Further research is needed to replicate these results now that the model has been fully implemented, and to understand better the impact that inspector uncertainty and disagreement may have on published CQC ratings.

Strategy and practise of external inspection in healthcare services – a Norwegian comparative case study

“…The Dutch study pointed out that not one single instrument could evaluate the complexity and multidimensional work of inspectors [34]. Relating these findings to our study, there seems to be evidence to support establishing arenas that allow for appropriate discussions between inspectors to improve reliability of their assessment but also to evaluate their own practice to ensure more equal weighting of factors in their assessment [15].…”

“…A recent study focusing on the reliability of inspectors' assessment among inspectors inspecting of English acute hospitals, showed that the level of agreement varied between inspectors and groups of inspectors made more reliable assessment in inspections compared to what individual inspectors did. The disagreement between inspectors partly related to diverse weight given to contextual factors in the assessment [15]. Inspector judgement has also been studied in the Netherlands.…”

“…Since external inspection systems, such as the NBHS and the County Governors in Norway, almost always involve inspectors who assess organizational or professional performance against standards, the credibility of a regulatory body can be threatened if inspectors' assessments of performance are unreliable [15]. A recent study focusing on the reliability of inspectors' assessment among inspectors inspecting of English acute hospitals, showed that the level of agreement varied between inspectors and groups of inspectors made more reliable assessment in inspections compared to what individual inspectors did.…”

“…It can be difficult to balance different roles of law enforcement and guidance [14]. There is an on-going debate on how to compose inspection teams, since team composition will have implications for inspection quality [15]. Differences in practises of external inspection disclose how standards and requirement should be set as an interpretation of legislation.…”

Exploring hospital certification processes from the certification body’s perspective - a qualitative study

“…In external assessments systems, such as ISO certification, the auditors' experience, selection of auditors, training, support and motivation may influence the performance, style, and reliability of the auditing (assessment) practices [6,8,[24][25][26]. The approach and methods that auditors use in their assessments and verification processes, such as role repertoire, auditor's conduct (e.g., inspection or guidance) and assessment practice need further exploration [15,25,[27][28][29]. Exploring these matters may identify elements that can be beneficial for training, development and consistency of certification programs both within and between certification bodies, and for further research.…”