Experimental studies to improve the reliability and validity of regulatory judgments on health care in the Netherlands: a randomized controlled trial and before and after case study

Abstract

Rationale, aims and objectivesWe examined the effect of two interventions on both the reliability and validity of regulatory judgments: adjusting the regulatory instrument and attending a consensus meeting.MethodWe adjusted the regulatory instrument. With a randomized controlled trial (RCT) we examined the effect of the adjustments we made to the instrument. In the consensus meeting inspectors discussed cases and had to reach consensus about the order of the cases. We used a before and after case study to assess the effect of the consensus meeting. We compared the judgments assigned in the RCT with the unadjusted instrument with the judgments assigned with the unadjusted instrument after the consensus meeting. Moreover we explored the effect of increasing the number of inspectors per regulatory visit based on the estimates of the two interventions.ResultsThe consensus meeting improved the agreement between inspectors; the variance between inspectors was smallest (0.03) and the reliability coefficient was highest (0.59). Validity is assessed by examining the relation between the assigned judgments and the corporate standard and expressed by a correlation coefficient. This coefficient was highest after the consensus meeting (0.48). Adjustment of the instrument did not increase reliability and validity coefficients.ConclusionsParticipating in a consensus meeting improved reliability and validity. Increasing the number of inspectors resulted in both higher reliability and validity values. Organizing consensus meetings and increasing the number of inspectors per regulatory visit seem to be valuable interventions for improving regulatory judgments.

Enhancing responsiveness and consistency: Comparing the collective use of discretion and discretionary room at inspectorates inEngland and theNetherlands

“…Because inspectors are in the position to make an impact, it is important that they do not follow their personal “logic of appropriateness” (May & Wood ; March & Olsen ; Tuijn et al . ). Both inspectorates had set rules to stimulate consistency.…”

“…Other scholars have pointed out that discussion and reaching consensus can increase the consistency of judgments (Pires ; Tuijn et al . ). Second, cooperative work performed within collective discretionary room opens up the possibility of adjusting and redefining goals, strategies, and tools, making them more practicable and effective.…”

“…To do this, inspectors need to act consistently on the one hand (Tuijn et al . ; OECD ; May & Wood ), and be responsive to the specific case on the other (Bardach & Kagan ).…”

Are inspectors’ assessments reliable? Ratings of NHS acute hospital trust services in England

“…5 The most recent study identified various potential sources of variation in assessments of nursing home care; some concerning the instrument itself and others related to differing inspector perspectives on regulation and inspection. 6 Agreement might be improved by prior participation in a consensus meeting and by increasing the number of inspectors.…”

The impact of national accreditation reform on survey reliability: a 2‐year investigation of survey coordinators’ perspectives

“…Professional cohesiveness, a shared understanding and a free exchange of views are factors that contribute to the reliability of judgements by health care accreditation survey teams and their members [1–3]. However, surveyor diversity and programme structure are also necessary to avoid groups falling prey to group‐think processes and becoming captured in erroneous ways of thinking [4,5]. Periods of change, such as the revision of existing standards, or episodes of significant reform, such as the introduction of completely new sets of standards, are difficult to negotiate and can challenge reliability and programme credibility [6].…”