Adherence to Structured Risk Assessment Guidelines: Development and Preliminary Evaluation of an Adherence Scale for the START:AV

Abstract

Risk assessment instruments are widely used to predict risk of adverse outcomes, such as violence or victimization, and to allocate resources for managing these risks among individuals involved in criminal justice and forensic mental health services. For risk assessment instruments to reach their full potential, they must be implemented with fidelity. A lack of information on administration fidelity hinders transparency about the implementation quality, as well as the interpretation of negative or inconclusive findings from predictive validity studies. The present study focuses on adherence, a dimension of fidelity. Adherence denotes the extent to which the risk assessment is completed according to the instrument's guidelines. We developed an adherence measure, tailored to the Short-Term Assessment of Risk and Treatability: Adolescent Version (START:AV), an evidence-based risk assessment instrument for adolescents. With the START:AV Adherence Rating Scale, we explored the degree to which 11 key features of the instrument were adhered to in 306 START:AVs forms, completed by 17 different evaluators in a Dutch residential youth care facility over a two-year period. Good to excellent interrater reliability was found for all adherence items. We identified differences in adherence scores on the various START:AV features, as well as significant improvement in adherence for those who attended a START:AV refresher workshop. Outcomes of risk assessment instruments potentially impact decision-making, for example, whether a youth's secure placement should be extended. Therefore, we recommend fidelity monitoring to ensure the risk assessment practice was delivered as intended.

Risk assessments for violence and reoffending: Implementation and impact on risk management.

“…In one study, assessors initially left an average of three risk estimates blank or missing per assessment, but missing estimates decreased after this finding was identified and discussed with assessors (Desmarais et al, 2012). Similarly, De Beuf, de Vogel, and de Ruiter (2020) found that scores on an 11-item adherence tool significantly increased after assessors attended a refresher training.…”

“…Although Proctor et al’s (2011) taxonomy provides a starting point for understanding what to measure (e.g., acceptability), as a next step, researchers need to develop instruments that map onto these outcomes. In a recent study, De Beuf et al (2020) created a fidelity tool with good interrater reliability. However, formal measures of other implementation outcomes, such as appropriateness and feasibility, are lacking.…”

Taking “the boss” into the real world: Field interrater reliability of the Short‐Term Assessment of Risk and Treatability: Adolescent Version

“…In an early field study with the HCR‐20, Belfrage (1998) actually excluded an evaluator from the reliability analyses, because this person scored significantly differently from the other clinicians, arguably due to a gap of 6 months between the training and the risk assessments. That said, in the present setting, some evaluators participated in a refresher workshop that took place halfway through the data collection period (De Beuf et al, 2020). This may have reduced rater drift, although the sample was too small to explore this further.…”

“…In the intervention literature, researchers have developed checklists to assess implementation fidelity of a practice of interest. Likewise, we developed a scale to assess the level of completion of START:AV rating forms: the START:AV Adherence Rating Scale (STARS; De Beuf, de Vogel, & de Ruiter, 2020). This scale codes how complete the rating forms are, but does not investigate the quality of the ratings as measured by the descriptors listed in the user guide.…”