How well do juvenile risk assessments measure factors to target in treatment? Examining construct validity.

Abstract

There has been a surge of interest in using 1 type of risk assessment instrument to tailor treatment to juveniles to reduce recidivism. Unlike prediction-oriented instruments, these reduction-oriented instruments explicitly measure variable risk factors as "needs" to be addressed in treatment. There is little evidence, however, that the instruments accurately measure specific risk factors. Based on a sample of 237 serious juvenile offenders (M = 18, SD = 1.6), we tested whether California Youth Assessment Inventory (CA-YASI) scores validly assess the risk factors they purport to assess. Youth were assessed by practitioners with good interrater reliability on the CA-YASI, and by research staff on a battery of validated, multimethod criterion measures of target constructs. We meta-analytically tested whether each CA-YASI risk domain score (e.g., Attitudes) related more strongly to scores on convergent measures of theoretically similar constructs (e.g., criminal thinking styles) than to scores on discriminant measures of theoretically distinct constructs (e.g., intelligence, somatization, and pubertal status). CA-YASI risk domain scores with the strongest validity support were those that assess criminal history. The only variable CA-YASI risk domain score that correlated more strongly with convergent (Zr = .35) than discriminant (Zr = .07) measures was Substance Use. There was little support for the construct validity of the remaining 6 variable CA-YASI risk domains-including those that ostensibly assess strong risk factors (e.g., "Attitudes," "Social Influence"). Our findings emphasize the need to test the construct validity of reduction-oriented instruments-and refine instruments to precisely measure their targets so they can truly inform risk reduction. (PsycINFO Database Record

Assessing Risk in North Dakota Juvenile Probation: A Preliminary Examination of the Predictive Validity of the Youth Assessment and Screening Instrument

“…Five other states utilize YASI statewide including Hawaii, Illinois, New York, Vermont, and Virginia. Note, though not implemented statewide, the California Division of Juvenile Justice has adopted the use of a modified YASI instrument known as the CA-YASI (Kennealy et al, 2017; Skeem et al, 2017). Finally, YASI is also used in the UK and Canada (Jones et al, 2016).…”

“…This relationship has since continued for nearly two decades. Though reportedly subject to an initial norming process, and further evaluations completed elsewhere (Jones et al, 2014, 2016; Kennealy et al, 2017; Orbis Partners, 2007; Scott et al, 2019; Skeem et al, 2017), the instrument’s use has gone untested within this predominantly rural Midwestern state.…”

“…This exploratory study represents only the second independent validation of this instrument (Scott et al, 2019), and the first to do so within the United States (with exception to a substantially modified version of the instrument implemented in California, see Skeem et al, 2013). Scott et al (2019) noted that prior published assessments of predictive validity for this instrument have otherwise come solely from the tool developers (Jones et al, 2016; Orbis Partners, 2007).…”

Lost in translation: “Risks,” “needs,” and “evidence” in implementing the First Step Act

“…In practice, when risk assessment is divorced from needs assessment, the former is often subject to empirical scrutiny while “anything goes” for the latter (see Skeem, Barnoski, Latessa, Robinson, & Tjaden, 2013). In reality, the core empirical requirement for a valid risk assessment (i.e., utility in predicting recidivism) is much easier to satisfy than that of a valid needs assessment (i.e., utility in measuring specific constructs that, when changed, reduce recidivism; see Monahan & Skeem, 2014; Skeem, Kennealy, Tatar, Hernandez, & Keith, 2017).…”

“…Almost no studies have tested the construct validity of “needs” assessments to determine whether given scales measure what they say they measure. In an exception, Skeem et al (2017) tested the correspondence between scales on a widely marketed juvenile risk‐needs assessment (e.g., “social‐cognitive skills”) and existing, well‐validated measures of those constructs (e.g., the Psychological Inventory of Criminal Thinking Styles; Walters, 1990). The authors found that scales labeled as violence‐aggression, social‐cognitive skills, social influences, education/employment, and family did not translate to treatment‐relevant target constructs of anger/hostility, executive function deficits, antisocial peer influence, poor school/work motivation, or problematic parental discipline and monitoring, respectively.…”