Evaluating fidelity to the wraparound service model for youth: Application of item response theory to the Wraparound Fidelity Index.

Abstract

The wraparound process is a mechanism for multi-system planning and care coordination for youth with serious emotional and behavioral problems. Fidelity monitoring is critical to effective implementation of evidence-based practices in children’s mental health, as it helps ensure that complex interventions like wraparound are implemented as intended. The 40-item Wraparound Fidelity Index, version 4 (WFI-4; Bruns, Burchard, Suter, Leverentz-Brady, & Force, 2004) is the most frequently used measure of fidelity to the wraparound process, but analysis of its psychometric properties is insufficient. An item response theory approach, Rasch partial credit models for ordered polytomous data, was used on ratings from 1,234 facilitators, 1,006 caregivers, and 221 team members, focused on 1,478 youths (55% male). Results indicated the WFI-4 measured a unidimensional construct, with little evidence of item bias and good item and model fit. However, the item information curve was skewed, with most people endorsing high-fidelity responses, and several items had duplicative location estimates. A reduced 20-item measure is proposed. Internal reliability estimates for scores from this reduced measure were approximately equivalent to the longer measure. However, both versions would benefit from additional items located in the highest-fidelity area of either version of the scale where scores by greater than half of our sample fall, but only three items are located.

Effectiveness of Wraparound Versus Case Management for Children and Adolescents: Results of a Randomized Study

“…Each item is related to one of the 10 principles of wraparound; e.g., “Does the team evaluate progress toward the goals of the plan at every team meeting?” (Outcome-based). Total scale alpha coefficients of .88 and .92 and test-retest reliabilities of .84 and .88 have been found for the caregiver and facilitator forms, respectively (Bruns et al, 2004, 2008; Pullmann, Bruns, & Sather, 2013). Support for validity is found in correlations with alternative fidelity measures (see below); and ability to discriminate between wraparound and other interventions, (Bruns et al, 2006; 2008; Pullmann et al, 2013).…”

“…To assess fidelity, caregivers and providers (facilitators and case managers) in both groups were interviewed using the Wraparound Fidelity Index, version 4 (WFI-4) (Bruns et al, 2004, 2005, 2008; Pullmann, Bruns, & Sather, 2013). The WFI is a measure of adherence to the principles of the wraparound process.…”

Psychometrics, Reliability, and Validity of a Wraparound Team Observation Measure

“…For the current study, only the WFI-4 caregiver total scores were used as they were the most readily available across these sites, and previous studies have shown the highest variability in caregiver WFI-4 scores compared to other respondents (Pullmann et al, 2013). For the 47 sites that had TOM and WFI-4 data, 918 teams had TOM administrations and WFI-4 interviews were conducted with caregivers on 1,098 teams.…”

“…Specifically, previous research has found that wraparound fidelity measures that rely on staff, parent, and youth report of implementation content and process can be susceptible to limited variation and “ceiling effects” that reduce utility of the tools and compromise psychometrics (Bruns et al, 2004; Pullmann, Bruns, & Sather, 2013). As an alternative or complement to interview or self-report, independent, direct observation methods can provide a rich account of behaviors and interactions of interest that reduces bias from treatment expectancy or overestimation (Aspland & Gardner, 2003; Eames et al, 2008; Webster-Stratton & Hancock, 1998).…”

Construct validity of the Chinese version of WHOQOL-BREF & Disabilities module in 1000 adults with disabilities: An Item Response Theory analysis

“…In particular, IRT analysis can also be used for evaluating construct validity of a scale. Rasch model for dichotomous data [10] and Rasch model for polytomous responses, called Partial Credit Model [11], were usually applied to evaluate dimensional structure and construct validity of a certain scale [12][13][14]. IRT Graded Response Model (GRM) [15], which was especially for the ordered polytomous response, can also be used for construct evaluation and item selection through evaluating the estimated test information and item psychometric parameters [16,17].…”