Bias through selective inclusion and attrition: Representativeness when comparing provider performance with routine outcome monitoring data

Abstract

Background Observational research based on routine outcome monitoring is prone to missing data, and outcomes can be biased due to selective inclusion at baseline or selective attrition at posttest. As patients with complete data may not be representative of all patients of a provider, missing data may bias results, especially when missingness is not random but systematic. Methods The present study establishes clinical and demographic patient variables relevant for representativeness of the outcome information. It applies strategies to estimate sample selection bias (weighting by inclusion propensity) and selective attrition bias (multiple imputation based on multilevel regression analysis) and estimates the extent of their impact on an index of provider performance. The association between estimated bias and response rate is also investigated. Results Provider‐based analyses showed that in current practice, the effect of selective inclusion was minimal, but attrition had a more substantial effect, biasing results in both directions: overstating and understating performance. For 22% of the providers, attrition bias was estimated to be in excess of 0.05 ES. Bias was associated with overall response rate (r = .50). When selective inclusion and attrition bring providers' response below 50%, it is more likely that selection bias increased beyond a critical level, and conclusions on the comparative performance of such providers may be misleading. Conclusions Estimates of provider performance were biased by selection, especially by missing data at posttest. Results on the extent and direction of bias and minimal requirements for response rates to arrive at unbiased performance indicators are discussed.

Comparing the effectiveness and predictors of cognitive behavioural therapy-enhanced between patients with various eating disorder diagnoses: a naturalistic study

“…In addition, efforts to ensure patients complete treatment may be more limited in effectiveness studies than in efficacy studies (Byrne et al, 2011). In addition, as more than 50% of all patients showed compliance regarding the assessments, the results can be generalised to other eating disorder patients (de Beurs et al, 2019). Motivational sessions prior to commencing treatment might reduce drop-out and enhance treatment completion among patients suffering from BN.…”

Feasibility of a Mobile Health App for Routine Outcome Monitoring and Feedback in SMART Recovery Mutual Support Groups: Stage 1 Mixed Methods Pilot Study

“…A few studies have examined ROM implementation report data on engagement and attrition rates [ 8 , 25 , 28 , 110 , 111 ]. Compared with recent data from the Netherlands, however, the proportion of participants using SMART Track to input ROM data is largely comparable, and in some cases, greater than traditional clinician-completed methods [ 112 ]. A 50% response rate has been recently suggested as an acceptable benchmark for ROM data and is likely sufficient to protect against bias and yield valid information about patient progress (see study by de Beurs et al [ 112 ] for a discussion).…”

“…Compared with recent data from the Netherlands, however, the proportion of participants using SMART Track to input ROM data is largely comparable, and in some cases, greater than traditional clinician-completed methods [ 112 ]. A 50% response rate has been recently suggested as an acceptable benchmark for ROM data and is likely sufficient to protect against bias and yield valid information about patient progress (see study by de Beurs et al [ 112 ] for a discussion). In this study, this benchmark was achieved each week across the first month of data collection, with 83% (60/72), 63% (45/72), 60% (43/72), and 53% (38/72) of the study participants completing at least one of the ROM instruments across the first 4 weeks of app use.…”

Impact of COVID‐19 social distancing measures on routine mental health care provision and treatment outcome for common mental disorders in the Netherlands

“…Only treatments with complete pretest and post‐test data for the first year were included, about 55% of all treatments that had been started, 43.8% of the S‐MHC and 70.3% of the B‐MHC treatments. We compared fully assessed treatments to treatments with incomplete data on pretest patient characteristics to check for selective ROM non‐response (de Beurs, Warmerdam, & Twisk, 2019 ).…”