Are Comparisons of Consumer Satisfaction With Providers Biased by Nonresponse or Case-Mix Differences?

Abstract

Objective To examine how consumer satisfaction ratings differ between mental health providers and to determine if comparison of ratings between providers is biased by differences in survey response rates or characteristics of consumers served. Methods Secondary analysis of routinely mailed consumer satisfaction surveys in a mixed-model prepaid health plan. Satisfaction survey data were linked to computerized record data regarding consumers’ demographic (age, sex, type of insurance coverage) and clinical (primary diagnosis, initial vs. return visit) characteristics. Statistical models examined both probability of returning the mailed satisfaction survey and (among those returning surveys) probability of giving an “Excellent” satisfaction rating. Variability in consumer characteristics was decomposed into within-provider effects and between-provider effects. Results Overall response rate was 33.8%, and 49.9% of those responding reported “Excellent” satisfaction. Neither response rate nor satisfaction rating was related to primary diagnosis. Within the practices of individual providers, both response rate and receiving an “Excellent” rating were significantly associated with female sex, older age, longer enrollment in the health plan, and making a return (vs. initial) visit. Analyses of between-provider effects, however, found that only having a higher proportion of return visitors was significantly associated with higher response rates and higher satisfaction ratings. Conclusions There is little evidence that differences in response rate or differences in consumers served bias comparison of satisfaction ratings between mental health providers. Bias might be greater in a setting with more heterogeneous consumers or providers. Returning consumers give higher ratings than first-time visitors, and analyses of satisfaction ratings may need to account for this difference. Extremely high or low ratings should be interpreted cautiously, especially for providers with a small number of surveys.

Individualization and Quality Improvement: Two New Scales to Complement Measurement of Program Fidelity

“…Intensive procedures, such as audiotaping or videotaping sessions, often used in psychotherapy research studies, are simply impractical for routine quality assurance purposes, particularly when numerous programs need to be assessed in a large system of care. Client ratings of satisfaction with services have a place, but information imbalance, demand characteristics of survey instruments, and response bias all compromise satisfaction ratings as valid measures of clinical quality (Simon et al 2009). Moreover, client satisfaction ratings provide a different perspective than objective indicators (Druss et al 1999).…”

Directions for Future Patient-Centered and Comparative Effectiveness Research for People With Serious Mental Illness

“…Simon et al 149 examined whether response bias might account for the frequently reported differences between consumers and providers. They found that the rate of response and type of consumer served did not account for the differences, suggesting that these are real divergences that deserve continued attention.…”

“…A great deal of research has led to the development of treatment algorithms and practice guidelines, 149,322,[439][440][441][442] yet studies have shown that there are continuing deficiencies in the quality of medication prescribing, 149 suggesting that clarified recommendations are insufficient to improve medication-based treatment of serious mental illnesses. 443 Research on consumer perspectives regarding medication guidelines could shed light on failed implementation and lack of adherence, 444-446 but we know very little about what individuals with serious mental illnesses want with respect to their medications.…”

“…62 Experts have identified factors associated with nonadherence, including believing medications are no longer needed, poor insight, distress associated with specific side effects (eg, sedation among people with bipolar disorder, or weight gain among people with schizophrenia), 445 general fear of medication side effects, and inadequate efficacy with persistent symptoms. 447 As a potential solution to this problem, researchers have tested supplementing medication with psychosocial interventions, including CBT, Family Based Services, 149,448 Cognitive Adaptation Training, and ACT, 446 and have found these to increase adherence 446,449 while simultaneously addressing functional symptoms not managed by medications. 446,450,451 However, to our knowledge, research addressing consumer perspectives on medication adherence and how it relates to psychosocial services is largely absent and much needed.…”

“…In another study, however, individuals diagnosed with recent-onset schizophrenia had similar relapse rates regardless of study arm, but those characterized as having multiple episodes in the targeted treatment arms had much higher relapse and rehospitalization rates than those in the maintenance arm. 458 The most recently published treatment recommendations from the Schizophrenia Patient Outcomes Research Team 149 do not endorse a targeted treatment approach, but concede that for select individuals diagnosed with recent-onset schizophrenia who refuse continuous maintenance pharmacotherapy, or for whom some other contraindication exists, targeted treatment is an alternative. They counsel, however, that it should be pursued only when the individual can be monitored closely enough that the targeted intervention can be initiated at the earliest signs of clinical exacerbation.…”