Preference in Random Assignment: Implications for the Interpretation of Randomized Trials

Abstract

Random assignment to a preferred experimental condition can increase service engagement and enhance outcomes, while assignment to a less-preferred condition can discourage service receipt and limit outcome attainment. We examined randomized trials for one prominent psychiatric rehabilitation intervention, supported employment, to gauge how often assignment preference might have complicated the interpretation of findings. Condition descriptions, and greater early attrition from services-as-usual comparison conditions, suggest that many study enrollees favored assignment to new rapid-job-placement supported employment, but no study took this possibility into account. Reviews of trials in other service fields are needed to determine whether this design problem is widespread. © Springer Science+Business Media, LLC 2009Correspondence to: Cathaleene Macias. NIH Public Access Author ManuscriptAdm Policy Ment Health. Author manuscript; available in PMC 2009 December 20. The validity of research in any field depends on the extent to which studies rule out alternative explanations for findings and provide meaningful explanations of how and why predicted outcomes were attained (e.g., Bickman 1987;Lewin 1943;Shadish et al. 2002;Trist and Sofer 1959). In mental health services research, participants' expectations about the pros and cons of being randomly assigned to each experimental intervention can offer post hoc explanations for study findings that rival the explanations derived from study hypotheses. Unlike most drug studies that can 'blind' participants to their condition assignment, studies that evaluate behavioral or psychosocial interventions typically tell each participant his or her experimental assignment soon after randomization, and being assigned to a non-preferred intervention could be disappointing, or even demoralizing (Shapiro et al. 2002), and thereby reduce participants' interest in services or motivation to pursue service goals (Cook and Campbell 1979;Shadish 2002). On the other hand, if most participants randomly assigned to one experimental condition believe they are fortunate, this condition may have an unfair advantage in outcome comparisons.Reasons for preferring assignment to a particular experimental condition can be idiosyncratic and diverse, but as long as each condition is assigned the same percentage of participants who are pleased or displeased with their condition assignment, then there will be no overall pattern of condition preferences that could explain differences in outcomes. The greater threat to a valid interpretation of findings occurs when most study enrollees share a general preference for random assignment to one particular condition. Greater preference for one experimental condition over another could stem from general impressions of relative service model effectiveness, or from information that is tangential, e.g., program location on a main bus route or in a safer area of town. Even if random assignment distributes service preferences in equal proportions across conditi...

Viability of Using Employment Rates from Randomized Trials as Benchmarks for Supported Employment Program Performance

“…The negative impact of being assigned to a non-preferred program could also account for the higher 39% quarterly employment rate for the full membership of this Massachusetts clubhouse, relative to the 26% quarterly employment rate achieved by the subset of clubhouse members who were in the research trial, because those who were randomly assigned to the clubhouse in spite of their preference for PACT represented a smaller proportion of the full membership than the research subsample. These findings suggest that SE programs that participate in randomized trials will tend to have lower than usual employment rates, unless the comparison condition is substantially less appealing than the study's SE program (Macias et al 2009). …”

Vocational services research: Recommendations for next stage of work

“…For example, Macias et al point out the importance and challenges of addressing the potential for differential initial interest in, and thus attrition from, different treatment assignments within a study with evidence that this may be a confounding factor in some past studies of IPS SE [45]. While it is true that attrition can be a meaningful outcome variable, it also has the potential to be a threat to the internal validity of a study.…”

Does Competitive Work Improve Quality of Life for Adults with Severe Mental Illness? Evidence from a Randomized Trial of Supported Employment

“…This unexpected finding might be explained by another primary difference between experimental programs in most randomized controlled trials: Whereas most supported employment interventions provide only employment services, 15 many comparison programs provide both employment services and opportunities to interact with peers, 16 suggesting that social interaction (e.g., emotional support, praise, camaraderie) may be an essential prerequisite for a positive association between competitive work and quality of life improvement. 12 …”