Randomized Controlled Trials : A Gold Standard or Gold Plated?

Abstract

R andomized controlled or clinical trials (RCTs) have been taking on increasing importance, especially outside of the medical field. 1 The number of RCTs is increasing as well as the number of areas in which they are conducted (Bloom, 2008; Boruch, Weisburd, Turner, Karpyn, & Littell, 2009). Moreover, these designs are being recommended and privileged over other designs by prestigious research organizations (e.g., Shavelson & Towne, 2002). In addition, several U.S. federal agencies deemed the RCT as the gold standard that should be used not only in considering the funding of research and evaluation but also in initiating and terminating programs (Brass, Nunez-Neto, & Williams, 2006). However, over the last several years, there has been considerable debate about whether RCTs should be considered the ultimate standard design (Cook & Payne, 2002; Maxwell, 2004). While most would argue that the RCT is a powerful research design, many debate whether it should be labeled as the gold standard for research trying to determine causality. Dissenters of this design as the model for research cite issues of appropriateness, ethics, feasibility, and cost, arguing that other methods can answer causal questions equally well. Most claim that RCTs are more appropriate for medical and basic science investigations, i AUTHORS' NOTE: We are not the first to use metals to describe standards. Both Rossi (1987) and Berk (2005) have used similar terms.

Examining Mindfulness Training for Teachers: Theoretical and Methodological Extensions of Intervention Effectiveness

“…Unfortunately, the RCT design's strong reputation makes it easy to overlook the many factors that can threaten a study once it is underway. Factors such as a study's design (e.g., the comparison group used, the sample size collected), the way in which an intervention is implemented, and the choices researchers make in analyzing data, can all influence conclusions about a program's effectiveness (Bickman & Reich, 2015;Durlak, 2010;Mark & Lenz-Watson, 2011;Wilson & Lipsey, 2001). The factors discussed here are those that can threaten an RCT study's integrity and limit its strengths, however, there are also limitations to the design itself (Skinner et al, 2019).…”

“…Analytic Decisions Bickman & Reich, 2015;Bloom, 2005;Durlak & DuPre, 2008;Enders, 2010;Graham, 2009;Mark & Lenz-Watson, 2011;Miller, 2007;Rhoads, 2011;What Works Clearinghouse, 2017 to elucidate unique active ingredients that might be driving changes due to intervention participation, for instance, asking, "is it relaxation or uniquely mindfulness practices that cause teachers to feel less stress through mindfulness training? ".…”

“…Attrition can bias intervention results in two ways: (1) the general loss of participants reduces the study's power to detect significant intervention effects, as is described above in the sample size section above, and (2) if the individuals who leave the study prematurely are systematically different in some way from those who stay, then the conclusions made about the intervention's effectiveness can be biased to no longer generalize to the entire sample population (Bickman & Reich, 2015). This latter situation may occur most often in intervention studies if the demands of the treatment itself seem unmanageable to a certain subset of individuals.…”

“…At the same time, however, there are numerous ways, that are less often discussed, in which the RCT design's ability to make strong causal inferences can be threatened. Factors such as insufficient recruitment of a large sample, incomplete randomization of participants to groups (where some participants are randomized but others are not), participant attrition, and low-quality implementation of the intervention can all threaten the foundation in which the RCT design is built upon (Bickman & Reich, 2015;Wilson & Lipsey, 2001). However, the RCT methodology's strong reputation as the "gold standard" research design often makes it easy to overlook these many ways in which the design can be vulnerable.…”

Neighbor Perceptions of Psychiatric Supportive Housing: The Role of Knowledge, Attitudes, and Behaviors

“…That is, observational studies of the same topic never found opposite results, whereas RCTs occasionally did. In addition, RCTs are not immune to threats to internal validity, including experimenter effects and attrition; and may be severely lacking in external validity (Bickman & Reich, 2009). It is also important to note that descriptive and correlational studies are often of more value in understanding complex interactions in localized areas.…”