Size is everything – large amounts of information are needed to overcome random effects in estimating direction and magnitude of treatment effects

Abstract

Variability in patients' response to interventions in pain and other clinical settings is large. Many explanations such as trial methods, environment or culture have been proposed, but this paper sets out to show that the main cause of the variability may be random chance, and that if trials are small their estimate of magnitude of effect may be incorrect, simply because of the random play of chance. This is highly relevant to the questions of 'How large do trials have to be for statistical accuracy?' and 'How large do trials have to be for their results to be clinically valid?' The true underlying control event rate (CER) and experimental event rate (EER) were determined from single-dose acute pain analgesic trials in over 5000 patients. Trial group size required to obtain statistically significant and clinically relevant (0.95 probability of number-needed-to-treat within -/+0.5 of its true value) results were computed using these values. Ten thousand trials using these CER and EER values were simulated using varying group sizes to investigate the variation due to random chance alone. Most common analgesics have EERs in the range 0.4-0.6 and CER of about 0.19. With such efficacy, to have a 90% chance of obtaining a statistically significant result in the correct direction requires group sizes in the range 30-60. For clinical relevance nearly 500 patients are required in each group. Only with an extremely effective drug (EER > 0.8) will we be reasonably sure of obtaining a clinically relevant NNT with commonly used group sizes of around 40 patients per treatment arm. The simulated trials showed substantial variation in CER and EER, with the probability of obtaining the correct values improving as group size increased. We contend that much of the variability in control and experimental event rates is due to random chance alone. Single small trials are unlikely to be correct. If we want to be sure of getting correct (clinically relevant) results in clinical trials we must study more patients. Credible estimates of clinical efficacy are only likely to come from large trials or from pooling multiple trials of conventional (small) size.

Comparison of sustained-release morphine with sustained-release oxycodone in advanced cancer patients

“…Side effects such as dry mouth, somnolence, constipation, pruritus, sensation of empty head, and good acceptance to the study drugs were not different among patients in the present study (Table 3). Nevertheless, group sizes in the range of 30 -60 would be more appropriate for accuracy of side effects (Moore et al, 1998).…”

Opioids added to local anesthetics for single-shot intrathecal anesthesia in patients undergoing minor surgery: A meta-analysis of randomized trials

“…To overcome random play of chance on estimation of treatment effects, we excluded studies with fewer than 10 participants per group [48,59].…”

Efficacy and safety of balloon kyphoplasty in the treatment of vertebral compression fractures: a systematic review

“…On the other hand, many of the studies present methodological shortcomings, with known sources of bias-including poor description of the criteria used for patient selection and their lack of homogeneity due to the inclusion of fractures with different characteristics, etiologies, locations, and evolutive stages. Additional sources of bias include small sample size; scant description of concomitant treatments; the combination of kyphoplasty with other treatment measures (making it almost impossible to establish the differential contribution to the outcome of each technique); the lack of standard measures and the subjectivity of some of the outcome variables; the absence of masking in the assessment of the results; the variability and limited duration of follow-up; the lack of identification of losses and their causes; and poor quality in describing the results obtained [7,35,37,49]. Such heterogeneity and limitations moreover make it impossible to establish a definitive estimate of the global effect of BK, since not all studies analyze the same variable of interest, and not all describe the results on a quantitative basis and in a comparable manner.…”