Are treatment effect assumptions in orthodontic studies overoptimistic?

Abstract

Summary Background At the clinical trial design stage, assumptions regarding the treatment effects to be detected should be appropriate so that the required sample size can be calculated. There is evidence in the medical literature that sample size assumption can be overoptimistic. The aim of this study was to compare the distribution of the assumed effects versus that of the observed effects as a proxy for overoptimistic treatment effect assumptions at the study design stage. Materials and method Systematic reviews (SRs) published between 1 January 2010 and 31 December 2019 containing at least one meta-analysis on continuous outcomes were identified electronically. SR and primary study level characteristics were extracted from the SRs and the individual trials. Details on the sample size calculation process and assumptions and the observed treatment effects were extracted. Results Eighty-five SRs with meta-analysis containing 347 primary trials were included. The median number of SR authors was 5 (interquartile range: 4–7). At the primary study level, the majority were single centre (78.1%), utilized a parallel design (52%), and rated as an unclear/moderate level of risk of bias (34.3%). A sample size was described in only 31.7% (110/347) of studies. From this cohort of 110 studies, in only 37 studies was the assumed clinical difference that the study was designed to detect reported (37/110). The assumed treatment effect was recalculated for the remaining 73 studies (73/110). The one-sided exact signed rank test showed a significant difference between the assumed and observed treatment effects (P < 0.001) suggesting greater values for the assumed effect sizes. Conclusions Careful consideration of the assumptions at the design stage of orthodontic studies are necessary in order to reduce the unreliability of clinical study results and research waste.

Characterizing the orthodontic research literature: 2020

“…Koletsi et al 13 reported that sample sizes were inadequate in 71% of the studies they reviewed. Seehra et al 14 found adequate sample size calculations in only 52% of the articles. In the current study, power tests were performed in 43% of the cases, and management of covariables was observed in 17%.…”

“…In the current study, power tests were performed in 43% of the cases, and management of covariables was observed in 17%. Seehra et al 14 also reported that 52% of studies of treatment outcomes used only single-site data, 23% were prospective, 19% retrospective, and 53% used cohort designs. In the current article, these numbers were 83%, 63%, 27%, and 37%.…”

“…There is literature critiquing various shortcomings in the field. Examples include conflict of interest, 1 retracted articles, 2 conformance with reporting standards such as CONSORT for randomized controlled trials (RCTs) [3][4][5][6][7][8][9] and SRs, 10,11 focus group reporting, 12 power (sample size) estimates, 13,14 cohort studies that test against baseline, 15,16 and country of authorship. 17,18 An exhaustive survey was performed of three journals during the year 2020 to quantify the topics, research methods, analytical approaches, and conventions for describing materials and methods and reporting findings.…”