Do acupuncture trials have lower risk of bias over the last five decades? A methodological study of 4 715 randomized controlled trials

Abstract

Objective To evaluate the change of the risk of bias (RoB) of acupuncture randomized controlled trials (RCTs) in the past five decades. Methods Multiple databases were searched. We included RCTs identified from systematic reviews (SRs) on acupuncture. General characteristics and RoB judgment for each domain were extracted. The proportions of RCTs at high and unclear RoB were calculated and the changes were examined by the Mann-Kendall test. Results We included 368 SRs including 4 715 RCTs. The rates of RCTs at unclear RoB were the highest in allocation concealment (63%), and at the lowest in incomplete outcome data (35%); in the last five decades, statistically significant reductions were found for random sequence generation (P < 0.001) and selective reporting (P = 0.01), and increases for blinding of participants and personnel (P < 0.001), blinding of outcome assessment (P < 0.001) and incomplete outcome data (P = 0.04). For the proportions of RCTs at high RoB, blinding of participants and personnel (47%) and blinding of outcome assessment (35%) were the poorest domains; there were no significant differences in changes for all domains. Conclusions Although improvements concerning unclear risk were observed for random sequence generation and selective reporting, major issues remain for allocation concealment and blinding. It is imperative to use valid randomization, specify how it is conducted, and try to test for selection bias and the success of masking by using the Berger Exner test.

Trends and quality of randomized controlled trials on acupuncture conducted in Japan by decade from the 1960s to the 2010s: a systematic review

“…One study assessed the trends in the RoB in RCTs on acupuncture in the last five decades, based on a search of the PubMed, Embase and three Chinese databases [ 2 ]. According to data derived from 368 systematic reviews, including 4,715 RCTs on acupuncture, there was a statistically significant reduction in the proportion of RCTs with unclear risk for random sequence generation and selective reporting, and a significant uptick in the trend of RCTs with unclear risk for blinding of participants and personnel, blinding of outcome assessment, and incomplete outcome data.…”

“…While the domains of the first version of the RoB assessment tool focused on a specific method (e.g., “blinding of participants”) [ 6 ], those of the updated tool (i.e., RoB 2) explicitly refer to bias itself (e.g., “bias due to deviations from intended interventions”) [ 12 ]. We used the first version of the Cochrane RoB tool [ 6 ] to compare our data with RCTs conducted in other countries based on a similar study [ 2 ].…”

“…However, the quality of the RCTs is not always high, cautioning against interpreting the results of RCTs with unwarranted confidence [ 1 ]. There is scope for improvement in the overall quality of RCTs on acupuncture with respect to the risk of bias (RoB) [ 2 ]. This is also true for clinical trials on acupuncture conducted in Japan at least before 2008, since their quality is not necessarily high [ 3 , 4 ].…”

Assessment of reporting quality in randomized controlled trials of acupuncture for labor pain

“…Most of studies conducted in China applied no-treatment or drug controls, while in Western studies, up to 36% used sham acupuncture controls. Control groups for acupuncture should also be rigorously described to facilitate other researchers seeking to replicate acupuncture control interventions, assess the internal validity of clinical outcomes, and promote the use of effective interventions in clinical practice ( 39 , 40 ). This is particularly important for empirical medicine like acupuncture, as different acupuncturists treating different study subjects may affect the generalizability of trial results.…”

Journal Club

“…Nevertheless, compliance and adherence to these formal quality requirements by the study trialists appears to be still not high enough. This is shown in particular by publications from Ma B. et al 2016 and Long Y. et al 2020, which examined the study quality of Chinese and other international randomised clinical trials (RCTs) on acupuncture and moxibustion over the past decades [19,20]. Ma Bin examined 1978 RCT studies published in Chinese medical journals between 1997 and 2012 and compared the quality of the studies in the periods before and after the introduction of CONSORT and STRICTA in China with regard to the fulfilment of the required items of the methodological quality domains [19].…”