Teasing apart quality and validity in systematic reviews: an example from acupuncture trials in chronic neck and back pain

Abstract

The objectives of the study were (1) to carry out a systematic review to assess the analgesic efficacy and the adverse effects of acupuncture compared with placebo for back and neck pain and (2) to develop a new tool, the Oxford Pain Validity Scale (OPVS), to measure validity of findings from randomized controlled trials (RCTs), and to enable ranking of trial findings according to validity within qualitative reviews. Published RCTs (of acupuncture at both traditional and non-traditional points) were identified from systematic searching of bibliographic databases (e.g. MEDLINE) and reference lists of retrieved reports. Pain outcome data were extracted with preference given to standardized outcomes such as pain intensity. Information on adverse effects was also extracted. All included trials were scored using a five-item 0-16 point validity scale (OPVS). The individual RCTs were ranked according to their OPVS score to enable more weight to be placed on the trials of greater validity when drawing an overall conclusion about the efficacy of acupuncture for relieving neck and back pain. Statistical analyses were carried out on the OPVS scores to assess the relationship between trial finding (positive or negative) and validity. Thirteen RCTs met the inclusion criteria. Five trials concluded that acupuncture was effective, and eight concluded that it was not effective for relieving back or neck pain. There was no obvious difference between the findings of trials using traditional and non-traditional points. Using the new OPVS scale, the validity scores of the included trials ranged from 4 to 14. There was no significant relationship between OPVS score and trial finding (positive versus negative). Authors' conclusions did not always agree with their data. We drew our own conclusions (positive/negative) based on the data presented in the reports. Re-analysis using our conclusions showed a significant relationship between OPVS score and trial finding, with higher validity scores associated with negative findings. OPVS is a useful tool for assessing the validity of trials in qualitative reviews. With acupuncture for chronic back and neck pain, we found that the most valid trials tended to be negative. There is no convincing evidence for the analgesic efficacy of acupuncture for back or neck pain.

How objective are systematic reviews? Differences between reviews on complementary medicine

“…The total number of included reviews was 38 ; three acupuncture reviews 14,18,21 contributed to two review sets, since they covered more than one topic. The sample of primary studies varied by more than 25% in fifteen review sets, and by more than 50% in ten.…”

Efficacy and safety of balloon kyphoplasty in the treatment of vertebral compression fractures: a systematic review

“…On the other hand, many of the studies present methodological shortcomings, with known sources of bias-including poor description of the criteria used for patient selection and their lack of homogeneity due to the inclusion of fractures with different characteristics, etiologies, locations, and evolutive stages. Additional sources of bias include small sample size; scant description of concomitant treatments; the combination of kyphoplasty with other treatment measures (making it almost impossible to establish the differential contribution to the outcome of each technique); the lack of standard measures and the subjectivity of some of the outcome variables; the absence of masking in the assessment of the results; the variability and limited duration of follow-up; the lack of identification of losses and their causes; and poor quality in describing the results obtained [7,35,37,49]. Such heterogeneity and limitations moreover make it impossible to establish a definitive estimate of the global effect of BK, since not all studies analyze the same variable of interest, and not all describe the results on a quantitative basis and in a comparable manner.…”

Controlled clinical trials in cancer pain. How controlled should they be? A qualitative systematic review

“…Validity was evaluated using the five item (1 -16) Oxford Pain Validity Scale (OPVS) (Smith et al, 2000). Scoring was performed independently by three reviewers (RFB, CE, EK).…”