Large variation existed in standardized mean difference estimates using different calculation methods in clinical trials

Luo, Yan; Funada, Satoshi; Yoshida, Katsukuni; Noma, Hisashi; Sahker, Ethan; Furukawa, Toshi A

doi:10.1016/j.jclinepi.2022.05.023

Cited by 9 publications

(8 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In (Luo et al, 2022). For instance, without prespecifying the calculation method, researchers may compute S.M.D.s using different methods, and select the largest one to report.…”

Section: To the Editormentioning

confidence: 99%

“…estimates can vary substantially, researchers should be careful in reporting them and readers should be mindful how the reported S.M.D.s were calculated. As it is still hard to recommend a single calculation method that should be used universally for now (Luo et al, 2022), it is desirable for researchers to report their calculation methods in detail to increase transparency and reproducibility.…”

Section: To the Editormentioning

confidence: 99%

“…converted from model statistics). The s.m.d .s estimated by these methods are substantially different from one another, raising potential problems regarding reproducibility, selective reporting, and proper interpretation of how large the effect is (Luo et al, 2022). For instance, without prespecifying the calculation method, researchers may compute s.m.d.…”

Section: To the Editormentioning

confidence: 99%

“…s were calculated. As it is still hard to recommend a single calculation method that should be used universally for now (Luo et al, 2022), it is desirable for researchers to report their calculation methods in detail to increase transparency and reproducibility.…”

Section: To the Editormentioning

confidence: 99%

See 3 more Smart Citations

Effect size calculation needs to be specified with details: comment on Ying et al.

Luo

Furukawa

2022

Psychol. Med.

Self Cite

View full text Add to dashboard Cite

“…In (Luo et al, 2022). For instance, without prespecifying the calculation method, researchers may compute S.M.D.s using different methods, and select the largest one to report.…”

Section: To the Editormentioning

confidence: 99%

Section: To the Editormentioning

confidence: 99%

Section: To the Editormentioning

confidence: 99%

Section: To the Editormentioning

confidence: 99%

See 2 more Smart Citations

Effect size calculation needs to be specified with details: comment on Ying et al.

Luo

Furukawa

2022

Psychol. Med.

Self Cite

View full text Add to dashboard Cite

“…However, many reports tend to oversimplify interpretations and provide insufficient details in this regard. 19 In addition, the commonly used SMD is criticized because of dividing estimates by sample SDs, which are themselves subject to sampling errors. 20 More importantly, the SMD exposes the findings to various types of distortion, as studies can differ significantly in patient population inclusion criteria, leading to concerns about generalizability.…”

Section: Introductionmentioning

confidence: 99%

Comparisons of the mean differences and standardized mean differences for continuous outcome measures on the same scale

Jing,

Lin

2024

JBI Evidence Synthesis

View full text Add to dashboard Cite

When conducting systematic reviews and meta-analyses of continuous outcomes, the mean differences (MDs) and standardized mean differences (SMDs) are 2 commonly used choices for effect measures. The SMDs are motivated by scenarios where studies collected in a systematic review do not report the continuous measures on the same scale. The standardization process transfers the MDs to be unit-free measures that can be synthesized across studies. As such, some evidence synthesis researchers tend to prefer the SMD over the MD. However, other researchers have concerns about the interpretability of the SMD. The standardization process could also yield additional heterogeneity between studies. In this paper, we use simulation studies to illustrate that, in a scenario where the continuous measures are on the same scale, the SMD could have considerably poorer performance compared with the MD in some cases. The simulations compare the MD and SMD in various settings, including cases where the normality assumption of continuous measures does not hold. We conclude that although the SMD remains useful for evidence synthesis of continuous measures on different scales, the SMD could have substantially greater biases, greater mean squared errors, and lower coverage probabilities of CIs than the MD. The MD is generally more robust to the violation of the normality assumption for continuous measures. In scenarios where continuous measures are inherently comparable or can be transformed to a common scale, the MD is the preferred choice for an effect measure.

show abstract

Combining endpoint and change data did not affect the summary standardised mean difference in pairwise and network meta‐analyses: An empirical study in depression

Ostinelli,

Efthimiou,

Luo

et al. 2024

Research Synthesis Methods

View full text Add to dashboard Cite

When studies use different scales to measure continuous outcomes, standardised mean differences (SMD) are required to meta‐analyse the data. However, outcomes are often reported as endpoint or change from baseline scores. Combining corresponding SMDs can be problematic and available guidance advises against this practice. We aimed to examine the impact of combining the two types of SMD in meta‐analyses of depression severity. We used individual participant data on pharmacological interventions (89 studies, 27,409 participants) and internet‐delivered cognitive behavioural therapy (iCBT; 61 studies, 13,687 participants) for depression to compare endpoint and change from baseline SMDs at the study level. Next, we performed pairwise (PWMA) and network meta‐analyses (NMA) using endpoint SMDs, change from baseline SMDs, or a mixture of the two. Study‐specific SMDs calculated from endpoint and change from baseline data were largely similar, although for iCBT interventions 25% of the studies at 3 months were associated with important differences between study‐specific SMDs (median 0.01, IQR −0.10, 0.13) especially in smaller trials with baseline imbalances. However, when pooled, the differences between endpoint and change SMDs were negligible. Pooling only the more favourable of the two SMDs did not materially affect meta‐analyses, resulting in differences of pooled SMDs up to 0.05 and 0.13 in the pharmacological and iCBT datasets, respectively. Our findings have implications for meta‐analyses in depression, where we showed that the choice between endpoint and change scores for estimating SMDs had immaterial impact on summary meta‐analytic estimates. Future studies should replicate and extend our analyses to fields other than depression.

show abstract

Large variation existed in standardized mean difference estimates using different calculation methods in clinical trials

Cited by 9 publications

References 14 publications

Effect size calculation needs to be specified with details: comment on Ying et al.

Effect size calculation needs to be specified with details: comment on Ying et al.

Comparisons of the mean differences and standardized mean differences for continuous outcome measures on the same scale

Combining endpoint and change data did not affect the summary standardised mean difference in pairwise and network meta‐analyses: An empirical study in depression

Contact Info

Product

Resources

About