Informing power and sample size calculations when using inverse probability of treatment weighting using the propensity score

Abstract

Propensity score weighting is increasingly being used in observational studies to estimate the effects of treatments. The use of such weights induces a within‐person homogeneity in outcomes that must be accounted for when estimating the variance of the estimated treatment effect. Knowledge of the variance inflation factor (VIF), which describes the extent to which the effective sample size has been reduced by weighting, allows for conducting sample size and power calculations for observational studies that use propensity score weighting. However, estimation of the VIF requires knowledge of the weights, which are only known once the study has been conducted. We describe methods to estimate the VIF based on two characteristics of the observational study: the anticipated prevalence of treatment and the anticipated c‐statistic of the propensity score model. We considered five different sets of weights: those for estimating the average treatment effect (ATE), the average treated effect in the treated (ATT), and three recently described sets of weights: overlap weights, matching weights, and entropy weights. The VIF was substantially smaller for the latter three sets of weights than for the first two sets of weights. Once the VIF has been estimated during the design phase of the study, sample size and power calculations can be done using calculations appropriate for a randomized controlled trial with similar prevalence of treatment and similar outcome variable, and then multiplying the requisite sample size by the estimated VIF. Implementation of these methods allows for improving the design and reporting of observational studies that use propensity score weighting.

A propensity score‐matched, real‐world comparison of ustekinumab vs vedolizumab as a second‐line treatment for Crohn’s disease. The Cross Pennine study II

“…Given the observational, exploratory nature of the study, and in the absence of a pre‐defined hypothesis, a sample size was not calculated a priori . A post‐hoc calculation of the power showed that this was greater than 80% for the primary outcome, according to the method by Austin based on calculation of the variance inflation factor (VIF), which describes the extent to which the effective sample size has been reduced by weighting 29 . The post‐hoc power of a chi‐square test for the observed difference in proportion of success at 3 months is 0.99, while VIF is 1.15 based on a c ‐statistics of the model equal to 0.67 and a prevalence of treatment equal to 0.6.…”

Use of Labor Neuraxial Analgesia for Vaginal Delivery and Severe Maternal Morbidity

“…Because there is no validated solution to estimate the required sample size or study power using the inverse propensity score-weighting method, we do not provide such estimates. 32…”

The Consequences of School Suspension at Different Developmental Stages: The Relationships Between Age, Race, Suspension, and Justice-Related Outcomes

“…Finally, this study is limited by the estimated statistical power. Power analyses for propensity score weighting approaches are complex, and post hoc power analyses suggested the study may be underpowered (Austin, 2021); future research should be conducted with larger samples and comparison groups to better power analyses. 2…”

“… 2. Power was checked in two ways for this study: using the method suggested by Austin (2021) and using post hoc logistic regression models. Estimates calculated using logistic regression estimates suggested the analyses were sufficiently powered; however, these estimates are likely inaccurate, as they are estimates for traditional logistic regression models rather than propensity score approaches.…”

“…Estimates calculated using logistic regression estimates suggested the analyses were sufficiently powered; however, these estimates are likely inaccurate, as they are estimates for traditional logistic regression models rather than propensity score approaches. Estimates produced using Austin’s (2021) method suggested analyses may be underpowered.…”