Use of Propensity Score Methodology in Contemporary High-Impact Surgical Literature

Abstract

BACKGROUND: Propensity score (PS) analysis is a statistical method commonly used in observational trials to account for confounding. Improper use of PS analysis can bias the effect estimate. The aim of this study is to review the use and reporting of PS methods in high-impact surgical journals with a focus on propensity score matching (PSM). STUDY DESIGN: The 10 surgical journals with the highest impact factors were searched to identify studies using PS analysis from January 1, 2016 to December 14, 2018. We selected evaluation criteria for the conduct of PS analysis based on previous reports. Two authors systematically appraised the quality of reporting of PS analyses. Univariate and multivariate regression was performed to determine the relationship between appropriate use of PSM and study conclusion. RESULTS: Three hundred and three studies using PS analysis were included. Ninety-one percent (n ¼ 275) of studies included the covariates used to generate the PS and 79% (n ¼ 239) included the type of regression model used. Ninety percent (n ¼ 272) of studies did not justify the covariates included in their PS. Eighty-four percent of studies used PSM (n ¼ 254), with 48% (n ¼ 123) failing to assess covariate balance between groups. We found that justification of the selection of covariates included in the PS and the characterization of unmatched patients were both associated with lower odds of the study finding a significant result (odds ratio 0.37; 95% CI 0.16 to 0.87; p ¼ 0.02 and odds ratio 0.35; 95% CI 0.17 to 0.75; p ¼ 0.007, respectively, at multivariate logistic regression). CONCLUSIONS: This study demonstrates that even in research published in high-quality surgical journals, several studies report their PS methodology inadequately. The inadequate conduct of PS analysis can impact a study's conclusion.

Propensity score matching in otolaryngologic literature: A systematic review and critical appraisal

“…Disagreements regarding exclusion criteria were resolved by a third-party with an advanced degree in epidemiology or biostatistics. Reporting quality was defined using guidelines adapted from Yao et al and Grose et al which have previously been used to evaluate PSM methodology in a variety of other fields [14,16,19,21]. The following study characteristics were recorded: journal, journal impact factor, subspecialty, year, initial sample size, matched sample size, and collaboration with a statistician.…”

“…Additionally, we recorded whether studies found statistically significant results for their primary outcome. If the primary outcome PLOS ONE was unclear, the outcome given the most attention in the discussion section by the length of text it was examined was selected [21]. PSM reproducibility was assessed based on inclusion of four reporting criteria as established by Lonjon et al and others [14,[24][25][26][27]: (1) the algorithm used for matching; (2) the matching ratio; (3) whether replacement was used in the matching process (4) whether the statistical tests used to compare PSM groups assumed independent or paired groups.…”

“…Indeed, past research, both in non-surgical and surgical fields, has shown that the reporting and reproducibility of PSM methodology in medical literature may be at times flawed [14,16,19,20]. Further, improper PSM reporting in high-impact surgical literature has been associated with increased odds of studies reporting statistically significant results [21].…”

Reporting quality evaluation of propensity score method of the published articles in the gastric cancer filed: a systematic review

“…When some discrepancies exited, we resolved them by making discussions or consulting with a third author. The items based on previously literatures [2,3,10] were critically adopted to extract information. The general characteristics of included articles contained year of publication, name of journal, origin region of first author, author's affiliations, participation of statistician or epidemiologist in author ( identified from author's affiliations or the acknowledgements part ), international cooperation, journal source of Science Citation Index(SCI), impact factors(IF), number of citations, number of pages, number of authors, funding of support, the way to determine the sample size and number of patients engaged.…”

“…As a widely used statistical method, the PS may control confounding factors by conditioning the probability of receiving the treatment of each participants. It performed better than multivariable regression in terms of randomized design, because the PS precluded those patients who have no similar distributions in groups[1] and it was not limited by the total sample size [2]. The PS also allows researchers to better understand the potential impact of medical interventions and complement the findings of observational studies.…”