On the limitations of comparative effectiveness research

Abstract

There is a little doubt that the topic of the target article is extremely important-how to provide guidance for making medical decisions that are based on evidence concerning which interventions are likely to work well in practice. The hope for this enterprise, now called CER for 'comparative effectiveness research,' which is similar to the previously used 'evidence-based medicine,' is to use the voluminous amounts of data often routinely collected in our society, such as for insurance and other administrative uses, or from federal, state, and private sample surveys, to learn about the relative effectiveness of all sorts of medical interventions.The target article itself is rich in its statement of goals and general objectives, but it seems to me to be quite thin in its statement of (a) specific suggestions for what characteristics the data sets should possess to make such effectiveness comparisons plausibly accurate or (b) what methodological techniques are most likely to be appropriate to apply to the data sets to achieve these goals. Even the specific examples cited, at least those with which I am familiar, seem to me to be relatively poor sources of guidance for either the kinds of data sets needed or for the kinds of statistical techniques to be applied to them to achieve credible answers.In my limited experience in trying to address such topics, I find that there is too little thoughtful guidance for the correct design and analysis of investigations that use complex data sets, mostly non-randomized observational ones, to estimate the effects of medical interventions; an exception is Cochran [1], but that article does not reflect the substantial changes that have taken place over the last half century in statistics, computing, medicine, and routinely collected databases. It is important for me to emphasize that much of the required effort when attacking a causal question using observational data takes place before any of the usual statistical analyses of the data are initiated, just as with welldesigned randomized experiments. Moreover, often the inevitable conclusion of a thoughtful attack may be that there are no reliable answers available from existing data sets, or even from summaries of them based on meta-analyses [2], effect-size surface estimations [3], or from research syntheses [4]. It is my firm view that it is better, for eventual progress, to know that we cannot yet reliably answer questions than to pretend that we know the answers based on bogus assumptions and hopelessly inadequate data.My comments are admittedly somewhat idiosyncratic in that they rely largely on references to my own work or work with which I have substantial familiarity, although there are many fine sources of ideas on the topics. I do this mainly because, when thinking about the issues that the target article raises, many of these concern what is needed to create a reliable observational study, and I have spent a large amount of time thinking and writing about this in the context of real problems for more than four decades. ...

Estimating the average treatment effects of nutritional label use using subclassification with regression adjustment

“…65,66 The balance and estimation procedures provided here are important extensions of propensity score analysis to causal effects estimation for observational studies when the exposure is ordinal. These procedures yield, under proper assumptions, unbiased and transitive estimates of average treatment effects.…”

Introduction

“…For example, in a 2010 commentary, Rubin warns against what he perceives as a tendency for researchers to use inadequate data sets for CER studies and recommends some ways to evaluate and select more relevant data sets. 8 In a 2011 article, Alemayehu and Cappelleri point out some of the historical weaknesses in conducting and reporting observational and non-randomized studies, including sources of bias, and recommend steps to minimize bias in the design, analysis and reporting stages of a study. 9 Without a doubt, the increase in CER will change the health care industry in nearly every area, from development costs to formulary decision making, from treatment decisions to product innovation.…”

Innovative Clinical Trial Designs

“…Despite considerable skepticism and warnings on the part of methodologists [73], most projections of CER in the future reflect even greater reliance on observational methods (statistical adjustment and instrumental variable methods). The low impact of recent large-scale effectiveness trials has prompted several calls for improvement of the trial designs.…”