Evaluating how clear the questions being investigated in randomised trials are: systematic review of estimands

Abstract

Objectives To evaluate how often the precise research question being addressed about an intervention (the estimand) is stated or can be determined from reported methods, and to identify what types of questions are being investigated in phase 2-4 randomised trials. Design Systematic review of the clarity of research questions being investigated in randomised trials in 2020 in six leading general medical journals. Data source PubMed search in February 2021. Eligibility criteria for selecting studies Phase 2-4 randomised trials, with no restrictions on medical conditions or interventions. Cluster randomised, crossover, non-inferiority, and equivalence trials were excluded. Main outcome measures Number of trials that stated the precise primary question being addressed about an intervention (ie, the primary estimand), or for which the primary estimand could be determined unambiguously from the reported methods using statistical knowledge. Strategies used to handle post-randomisation events that affect the interpretation or existence of patient outcomes, such as intervention discontinuations or uses of additional drug treatments (known as intercurrent events), and the corresponding types of questions being investigated. Results 255 eligible randomised trials were identified. No trials clearly stated all the attributes of the estimand. In 117 (46%) of 255 trials, the primary estimand could be determined from the reported methods. Intercurrent events were reported in 242 (95%) of 255 trials; but the handling of these could only be determined in 125 (49%) of 255 trials. Most trials that provided this information considered the occurrence of intercurrent events as irrelevant in the calculation of the treatment effect and assessed the effect of the intervention regardless (96/125, 77%)—that is, they used a treatment policy strategy. Four (4%) of 99 trials with treatment non-adherence owing to adverse events estimated the treatment effect in a hypothetical setting (ie, the effect as if participants continued treatment despite adverse events), and 19 (79%) of 24 trials where some patients died estimated the treatment effect in a hypothetical setting (ie, the effect as if participants did not die). Conclusions The precise research question being investigated in most trials is unclear, mainly because of a lack of clarity on the approach to handling intercurrent events. Clear reporting of estimands is necessary in trial reports so that all stakeholders, including clinicians, patients and policy makers, can make fully informed decisions about medical interventions. Systematic review registration PROSPERO CRD42021238053.

Long-term safety and efficacy of treating symptomatic, partial-thickness rotator cuff tears with fresh, uncultured, unmodified, autologous, adipose-derived regenerative cells isolated at the point of care: 41 months follow-up of a prospective, randomized, controlled, first-in-human clinical trial

“…In line with the new The International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH) E9 (R1) Addendum on the use of estimands in clinical trials (i.e., a precise description of the treatment effect to be estimated from a trial (the question) [24–26]) a comprehensive estimand was constructed for the present study. The four components of this estimand (Population (i.e., the target population for the research question), Variables (i.e., the endpoints that were obtained from all subjects), Intercurrent Events (i.e., all events that occurred after treatment initiation and either precluded the observation of a variable, or affected its interpretation) and Population-Level Summary (i.e., the variables on which the comparison between treatments was based) are outlined in Appendix 2 .…”

“…In short, in case of treatment failures (comprising all intercurrent events that required additional injections of corticosteroid into the index shoulder or surgery of the index shoulder that were definitely, probably or possibly related to the study treatments, including development of a full-thickness rotator cuff tear) subjects’ data were handled using a combination of the While-on-Treatment Strategy [24–26] and the Composite Strategy [24–26]. Specifically, response to study treatment before the occurrence of the intercurrent event was handled using the While-on-Treatment Strategy (i.e., subjects’ data were used as collected), whereas response to study treatment after the occurrence of the intercurrent event was imputed according to the Composite Variable Strategy as minimum ASES Total score (0), minimum SF-36 Total score (0), maximum VAS pain score (10) and maximum tear volume measured on MRIs (150 mm 3 , which was greater than all data measured during the present and the former studies [12]).…”

“…In contrast, intercurrent events that required additional injections of corticosteroid into the index shoulder or surgery of the index shoulder that were unlikely related or unrelated to the study treatments (e.g., accidents that affected the index shoulder) were handled using a combination of the While-on-Treatment Strategy and a Hyopthetical Strategy [24–26]. Specifically, response to the study treatment before the occurrence of the intercurrent event was handled according to the While-on-Treatment Strategy, whereas response to the study treatment after the occurrence of the intercurrent event was imputed according to a Hypothetical Strategy in which the intercurrent event would not occur.…”

Time to improve the clarity of clinical trial reports by including estimands

“…These call for trialists to clarify the precise questions being addressed in trials by using estimands . An estimand is a clear description of precisely what treatment effect a trial is aiming to investigate (for examples see23). Another helpful way to think about an estimand is a definition of precisely what the trial demands to find out (as demand nicely rhymes with estimand).…”

“…Even with statistical knowledge I have found it is not always possible to understand exactly what question a clinical trial addresses from the statistical methods alone. As this is a new area of interest, while medicine regulators worldwide are adopting the ICH E9(R1) guidelines,4 my statistical colleagues and I conducted a review of published trials to assess how often the precise question being investigated could be unravelled from the reported methods 2. This was to establish whether the reporting of estimands is necessary in trial reports to understand what question has been addressed.…”