Real‐world reproducibility study characterizing patients newly diagnosed with multiple myeloma using Clinical Practice Research Datalink, a UK‐based electronic health records database

Abstract

Purpose We evaluated the reproducibility of a study characterizing newly‐diagnosed multiple myeloma (MM) patients within an electronic health records (EHR) database using different analytic tools. Methods We reproduced the findings of a descriptive cohort study using an iterative two‐phase approach. In Phase I, a common protocol and statistical analysis plan (SAP) were implemented by independent investigators using the Aetion Evidence Platform® (AEP), a rapid‐cycle analytics tool, and SAS statistical software as a gold standard for statistical analyses. Using the UK Clinical Practice Research Datalink (CPRD) dataset, the study included patients newly diagnosed with MM within primary care setting and assessed baseline demographics, conditions, drug exposure, and laboratory procedures. Phase II incorporated analysis revisions based on our initial comparison of the Phase I findings. Reproducibility of findings was evaluate by calculating the match rate and absolute difference in prevalence between the SAS and AEP study results. Results Phase I yielded slightly discrepant results, prompting amendments to SAP to add more clarity to operational decisions. After detailed specification of data and operational choices, exact concordance was achieved for the number of eligible patients (N = 2646), demographics, comorbidities (i.e., osteopenia, osteoporosis, cardiovascular disease [CVD], and hypertension), bone pain, skeletal‐related events, drug exposure, and laboratory investigations in the Phase II analyses. Conclusions In this reproducibility study, a rapid‐cycle analytics tool and traditional statistical software achieved near‐exact findings after detailed specification of data and operational choices. Transparency and communication of the study design, operational and analytical choices between independent investigators were critical to achieve this reproducibility.

Clinical features and diagnosis of multiple myeloma: a population-based cohort study in primary care

“…Reasons for imaging procedures were not available and patients may have received those for reasons not related to the CRAB criteria work-up. Details and lists of Read code have been previously described 21…”

The Implication of Latent Information Quality to the Reproducibility of Secondary Use of Electronic Health Records

“…Reproducibility is however broadly defined [ 2 ], one pertinent definition being the ability to obtain consistent results when using the same data, methods, and tools [ 3 ]. One particular challenge for ensuring reproducibility in EHR-based research is data quality-related issues caused by non-standard data representations, ambiguous definitions, and missing or redundant documentation [ 4 ]. As illustrated in Figure 1 , since the primary functions of EHR systems are centered around patient-oriented care management and billing rather than for research purposes, the objective and priority of data documentation and reporting may vary significantly by providers and care settings [ 5 ].…”

Describing diversity of real world data sources in pharmacoepidemiologic studies: The DIVERSE scoping review

“…For example, in the topic Inclusion in the population, the theme 'Data source entry and exit dates not mentioned' found cases where dates of entry and exit from the study were mentioned, with no apparent awareness that a study cohort derived from the secondary use of a data source must be nested in the data source in the first place. 43,47,53,54 In one paper, 55 a statement implied that the date of entry in the data source was not retrieved from the data source itself but was instead chosen as the date of the first record prompted in the data source ('population of all patients with at least one record in the database'), and there was an acknowledgement that this may induce important bias. Without transparent reporting on such elements, it becomes challenging to determine their impact on study results, which may ultimately hamper their reproducibility.…”