Intra-database validation of case-identifying algorithms using reconstituted electronic health records from healthcare claims data

Abstract

Background Diagnosis performances of case-identifying algorithms developed in healthcare database are usually assessed by comparing identified cases with an external data source. When this is not feasible, intra-database validation can present an appropriate alternative. Objectives To illustrate through two practical examples how to perform intra-database validations of case-identifying algorithms using reconstituted Electronic Health Records (rEHRs). Methods Patients with 1) multiple sclerosis (MS) relapses and 2) metastatic castration-resistant prostate cancer (mCRPC) were identified in the French nationwide healthcare database (SNDS) using two case-identifying algorithms. A validation study was then conducted to estimate diagnostic performances of these algorithms through the calculation of their positive predictive value (PPV) and negative predictive value (NPV). To that end, anonymized rEHRs were generated based on the overall information captured in the SNDS over time (e.g. procedure, hospital stays, drug dispensing, medical visits) for a random selection of patients identified as cases or non-cases according to the predefined algorithms. For each disease, an independent validation committee reviewed the rEHRs of 100 cases and 100 non-cases in order to adjudicate on the status of the selected patients (true case/ true non-case), blinded with respect to the result of the corresponding algorithm. Results Algorithm for relapses identification in MS showed a 95% PPV and 100% NPV. Algorithm for mCRPC identification showed a 97% PPV and 99% NPV. Conclusion The use of rEHRs to conduct an intra-database validation appears to be a valuable tool to estimate the performances of a case-identifying algorithm and assess its validity, in the absence of alternative.

Assessing the Risk of Relapse Requiring Corticosteroids After In Vitro Fertilization in Women With Multiple Sclerosis

“…22,23 Because this clinical event was not directly available in the database, we used a recently published specific algorithm based on hospital admissions and/or the use of corticosteroids (either oral or intravenous). 24-26…”

“…The event of interest was the occurrence of relapse which is usually defined as new symptoms or worsening of pre-existing symptoms for at least 24 hours and occurring at least 30 days after a previous relapse 22,23 . As this clinical event was not directly available in the database, we used a recently published specific algorithm based on hospital admissions and/or use of corticosteroids (either oral or intravenous) [24][25][26] .…”

“…22,23 Because this clinical event was not directly available in the database, we used a recently published specific algorithm based on hospital admissions and/or the use of corticosteroids (either oral or intravenous). [24][25][26] Four outcomes were considered: the ARR and the proportion of IVF where a woman had at least 1 relapse after IVF were compared with those before IVF; the difference in the number of relapses after vs before IVF was also calculated; and the delay from IVF to the first relapse within 1 year.…”

Validation of Algorithms to Identify Acute Myocardial Infarction, Stroke, and Cardiovascular Death in German Health Insurance Data

“…However, Thurin et al conclude that in the absence of an alternative, tabular patient profiles based on claims data ("reconstituted electronic health records") are a valuable tool for intra-database validation and performance estimation of case-identifying algorithms. 5 In fact, Thurin et al suggested specific conditions necessary for a successful validation process, which were satisfied in our study: "1) the health outcome of interest must be managed by a specific sequence of cares and encounters; and 2) the considered health-care database must capture in an exhaustive way a sufficient number of medical elements in line with the outcome of interest". Misclassification of outcomes might seriously bias study findings.…”

“…Patient profiles extracted from French claims data, ie, chronological tabular summaries of the relevant information available on a patient, have been shown to serve as an alternative to patient charts. 5 To date, no study has been published on using patient profiles extracted from German claims data for outcome validation.…”