How Good Are the Data? Feasible Approach to Validation of Metrics of Quality Derived From an Outpatient Electronic Health Record

Abstract

Although electronic health records (EHRs) promise to be efficient resources for measuring metrics of quality, they are not designed for such population-based analyses. Thus, extracting meaningful clinical data from them is not straightforward. To avoid poorly executed measurements, standardized methods to measure and to validate metrics of quality are needed. This study provides and evaluates a use case for a generally applicable approach to validating quality metrics measured electronically from EHR-based data. The authors iteratively refined and validated 4 outpatient quality metrics and classified errors in measurement. Multiple iterations of validation and measurement resulted in high levels of sensitivity and agreement versus the "gold standard" of manual review. In contrast, substantial differences remained for measurement based on coded billing data. Measuring quality metrics using an EHR-based electronic process requires validation to ensure accuracy; approaches to validation such as those described in this study should be used by organizations measuring quality from EHR-based information.

Caveats for the Use of Operational Electronic Health Record Data in Comparative Effectiveness Research

“…For example, different ways of calculating adverse drug event rates from a single institution’s EHR were associated with significantly different results [43]. Likewise, quality metrics using EHR data required substantial validation to ensure accuracy [44]. An additional analysis compared a manually abstracted observational study of community-acquired pneumonia to a fully EHR-based study without manual data abstraction [45].…”

Pain research using Veterans Health Administration electronic and administrative data sources

“…Because these sources are designed and implemented to support patient care and clinical processes, not research [40], the data needed to accomplish research objectives may not be available, may lack the information and details needed, or may be hard to retrieve. For example, EHR and administrative data omit valuable information [2], such as the severity of a patient's illness [30] and degree of disability.…”

Measuring health data management maturity in Abu Dhabi

“…Although there is acknowledgement that access to good data is a critical component of health data management, across all varieties of health systems, in developed as well as developing countries, good data are not always available when and where they are needed [11][12][13][14]. Improving access to relevant, high quality data is a critical step to become a mature health data management organization [10].…”