Development of standard indicators to assess use of electronic health record systems implemented in low-and medium-income countries

Abstract

Background Electronic Health Record Systems (EHRs) are being rolled out nationally in many low- and middle-income countries (LMICs) yet assessing actual system usage remains a challenge. We employed a nominal group technique (NGT) process to systematically develop high-quality indicators for evaluating actual usage of EHRs in LMICs. Methods An initial set of 14 candidate indicators were developed by the study team adapting the Human Immunodeficiency Virus (HIV) Monitoring, Evaluation, and Reporting indicators format. A multidisciplinary team of 10 experts was convened in a two-day NGT workshop in Kenya to systematically evaluate, rate (using Specific, Measurable, Achievable, Relevant, and Time-Bound (SMART) criteria), prioritize, refine, and identify new indicators. NGT steps included introduction to candidate indicators, silent indicator ranking, round-robin indicator rating, and silent generation of new indicators. 5-point Likert scale was used in rating the candidate indicators against the SMART components. Results Candidate indicators were rated highly on SMART criteria (4.05/5). NGT participants settled on 15 final indicators, categorized as system use (4); data quality (3), system interoperability (3), and reporting (5). Data entry statistics, systems uptime, and EHRs variable concordance indicators were rated highest. Conclusion This study describes a systematic approach to develop and validate quality indicators for determining EHRs use and provides LMICs with a multidimensional tool for assessing success of EHRs implementations.

A multivariate statistical evaluation of actual use of electronic health record systems implementations in Kenya

“…In most LMICs, measure of success of EHRs scale-up often relies on simple counts of the number of EHRs implementations. This study demonstrates that: (a) through use of standardized indicators [ 27 ], key new insights and gaps on actual status of EHRs implementations within countries use can be identified; (b) aspects of national-level EHRs usage assessments need not be time- or resource-intensive, as assessments can be automated using data already within the EHRs; and (c) mechanisms that allow efficient EHRs usage assessments offer insights to enable any identified EHRs usage gaps to be addressed in a timely manner.…”

“…The EHRs use indicators used for this study are detailed in Ngugi et al [ 27 ]. The 15 rigorously derived indicators are modelled after the HIV Monitor, Evaluation and Reporting (MER) indicators, that facilities and implementations providing HIV care would be well-familiar with [ 34 ].…”

“…Three of the eight excluded indicators (namely Reporting rate , Report timeliness and Report completeness ) rely on data in the national data aggregate system, the Kenya Health Information System (KHIS), and had already been evaluated and reported in a different study [ 29 ]. The other five excluded indicators (namely: Data entry statistics , System Uptime , EHR Variable concordance , Report Concordance and Clinical Data timeliness ) required a level of human input to generate based on how the indicators are defined [ 27 ].…”

Key Indicators for Monitoring the Efficiency of Iranian Health System: A Synthesized Design Study

“…In this stage, the final indicators were prepared in the form of a questionnaire and sent to 15 health management and economics experts. They were asked to rate the indicators between 1 and 10, taking into account various criteria based on the RACER (relevant, acceptable, credible, easy to monitor and robust) ( 30 ), SMART (Specific, Measurable, Attainable, Relevant, Time-bounded) ( 31 ), CREAM (Clear, Realistic, Economic, Attainable, Measureable) and RAVES (Relevant, Achievable, Valid, Ethical, Simple) frameworks. Final indicators were regarded as those to which assigned a score higher than seven on average.…”

Idealizing Information Systems for the Brazilian Unified Health System: Perspectives of Health Professionals

“…Studies have evaluated Health Information Systems from different perspectives, whether in the context of guidelines for developing countries (Ngugi et al, 2021), specific contexts and policies (Ahmadian et al, 2015), or even considering induced errors and performance quality (Yusof & Sahroni, 2018). In the Brazilian scenario, reviews consider the approach by institutional context, whether hospital (Pereira et al, 2016), primary care (Carreno et al, 2015), or the performance and quality of stored data (Correia et al, 2014).…”