A Method for Evaluating Computer Programs for Electrocardiographic Interpretation

Abstract

The programs studied in these three reports were chosen for two reasons: first, they were available to the National Institutes of Health, and second, they could be implemented on computers readily available to the National Institutes of Health. However the methods for evaluation reported here could be used at any other institution and are not dependent upon a specific program or computer system. The authors have a continuing interest in automated ECG analysis and may select for study additional programs, the treatment of ECG analysis of which differs significantly from the programs reported here. The results of these studies will be submitted for publication in an appropriate journal. Conclusions drawn from these studies including the present three reports represent the considered opinions of the authors and do not represent any official policy of the Department of Health, Education and Welfare. SUMMARYA method for evaluating computer programs for electrocardiographic interpretation is described. This method allows a clinician to judge the usefulness of a program for his specific setting and needs. The method requires a significant proportion and variety of abnormal tracings, the application of specific fixed criteria, and the separation of disagreements between the computer program and the clinician into those resulting from criteria differences and those resulting from programming errors, viz., pattern recognition failures, mismeasurements, and/or deficient program logic. When applied to the experimental IBM program 1971, staff cardiologists found essential agreement with the program's results in 76% of 1150 unselected tracings. Clinically significant disagreements based strictly on the application of different criteria occurred in 20% of the tracings, whereas disagreements based on program errors were found in only 4%. Although this program requires some system of human overview and quality checking, its potential for clinical implementation is worthy of consideration.