Reference Standards for Software Evaluation

Abstract

The field of automated ECG analysis was one of the earliest topics in Medical Informatics and may be regarded as a model both for computer-assisted medical diagnosis and for evaluating medical diagnostic programs. The CSE project has set reference standards of two kinds: In a broad sense, a standard how to perform a comprehensive evaluation study, in a narrow sense, standards as specific references for evaluating computer ECG programs. The evaluation methodology used within the CSE project is described as a basis for presentation of results which are published elsewhere in this issue.

Key-Words

Computerized Electrocardiography - Software Evaluation - Computer-Aided Diagnosis - Reference Standards - Diagnostic Efficacy

• REFERENCES

• 1 Willems JL, Abreu-Lima C, Arnaud P, Brohet C, Denis B, Gehring J, Graham I, van Herpen G, Machado H, Michaelis J, Moulopoulos SD. Evaluation of ECG interpretation results obtained by computer and cardiologists. Meth Inf Med 1990; 29: 308-16.
• 2 American Heart Association Committee on Electrocardiography. Pipberger HV, Arzbaecher RC, Berson AS. et al. Recommendations for standardization of leads and of specifications for instruments in electrocardiography and vectorcardiography. Circulation 1975; 52: 11-31.
• 3 Bailey JJ, Berson A, Jackson L, Milliken J, Stevens J, Tolan G, Wolf H. Evaluation methodologies for ECG diagnostic systems. In: Bonner RE, Pryor TA, Laks MM, Colle SS. eds. Computerized Interpretation of the ECG VI. New York: Engineering Foundation; 1981: 53-62.
• 4 Willems JL, Arnaud P, van Bemmel JH. et al. Assessment of the performance of electrocardiographic computer programs with the use of a reference database. Circulation 1985; 71: 523-34.
• 5 Willems JL, Arnaud P, van Bemmel JH, Degani R, Macfarlane PW, Zywietz C. Common standards for quantitative electrocardiography: goals and main results. Meth Inform Med 1990; 29: 263-271.
• 6 Dalkey N. Analyses from a group opinion study. Rand Corporation. Futures December 1969; 541-50.
• 7 Willems JL, Arnaud P, van Bemmel JH. et al. A reference database for multilead electrocardiographic computer measurement programs. J Am Coll Cardiol 1987; 10: 1313-21.
• 8 Zywietz Chr, Alraun W, Willems JL. on behalf of the CSE Working Party. Results of ECG program noise tests within the CSE-project. In: Ripley K. (ed.) Computers in Cardiology. Long Beach Cal: IEEE Computer Society; 1984: 377-80.
• 9 Rautaharju PM, Ariet M, Pryor TA. et al. The quest for optimal electrocardiography. Task Force III on computers in diagnostic electrocardiography. Am J Cardiol 1978; 41: 155-71.
• 10 Willems JL, Abreu-Lima C, Arnaud P, van Bemmel JH, Brohet CR. et al. Effect of combining electrocardiographic interpretation results on diagnostic accuracy. Eur Heart J 1988; 09: 1348-55.
• 11 Kors JA, Sittig AC, van Bemmel JH. The Delphi method to validate diagnostic knowledge in computerized ECG interpretation. Meth Inform Med 1990; 29: 44-50.
• 12 The CSE working party. Recommendations for measurement standards in quantitative electrocardiography. Eur Heart J 1985; 06: 815-25.
• 13 Simonson E, Tuna N, Okamoto N, Toshima H. Diagnostic accuracy of the vectorcardiogram and electrocardiogram. Am J Cardiol 1966; 17: 829-78.
• 14 Rautaharju PM, Blackburn HW, Warren J. The concepts of sensitivity, specificity and accuracy in evaluation of electrocardiographic, vectorcardiographic and polar-cardiographic criteria. J Electrocardiol 1976; 09: 275-81.
• 15 Rautaharju PM, Smets Ph. Evaluation of computer-ECG programs. The strange case of the golden standard. Comp Biomed Res 1979; 12: 39-46.
• 16 Smets Ph, Willems JL, Talmon J, De Maertelaer V, Komreich F. Methodology for the comparison of various diagnostic procedures. Biometrie-Praximetrie 1975; 15: 3-35.
• 17 Dudeck J, Michaelis J. Problems in the diagnostic process in electrocardiography. In: Zywietz Chr, Schneider B. eds. Computer Application on ECG and VCG Analysis. Amsterdam: North-Holland Publ; 1973: 283-95.
• 18 Connell FA, Koepsell TD. Measures of gain in certainty from a diagnostic test. Am J Epidemiol 1985; 121: 744-59.
• 19 Diamond GA, Hirsch M, Forrester J, Stanilow HM, Vas R, Halperion SW, Swan HJC. Application of information theory to clinical diagnostic testing. Circulation 1981; 63: 915-21.
• 20 Willems JL, Campbell G, Bailey JJ. Progress on the CSE diagnostic study. Application of McNemar’s test revisted. J Electrocardiol 1989; 22: 135-41.
• 21 Gjörup T. The Kappa coefficient and the prevalence of a diagnosis. Meth Inform Med 1988; l27: 184-6.
• 22 Morlet D, Rubel P, Willems JL. Value of scatter-graphs for the evaluation of ECG computer measurement results. Meth Inf Med 1990; 29: 413-23.
• 23 Willems JL, Zywietz Chr, van Bemmel JH, Degani R, MacFarlane PW. Influence of noise on wave boundary recognition by ECG measurement programs. Comp Biomed Res 1987; 20: 543-62.
• 24 Zywietz Chr, Willems JL, Arnaud P, van Bemmel JH, Degani R, MacFarlane PW. Stability of computer ECG measurements in the presence of noise. Comp Biomed Res 1990; 23: 10-31.
• 25 Schork MA, Williams GW. Number of observations required for the comparison of two correlated proportions. Comm Stat Simul Comput. 1980. 9 349-57.
• 26 Lehmann EL. Nonparametrics: Statistical Methods Based on Ranks. San Francisco: Holden-Day Inc; 1975
• 27 Wellek S. Testing for equivalence with binomial data (to be published).
• 28 Michaelis J, Willems JL. Performance evaluation of diagnostic ECG programs. In: Ripley KL. ed. Computers in Cardiology. Long Beach Cal: IEEE Computer Society; 1988: 25-30.