Download PDF
Methods Inf Med 1990; 29(04): 289-297
DOI: 10.1055/s-0038-1634806
Evaluation of Diagnostic Support Systems
Schattauer GmbH

Reference Standards for Software Evaluation

J. Michaelis
1   Institut für Medizinische Statistik und Dokumentation der Johannes-Gutenberg-Universität, Mainz, FRG
,
S. Wellek
1   Institut für Medizinische Statistik und Dokumentation der Johannes-Gutenberg-Universität, Mainz, FRG
,
J. L. Willems
2   CSE Coordinating Center, University of Leuven, Belgium
› Author AffiliationsThe research presented in this paper was supported by the European Commission within the frame of its 2nd, 3rd and 4th Medical Public Health Research Programmes of DG XII, Section Biomedical Engineering.
Further Information

Publication History

Publication Date:
06 February 2018 (online)

    Permissions and Reprints

Abstract

The field of automated ECG analysis was one of the earliest topics in Medical Informatics and may be regarded as a model both for computer-assisted medical diagnosis and for evaluating medical diagnostic programs. The CSE project has set reference standards of two kinds: In a broad sense, a standard how to perform a comprehensive evaluation study, in a narrow sense, standards as specific references for evaluating computer ECG programs. The evaluation methodology used within the CSE project is described as a basis for presentation of results which are published elsewhere in this issue.

Key-Words

Computerized Electrocardiography - Software Evaluation - Computer-Aided Diagnosis - Reference Standards - Diagnostic Efficacy
 

  • REFERENCES

  • 1 Willems JL, Abreu-Lima C, Arnaud P, Brohet C, Denis B, Gehring J, Graham I, van Herpen G, Machado H, Michaelis J, Moulopoulos SD. Evaluation of ECG interpretation results obtained by computer and cardiologists. Meth Inf Med 1990; 29: 308-16.
    PubMedSearch in Google Scholar
  • 2 American Heart Association Committee on Electrocardiography. Pipberger HV, Arzbaecher RC, Berson AS. et al. Recommendations for standardization of leads and of specifications for instruments in electrocardiography and vectorcardiography. Circulation 1975; 52: 11-31.
    PubMedSearch in Google Scholar
  • 3 Bailey JJ, Berson A, Jackson L, Milliken J, Stevens J, Tolan G, Wolf H. Evaluation methodologies for ECG diagnostic systems. In: Bonner RE, Pryor TA, Laks MM, Colle SS. eds. Computerized Interpretation of the ECG VI. New York: Engineering Foundation; 1981: 53-62.
    Search in Google Scholar
  • 4 Willems JL, Arnaud P, van Bemmel JH. et al. Assessment of the performance of electrocardiographic computer programs with the use of a reference database. Circulation 1985; 71: 523-34.
    CrossrefPubMedSearch in Google Scholar
  • 5 Willems JL, Arnaud P, van Bemmel JH, Degani R, Macfarlane PW, Zywietz C. Common standards for quantitative electrocardiography: goals and main results. Meth Inform Med 1990; 29: 263-271.
    PubMedSearch in Google Scholar
  • 6 Dalkey N. Analyses from a group opinion study. Rand Corporation. Futures December 1969; 541-50.
    PubMedSearch in Google Scholar
  • 7 Willems JL, Arnaud P, van Bemmel JH. et al. A reference database for multilead electrocardiographic computer measurement programs. J Am Coll Cardiol 1987; 10: 1313-21.
    CrossrefPubMedSearch in Google Scholar
  • 8 Zywietz Chr, Alraun W, Willems JL. on behalf of the CSE Working Party. Results of ECG program noise tests within the CSE-project. In: Ripley K. (ed.) Computers in Cardiology. Long Beach Cal: IEEE Computer Society; 1984: 377-80.
    Search in Google Scholar
  • 9 Rautaharju PM, Ariet M, Pryor TA. et al. The quest for optimal electrocardiography. Task Force III on computers in diagnostic electrocardiography. Am J Cardiol 1978; 41: 155-71.
    PubMedSearch in Google Scholar
  • 10 Willems JL, Abreu-Lima C, Arnaud P, van Bemmel JH, Brohet CR. et al. Effect of combining electrocardiographic interpretation results on diagnostic accuracy. Eur Heart J 1988; 09: 1348-55.
    CrossrefPubMedSearch in Google Scholar
  • 11 Kors JA, Sittig AC, van Bemmel JH. The Delphi method to validate diagnostic knowledge in computerized ECG interpretation. Meth Inform Med 1990; 29: 44-50.
    PubMedSearch in Google Scholar
  • 12 The CSE working party. Recommendations for measurement standards in quantitative electrocardiography. Eur Heart J 1985; 06: 815-25.
    PubMedSearch in Google Scholar
  • 13 Simonson E, Tuna N, Okamoto N, Toshima H. Diagnostic accuracy of the vectorcardiogram and electrocardiogram. Am J Cardiol 1966; 17: 829-78.
    CrossrefPubMedSearch in Google Scholar
  • 14 Rautaharju PM, Blackburn HW, Warren J. The concepts of sensitivity, specificity and accuracy in evaluation of electrocardiographic, vectorcardiographic and polar-cardiographic criteria. J Electrocardiol 1976; 09: 275-81.
    CrossrefPubMedSearch in Google Scholar
  • 15 Rautaharju PM, Smets Ph. Evaluation of computer-ECG programs. The strange case of the golden standard. Comp Biomed Res 1979; 12: 39-46.
    CrossrefPubMedSearch in Google Scholar
  • 16 Smets Ph, Willems JL, Talmon J, De Maertelaer V, Komreich F. Methodology for the comparison of various diagnostic procedures. Biometrie-Praximetrie 1975; 15: 3-35.
    PubMedSearch in Google Scholar
  • 17 Dudeck J, Michaelis J. Problems in the diagnostic process in electrocardiography. In: Zywietz Chr, Schneider B. eds. Computer Application on ECG and VCG Analysis. Amsterdam: North-Holland Publ; 1973: 283-95.
    Search in Google Scholar
  • 18 Connell FA, Koepsell TD. Measures of gain in certainty from a diagnostic test. Am J Epidemiol 1985; 121: 744-59.
    CrossrefPubMedSearch in Google Scholar
  • 19 Diamond GA, Hirsch M, Forrester J, Stanilow HM, Vas R, Halperion SW, Swan HJC. Application of information theory to clinical diagnostic testing. Circulation 1981; 63: 915-21.
    CrossrefPubMedSearch in Google Scholar
  • 20 Willems JL, Campbell G, Bailey JJ. Progress on the CSE diagnostic study. Application of McNemar’s test revisted. J Electrocardiol 1989; 22: 135-41.
    PubMedSearch in Google Scholar
  • 21 Gjörup T. The Kappa coefficient and the prevalence of a diagnosis. Meth Inform Med 1988; l27: 184-6.
    PubMedSearch in Google Scholar
  • 22 Morlet D, Rubel P, Willems JL. Value of scatter-graphs for the evaluation of ECG computer measurement results. Meth Inf Med 1990; 29: 413-23.
    PubMedSearch in Google Scholar
  • 23 Willems JL, Zywietz Chr, van Bemmel JH, Degani R, MacFarlane PW. Influence of noise on wave boundary recognition by ECG measurement programs. Comp Biomed Res 1987; 20: 543-62.
    CrossrefPubMedSearch in Google Scholar
  • 24 Zywietz Chr, Willems JL, Arnaud P, van Bemmel JH, Degani R, MacFarlane PW. Stability of computer ECG measurements in the presence of noise. Comp Biomed Res 1990; 23: 10-31.
    CrossrefPubMedSearch in Google Scholar
  • 25 Schork MA, Williams GW. Number of observations required for the comparison of two correlated proportions. Comm Stat Simul Comput. 1980. 9 349-57.
    Search in Google Scholar
  • 26 Lehmann EL. Nonparametrics: Statistical Methods Based on Ranks. San Francisco: Holden-Day Inc; 1975
    Search in Google Scholar
  • 27 Wellek S. Testing for equivalence with binomial data (to be published).
    PubMed
  • 28 Michaelis J, Willems JL. Performance evaluation of diagnostic ECG programs. In: Ripley KL. ed. Computers in Cardiology. Long Beach Cal: IEEE Computer Society; 1988: 25-30.
    Search in Google Scholar