Evaluation of Health Information Applications – Challenges Ahead of Us

Summary

Objectives: The aim of the paper is to review the challenges for evaluation in the light of characteristics of the healthcare sector, present as well as future.

Methods: The approach is a synthesis based on highlights from the literature.

Results: The review addresses the following issues: 1) the role of evaluation activities within a systems development or implementation context; 2) suggestions on the nature of success and failure characteristics; and 3) evaluation aspects viewed in the perspective of different types of systems. Constructive evaluation, evaluation being the act of bringing about a decision-making basis, is perceived as the means to minimize failure and maximize success from the very beginning of the development or implementation. Based on these discussions, the challenges that evaluation and evaluators are facing are debated.

Conclusion: The ultimate challenge ahead is first to fill the gap of presently needed evaluation methods. This need is in particular related to evaluation of cognitive and work process-oriented aspects of IT-based solutions. Finally, the challenge is to provide constructive evaluation methods and methodologies for dealing with the full complexity and dynamics of the target domain, for application within the entire life-cycle of the IT-based systems and solutions.

Keywords

Evaluation - information systems - success and failure criteria - obstacles and barriers - challenges

• References

• 1 Dinan S. Towards the Transformation Age. Journal of World Futures 2002. Jan.
• 2 Brender J, McNair P, Nøhr C. Research Needs and Priorities in Health Informatics. Int J Med Informatics 2000; 58 (59) (01) 257-89.
• 3 Kaplan B, Shaw NT. Future directions in evaluation research: People, organizational, and social issues. Methods Inf Med 2004; 43: 215-31.
• 4 Haimes YY, Schneiter C. Covey’s Seven Habits and the Systems Approach: A Comparative Approach. IEEE transactions on Systems, Man and Cybernetics 1996; 26 (04) 483-7.
• 5 Ammenwerth E, Brender J, Nykänen P, Prokosch H-U. Rigby M, Talmon J. Visions and strategies to improve evaluation of health information systems – Reflections and lessons based on the HISEVAL workshop in Innsbruck. Int J Med Informatics 2004; 73 (06) 479-91.
• 6 Brender J. Handbook of Evaluation Methods for Health Informatics. New York: : Elsevier Academic Press; ISBN 13: 978-0-12-370464-1, ISBN 10: 0-12-370464-2 2006: 361
• 7 Brender J. Methodological and methodical perils and pitfalls within assessment studies performed on IT-based solutions in healthcare. Technical Report of the Virtual Centre for Health Informatics May 2003; 03 (01) (ISSN 1397-9507) 69 (also included in [6]).
• 8 Bikson TK, Eveland JD. Technology Transfer as a Framework for Understanding Social Impacts of Computerization. In: Smith MJ, Salvendy G. Work with Computers: Organizational, Management, Stress and Health Aspects. Proceedings of the 3rd Conference on Human-Computer Interaction. Amsterdam :: Elsevier.; 1989. 1 28-37.
• 9 Crosswell PL. Obstacles to GIS Implementation and Guidelines to Increase the Opportunities for Success. URISA J 1991; 3 (01) 43-56.
• 10 Without Change There is no Progress – Coping with Chaos, a Global Survey. Price Waterhouse 1997
• 11 Malling P. Implementation of Information Technology in Thailand; Master Thesis (Organisational Psychology) at Copenhagen University, Psychology Laboratory at Copenhagen University, Copenhagen. 1992. (in Danish).
• 12 Fröhlich D, Gill C, Krieger H. Workplace Involvement in Technological Innovation in the European Community, Vol I: Roads to Participation, European Foundation for the Improvement of Living and Working Conditions, Dublin. 1993
• 13 van Gennip EMSJ, Lorenzi NM. Results of discussions at the IMIA WG 13 and 15 working conference. Int J Med Informatics 1999; 56 (01) (03) 177-80.
• 14 Brender J. Methodology for Assessment of Medical IT-based Systems – in an Organisational Context. Amsterdam:: IOS Press, Studies in Health Technology and Informatics; 1997: 42 295
• 15 Brender J. Methodology for Constructive Assessment of IT-based Systems in an Organisational Context. Int J of Med Informatics 1999; 56: 67-86.
• 16 Beuscart-Zéphir MC, Brender J, Beuscart R, Ménager-Depriester I. Cognitive Evaluation: How to Assess the Usability of Information Technology in Healthcare. Computer Methods and Programs in Biomedicine 1997; 54 (01) (02) 19-28.
• 17 Goodstein LP. Computer Aids for Decision-Making in Process Control. In: Falzon P. Cognitive Ergonomics: Understanding, Learning and Designing Human-Computer Interaction. London: : Academic Press; 1990: 231-43.
• 18 Gruding J. Groupware and Social Dynamics: Eight Challenges for Developers. Scientific American 1991; September: 762-74.
• 19 Lehner P, Seyed-Solorforough M-M, O’Connor MF, Sak S, Mullin T. Cognitive Biases and Time Stress in Team Decision Making. IEEE Transaction on Systems, Man and Cybernetics Part A: Systems and Humans 1997; 27 (05) 698-703.