User Testing an Information Foraging Tool for Ambulatory Surgical Site Infection Surveillance

Abstract

Background Surveillance for surgical site infections (SSIs) after ambulatory surgery in children requires a detailed manual chart review to assess criteria defined by the National Health and Safety Network (NHSN). Electronic health records (EHRs) impose an inefficient search process where infection preventionists must manually review every postsurgical encounter (< 30 days). Using text mining and business intelligence software, we developed an information foraging application, the SSI Workbench, to visually present which postsurgical encounters included SSI-related terms and synonyms, antibiotic, and culture orders.

Objective This article compares the Workbench and EHR on four dimensions: (1) effectiveness, (2) efficiency, (3) workload, and (4) usability.

Methods Comparative usability test of Workbench and EHR. Objective test metrics are time per case, encounters reviewed per case, time per encounter, and retrieval of information meeting NHSN definitions. Subjective measures are cognitive load using the National Aeronautics and Space Administration (NASA) Task Load Index (NASA TLX), and a questionnaire on system usability and utility.

Results Eight infection preventionists participated in the test. There was no difference in effectiveness as subjects retrieved information from all cases, using both systems, to meet the NHSN criteria. There was no difference in efficiency in time per case between the Workbench and EHR (8.58 vs. 7.39 minutes, p = 0.36). However, with the Workbench subjects opened fewer encounters per case (3.0 vs. 7.5, p = 0.002), spent more time per encounter (2.23 vs. 0.92 minutes, p = 0.002), rated the Workbench lower in cognitive load (NASA TLX, 24 vs. 33, p = 0.02), and significantly higher in measures of usability.

Conclusion Compared with the EHR, the Workbench was more usable, short, and reduced cognitive load. In overall efficiency, the Workbench did not save time, but demonstrated a shift from between-encounter foraging to within-encounter foraging and was rated as significantly more efficient. Our results suggest that infection surveillance can be better supported by systems applying information foraging theory.

Keywords

electronic health records and systems - patient records - specific types - clinical information systems - surgical wound infection - ambulatory surgical procedures - medical informatics - decision making - computer-assisted - user–computer interface

Authors' Contributions

D.K. designed the Workbench, developed and facilitated the test, analyzed data, and authored and revised the manuscript. M.M. contributed to literature searches, design of the Workbench, and manuscript editing. M.R. developed the Workbench and contributed to generating study data. S.R. contributed to the design and testing of the Workbench and manuscript editing. R.R. contributed to methods, study design, participant randomization, data analysis, and manuscript editing. R.X. contributed to study design, data analysis, and manuscript editing. N.M. contributed to study design and manuscript editing. R.L. contributed to study design and manuscript editing. J.G. was project co-PI, contributed to literature searches, study design, and manuscript editing. S.C. was the project PI, led the team in study design, analysis, and manuscript authoring and editing. R.G. contributed to Workbench development, study design, and manuscript editing.

Protection of Human and Animal Subjects

This study was reviewed by the Children's Hospital of Philadelphia Institutional Review Board.

• References

• 1 Klevens RM, Edwards JR, Richards Jr CL. , et al. Estimating health care-associated infections and deaths in U.S. hospitals, 2002. Public Health Rep 2007; 122 (02) 160-166
• 2 Ambulatory Surgery Centers. A Positive Trend in Healthcare. Ambulatory Surgical Center Coalition. Available at: http://www.ascassociation.org . Accessed 2017
• 3 Haley RW. The scientific basis for using surveillance and risk factor data to reduce nosocomial infection rates. J Hosp Infect 1995; 30 (Suppl): 3-14
• 4 Consensus paper on the surveillance of surgical wound infections. The Society for Hospital Epidemiology of America; The Association for Practitioners in Infection Control; The Centers for Disease Control; The Surgical Infection Society. Infect Control Hosp Epidemiol 1992; 13 (10) 599-605
• 5 Surgical Site Infection (SSI) Event, Procedure-associated model, Center for Disease Control. Available at: https://www.cdc.gov/nhsn/pdfs/pscmanual/9pscssicurrent.pdf . Accessed 2017
• 6 Grammatico-Guillon L, Baron S, Gaborit C, Rusch E, Astagneau P. Quality assessment of hospital discharge database for routine surveillance of hip and knee arthroplasty-related infections. Infect Control Hosp Epidemiol 2014; 35 (06) 646-651
• 7 van Mourik MS, Troelstra A, van Solinge WW, Moons KG, Bonten MJ. Automated surveillance for healthcare-associated infections: opportunities for improvement. Clin Infect Dis 2013; 57 (01) 85-93
• 8 Woeltje KF, Lin MY, Klompas M, Wright MO, Zuccotti G, Trick WE. Data requirements for electronic surveillance of healthcare-associated infections. Infect Control Hosp Epidemiol 2014; 35 (09) 1083-1091
• 9 Pirolli P, Card S. Information foraging. Psychol Rev 1999; 106 (04) 643
• 10 Schraagen JM, Chipman SF, Shalin VL. Cognitive Task Analysis. Psychology Press; 2000
• 11 Paas F, Renkl A, Sweller J. Cognitive load theory and instructional design: recent developments. Educ Psychol 2003; 38 (01) 1-4
• 12 Chi EH, Pirolli P, Chen K. , et al. Using information scent to model user information needs and actions and the Web. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM; 2001 :490–497
• 13 Card SK, Pirolli P, Van Der Wege M. , et al. Information scent as a driver of Web behavior graphs: results of a protocol analysis method for Web usability. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM; 2001 :498–505
• 14 Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform 2004; 37 (01) 56-76
• 15 Nielsen J. Usability inspection methods. In Conference Companion on Human Factors in Computing Systems. ACM; 1999: 413-414
• 16 Rubin J, Chisnell D. Handbook of Usability Testing: How to Plan, Design and Conduct Effective Tests. Indianapolis, IN: John Wiley & Sons; 2008
• 17 Grundmeier RW, Xiao R, Ross RK. , et al. Identifying surgical site infections in electronic health data using predictive models. J Am Med Inform Assoc 2018; 25 (09) 1160-1166
• 18 Cox GM, Cochran WG. Experimental Designs. New York: Wiley; 1957
• 19 Ericsson K, Simon H. Verbal reports as data. Psychol Rev 1980; 87 (03) 215-251
• 20 Hart SG, Staveland LE. Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. Adv Psychol 1988; 52: 139-183
• 21 Hart SG. NASA-task load index (NASA-TLX); 20 years later. Proc Hum Factors Ergon Soc Annu Meet 2006; 50 (09) 904-908
• 22 R Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing;2017. Available at: http://www.r-project.org/ . Accessed March 2, 2018
• 23 Frøkjær E, Hertzum M, Hornbæk K. . Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.ACM;2000 April 1:345–352
• 24 Sweller John. Cognitive load theory, learning difficulty, and instructional design. Learning Instruct 1994; 4 (04) 295-312
• 25 Ahmed A, Chandra S, Herasevich V, Gajic O, Pickering BW. The effect of two different electronic health record user interfaces on intensive care provider task load, errors of cognition, and performance. Crit Care Med 2011; 39 (07) 1626-1634
• 26 Plaisant C, Milash B, Rose A. , et al. LifeLines: visualizing personal histories. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM;1996:221–227
• 27 Card SK, Mackinlay JD, Schneiderman B. Information Visualization: Using Vision to Think. San Francisco, CA: Morgan-Kaufmann; 1999