A Statistical Approach for the Learning Curve of Physicians in Utilization of Electronic Order Sets

Abstract

Background Understanding a physician's behavior toward learning order sets is important as it is a key information to design order sets with optimized contents.

Objective The objective of this article is to test a hypothesis: for a physician using a new order set repeatedly, the utilization rate of order set contents has a pattern of either increase or decrease.

Methods To test the hypothesis, we retrieved empirical data of order set usage in local hospitals that adopted a new computerized physician order entry (CPOE) system and enterprise wide standard order sets. We extracted 4-year data including 63,583 orders made by 600 physicians in the inpatient setting and analyzed patterns of the learning curve at several aggregation levels.

Result The analysis results demonstrated that content modification rates over time were relatively flat except for a few localized patterns.

Conclusion Based on our finding, we reject our initial hypothesis.

Keywords

order set - computerized physician order entry - learning curve - electronic health record

Note

This research does not involve human patients.

• References

• 1 Wright A, Sittig DF, Ash JS. , et al. Development and evaluation of a comprehensive clinical decision support taxonomy: comparison of front-end tools in commercial and internally developed electronic health record systems. J Am Med Inform Assoc 2011; 18 (03) 232-242
• 2 Starmer J, Lorenzi N, Pinson CW. The Vanderbilt EvidenceWeb - developing tools to monitor and improve compliance with evidence-based order sets. AMIA Annu Symp Proc 2006; 749-753
• 3 Jacobs BR, Hart KW, Rucker DW. Reduction in clinical variance using targeted design changes in computerized provider order entry (CPOE) order sets: Impact on hospitalized children with acute asthma exacerbation. Appl Clin Inform 2012; 3 (01) 52-63
• 4 Starmer J, Waitman LR. Orders and evidence-based order sets - Vanderbilt's experience with CPOE ordering patterns between 2000 and 2005. AMIA Annu Symp Proc 2006; 1108
• 5 Ahmadian L, Khajouei R. Impact of computerized order sets on practitioner performance. Stud Health Technol Inform 2012; 180: 1129-1131
• 6 Wright A, Sittig DF. Automated development of order sets and corollary orders by data mining in an ambulatory computerized physician order entry system. AMIA Annu Symp Proc 2006; 819-823
• 7 Hulse NC, Del Fiol G, Bradshaw RL, Roemer LK, Rocha RA. Towards an on-demand peer feedback system for a clinical knowledge base: a case study with order sets. J Biomed Inform 2008; 41 (01) 152-164
• 8 Bobb AM, Payne TH, Gross PA. Viewpoint: controversies surrounding use of order sets for clinical decision support in computerized provider order entry. J Am Med Inform Assoc 2007; 14 (01) 41-47
• 9 Lee F, Teich JM, Spurr CD, Bates DW. Implementation of physician order entry: user satisfaction and self-reported usage patterns. J Am Med Inform Assoc 1996; 3 (01) 42-55
• 10 Khajouei R, Peek N, Wierenga PC, Kersten MJ, Jaspers MW. Effect of predefined order sets and usability problems on efficiency of computerized medication ordering. Int J Med Inform 2010; 79 (10) 690-698
• 11 Bates DW, Kuperman GJ, Wang S. , et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003; 10 (06) 523-530
• 12 Horsky J, Kaufman DR, Oppenheim MI, Patel VL. A framework for analyzing the cognitive complexity of computer-assisted clinical ordering. J Biomed Inform 2003; 36 (1,2): 4-22
• 13 Munasinghe RL, Arsene C, Abraham TK, Zidan M, Siddique M. Improving the utilization of admission order sets in a computerized physician order entry system by integrating modular disease specific order subsets into a general medicine admission order set. J Am Med Inform Assoc 2011; 18 (03) 322-326
• 14 Zhang Y, Levin JE, Padman R. Data-driven order set generation and evaluation in the pediatric environment. AMIA Annu Symp Proc 2012; 2012: 1469-1478
• 15 Zhang Y, Padman R, Levin JE. Paving the COWpath: data-driven design of pediatric order sets. J Am Med Inform Assoc 2014; 21 (e2): e304-e311
• 16 Hulse NC, Lee J. Extracting actionable recommendations for modifying enterprise order set templates from CPOE utilization patterns. AMIA Annu Symp Proc 2018; 2017: 950-958
• 17 Wu RC, Abrams H, Baker M, Rossos PG. Implementation of a computerized physician order entry system of medications at the University Health Network--physicians' perspectives on the critical issues. Healthc Q 2006; 9 (01) 106-109
• 18 Brunette DD, Tersteeg J, Brown N. , et al. Implementation of computerized physician order entry for critical patients in an academic emergency department is not associated with a change in mortality rate. West J Emerg Med 2013; 14 (02) 114-120