Download PDF
Appl Clin Inform 2012; 03(02): 211-220
DOI: 10.4338/ACI-2012-02-R-0004
Review
Schattauer GmbH

Measurement Error in Performance Studies of Health Information Technology: Lessons from the Management Literature

A.S. Litwin
1   Carey Business School and School of Medicine, Johns Hopkins University, Baltimore, MD
,
A.C. Avgar
2   School of Labor & Employment Relations and College of Medicine, University of Illinois at Urbana-Champaign, Urbana-Champaign, IL
,
P.J. Pronovost
3   School of Medicine, Bloom-berg School of Public Health, and Carey Business School, Johns Hopkins University, Baltimore, MD
› Author Affiliations
Further Information

Publication History

received: 16 February 2012

accepted: 20 May 2012

Publication Date:
16 December 2017 (online)

   Permissions and Reprints

Summary

Just as researchers and clinicians struggle to pin down the benefits attendant to health information technology (IT), management scholars have long labored to identify the performance effects arising from new technologies and from other organizational innovations, namely the reorganization of work and the devolution of decision-making authority. This paper applies lessons from that literature to theorize the likely sources of measurement error that yield the weak statistical relationship between measures of health IT and various performance outcomes. In so doing, it complements the evaluation literature’s more conceptual examination of health IT’s limited performance impact. The paper focuses on seven issues, in particular, that likely bias downward the estimated performance effects of health IT. They are 1.) negative self-selection, 2.) omitted or unobserved variables, 3.) mis-measured contextual variables, 4.) mismeasured health IT variables, 5.) lack of attention to the specific stage of the adoption-to-use continuum being examined, 6.) too short of a time horizon, and 7.) inappropriate units-of-analysis. The authors offer ways to counter these challenges. Looking forward more broadly, they suggest that researchers take an organizationally-grounded approach that privileges internal validity over generalizability. This focus on statistical and empirical issues in health IT-performance studies should be complemented by a focus on theoretical issues, in particular, the ways that health IT creates value and apportions it to various stakeholders.

Keywords

Health information technology - electronic health records - research methods - organizational behavior
 

  • References

  • 1 Blumenthal D. Launching HITECH. N Engl J Med 2010; 362 (05) 382-385.
    CrossrefPubMedGoogle Scholar
  • 2 Jha AK, DesRoches CM, Kralovec PD, Joshi MS. A progress report on electronic health records in US hospitals. Health Aff 2010; 29 (010) 1951-1957.
    CrossrefPubMedGoogle Scholar
  • 3 Chaudhry B. et al. Systematic review: Impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006; 144 (010) 742-752.
    CrossrefPubMedGoogle Scholar
  • 4 Garrido T, Jamieson L, Zhou Y, Wiesenthal A, Liang L. Effect of electronic health records in ambulatory care: Retrospective, serial, cross sectional study. Br Med J 2005; 330: (581).
    CrossrefPubMedGoogle Scholar
  • 5 DesRoches CM. et al. Electronic health records’ limited successes suggest more targeted uses. Health Aff 2010; 29 (04) 639-646.
    CrossrefPubMedGoogle Scholar
  • 6 Parente ST, McCullough JS. Health information technology and patient safety: Evidence from panel data. Health Aff 2009; 28 (02) 357-360.
    CrossrefPubMedGoogle Scholar
  • 7 Himmelstein DU, Wright A, Woolhandler S. Hospital computing and the costs and quality of care: A national study. Am J Med 2010; 123 (01) 40-46.
    CrossrefPubMedGoogle Scholar
  • 8 Koppel R. et al. Role of computerized physician order entry systems in facilitating medication errors. J Am Med Assoc 2005; 293 (010) 1197-1203.
    CrossrefPubMedGoogle Scholar
  • 9 Avgar AC, Litwin AS, Pronovost PJ. Using management research to conceptualize the drivers and barriers to health IT adoption. Working paper. 2012
    PubMedGoogle Scholar
  • 10 Ancker JS, Kern LM, Abramson E, Kaushal R. The triangle model for evaluating the effect of health information technology on healthcare quality and safety. J Am Med Inform Assn 2012; 19 (01) 61-65.
    CrossrefPubMedGoogle Scholar
  • 11 Ammenwerth E. et al. Visions and strategies to improve evaluation of health information systems: Reflections and lessons based on the HIS-EVAL workshop in innsbruck. Int J Med Inf 2004; 73 (06) 479-491.
    CrossrefPubMedGoogle Scholar
  • 12 Nykänen P. et al. Introducing guidelines for good evaluation practice in health informatics. In: Adlassnig K, Blobel B, Mantas J, Masic I. editors. Medical Informatics in a United and Healthy Europe. Amsterdam; Washington, DC: IOS; 2009: 958-962.
    Google Scholar
  • 13 Westbrook JI. et al. Multimethod evaluation of information and communication technologies in health in the context of wicked problems and sociotechnical theory. J Am Med Inform Assn 2007; 14 (06) 746-755.
    CrossrefPubMedGoogle Scholar
  • 14 Kochan TA. On the human side of technology. ICL Tech J 1988; 6 (02) 391-400.
    PubMedGoogle Scholar
  • 15 MacDuffie JP, Krafcik JF. Integrating technology and human resources for high-performance manufacturing: Evidence from the international auto industry. In: Kochan TA, Useem M. editors. Transforming Organizations. New York: Oxford; 1992: 209-226.
    Google Scholar
  • 16 MacDuffie JP. Human-resource bundles and manufacturing performance: Organizational logic and flexible production systems in the world auto industry. Ind Labor Relat Rev 1995; 48 (02) 197-221.
    CrossrefPubMedGoogle Scholar
  • 17 Kelley MR. Participative bureaucracy and productivity in the machined products sector. Ind Relat 1996; 35 (03) 374-399.
    CrossrefPubMedGoogle Scholar
  • 18 Batt RL. Work organization, technology, and performance in customer service and sales. Ind Labor Relat Rev 1999; 52 (04) 539-564.
    CrossrefPubMedGoogle Scholar
  • 19 Litwin AS. Technological change at work: The impact of employee involvement on the effectiveness of health information technology. Ind Labor Relat Rev 2011; 64 (05) 863-888.
    CrossrefPubMedGoogle Scholar
  • 20 Brynjolfsson E, Hitt LM. Beyond computation: Information technology, organizational transformation and business performance. J Econ Perspect 2000; 14 (04) 23-48.
    CrossrefPubMedGoogle Scholar
  • 21 Solow R. We’d better watch out. New York Times Book Review 1987; July (012) (36).
    PubMedGoogle Scholar
  • 22 Brynjolfsson E, Hitt LM. Computing productivity: Firm-level evidence. Rev Econ Stat 2003; 85 (04) 793-808.
    CrossrefPubMedGoogle Scholar
  • 23 Cappelli P, Neumark D. Do ‘High-performance’ work practices improve establishment-level outcomes?. Ind Labor Relat Rev 2001; 54 (04) 737-775.
    PubMedGoogle Scholar
  • 24 Freeman RB, Kleiner MM. Who benefits most from employee involvement: Firms or workers?. Am Econ Rev 2000; 90 (02) 219-223.
    CrossrefPubMedGoogle Scholar
  • 25 Yoo KH. et al. The impact of electronic medical records on timeliness of diagnosis of asthma. J Asthma 2007; 44 (09) 753-758.
    CrossrefPubMedGoogle Scholar
  • 26 Ammenwerth E. et al. The effect of electronic prescribing on medication errors and adverse drug events: A systematic review. J Am Med Inform Assoc 2008; 15 (05) 585-600.
    CrossrefPubMedGoogle Scholar
  • 27 Litwin AS. Not featherbedding but feathering the nest: Human resource management and investments in information technology. Ind Relat. forthcoming.
    PubMedGoogle Scholar
  • 28 Ichniowski C. et al. What works at work: Overview and assessment. Ind Relat 1996; 35 (03) 299-333.
    CrossrefPubMedGoogle Scholar
  • 29 Sittig DF. et al. Lessons from “Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system”. Pediatrics 2006; 118 (02) 797-801.
    CrossrefPubMedGoogle Scholar
  • 30 Ash JS. et al. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2007; 14 (04) 415-423.
    CrossrefPubMedGoogle Scholar
  • 31 Talmon J. et al. STARE-HI –statement on reporting of evaluation studies in health informatics. Int J Med Inf 2009; 78 (01) 1-9.
    CrossrefPubMedGoogle Scholar
  • 32 Brynjolfsson E, Hitt LM, Yang S. Intangible assets: Computers and organizational capital. Brookings Pap Eco Ac 2002; 1: 137-181.
    PubMedGoogle Scholar
  • 33 Poon EG. et al. Relationship between use of electronic health record features and health care quality: Results of a statewide survey. Med Care 2010; 48 (03) 203-209.
    CrossrefPubMedGoogle Scholar
  • 34 Yu FB. et al. Full implementation of computerized physician order entry and medication-related quality outcomes: A study of 3364 hospitals. Am J Med Qual 2009; 24 (04) 278-286.
    CrossrefPubMedGoogle Scholar
  • 35 McCullough JS, Casey M, Moscovice I, Prasad S. The effect of health information technology on quality in US hospitals. Health Aff 2010; 29 (04) 647-654.
    CrossrefPubMedGoogle Scholar
  • 36 Devaraj S, Kohli R. Performance impacts of information technology: Is actual usage the missing link. Manage Sci 2003; 49 (03) 273-289.
    CrossrefPubMedGoogle Scholar
  • 37 Keyhani S. et al. Electronic health record components and the quality of care. Med Care 2008; 46 (012) 1267-1272.
    CrossrefPubMedGoogle Scholar
  • 38 Freeman RB. Longitudinal analyses of the effects of trade unions. J Labor Econ 1984; 2 (01) 1-26.
    CrossrefPubMedGoogle Scholar
  • 39 Robinson JC. et al. Financial incentives, quality improvement programs, and the adoption of clinical information technology. Med Care 2009; 47 (04) 411-417.
    CrossrefPubMedGoogle Scholar
  • 40 Hitt LM, Brynjolfsson E. Information technology and internal firm organization: An exploratory analysis. J Manage Inform Syst 1997; 14 (02) 81-101.
    CrossrefPubMedGoogle Scholar
  • 41 Han YY. et al. Unexpected increased mortality after implementation of a commercially sold physician order entry system. Pediatrics 2005; 116 (06) 1506-1512.
    CrossrefPubMedGoogle Scholar
  • 42 Eaton AE. The survival of employee participation programs in unionized settings. Ind Labor Relat Rev 1994; 47 (03) 371-389.
    CrossrefPubMedGoogle Scholar
  • 43 Amarasingham R. et al. Clinical information technologies and inpatient outcomes: A multiple hospital study. Arch Intern Med 2009; 169 (02) 108-114.
    CrossrefPubMedGoogle Scholar
  • 44 Skrondal A, Rabe-Hesketh S. Generalized latent variable modeling: Multilevel, longitudinal, and structural equation models. Boca Raton: Chapman and Hall/CRC; 2004
    Google Scholar
  • 45 Huber GP, Power DJ. Retrospective reports of strategic-level managers: Guidelines for increasing their accuracy. Strateg Manage J 1985; 6 (02) 171-180.
    CrossrefPubMedGoogle Scholar
  • 46 Gerhart B, Wright PM, MacMahan GC, Snell SA. Measurement error in research on human resources and firm performance: How much error is there and how does it influence effect size estimates?. Pers Psychol 2000; 53 (04) 803-834.
    CrossrefPubMedGoogle Scholar
  • 47 Kaushal R. et al. Electronic prescribing improves medication safety in community-based office practices. J Gen Intern Med 2010; 25 (06) 530-536.
    CrossrefPubMedGoogle Scholar
  • 48 Buntin MB, Burke MF, Hoaglin MC, Blumenthal D. The benefits of health information technology: A review of the recent literature shows predominantly positive results. Health Aff 2011; 30 (03) 464-471.
    CrossrefPubMedGoogle Scholar