Download PDF
Methods Inf Med 2021; 60(01/02): 032-048
DOI: 10.1055/s-0041-1731784
Original Article

Why Is the Electronic Health Record So Challenging for Research and Clinical Care?

John H. Holmes
1   Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
James Beinlich
2   Information Technology Entity Services and Corporate Information Services, University of Pennsylvania Health System, Philadelphia, Pennsylvania, United States
,
Mary R. Boland
1   Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Kathryn H. Bowles
3   Department of Biobehavioral Health Sciences, University of Pennsylvania School of Nursing, Philadelphia, Pennsylvania, United States
,
Yong Chen
1   Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Tessa S. Cook
4   Department of Radiology, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
George Demiris
3   Department of Biobehavioral Health Sciences, University of Pennsylvania School of Nursing, Philadelphia, Pennsylvania, United States
,
Michael Draugelis
5   Department of Predictive Health Care, University of Pennsylvania Health System, Philadelphia, Pennsylvania, United States
,
Laura Fluharty
6   Clinical Research Operations, University of Pennsylvania Health System, Philadelphia, Pennsylvania, United States
,
Peter E. Gabriel
4   Department of Radiology, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Robert Grundmeier
7   Department of Pediatrics, The Children's Hospital of Philadelphia, Philadelphia, Pennsylvania, United States
,
C. William Hanson
8   Department of Anesthesiology and Critical Care, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Daniel S. Herman
9   Department of Pathology and Laboratory Medicine, University of Pennsylvania Perelman School of Medicine Philadelphia, Pennsylvania, United States
,
Blanca E. Himes
1   Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Rebecca A. Hubbard
1   Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Charles E. Kahn Jr.
4   Department of Radiology, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Dokyoon Kim
1   Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Ross Koppel
10   Department of Sociology, University of Pennsylvania, Philadelphia, Pennsylvania, United States
,
Qi Long
1   Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Nebojsa Mirkovic
11   Department of Research Analytics, University of Pennsylvania Health System, Philadelphia, Pennsylvania, United States
,
Jeffrey S. Morris
1   Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Danielle L. Mowery
1   Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Marylyn D. Ritchie
12   Department of Genetics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Ryan Urbanowicz
1   Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
,
Jason H. Moore
1   Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States
› Author Affiliations Funding Research reported in this publication was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health under award number: UL1TR001878. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
› Further Information
    Permissions and Reprints

Abstract

Background The electronic health record (EHR) has become increasingly ubiquitous. At the same time, health professionals have been turning to this resource for access to data that is needed for the delivery of health care and for clinical research. There is little doubt that the EHR has made both of these functions easier than earlier days when we relied on paper-based clinical records. Coupled with modern database and data warehouse systems, high-speed networks, and the ability to share clinical data with others are large number of challenges that arguably limit the optimal use of the EHR

Objectives Our goal was to provide an exhaustive reference for those who use the EHR in clinical and research contexts, but also for health information systems professionals as they design, implement, and maintain EHR systems.

Methods This study includes a panel of 24 biomedical informatics researchers, information technology professionals, and clinicians, all of whom have extensive experience in design, implementation, and maintenance of EHR systems, or in using the EHR as clinicians or researchers. All members of the panel are affiliated with Penn Medicine at the University of Pennsylvania and have experience with a variety of different EHR platforms and systems and how they have evolved over time.

Results Each of the authors has shared their knowledge and experience in using the EHR in a suite of 20 short essays, each representing a specific challenge and classified according to a functional hierarchy of interlocking facets such as usability and usefulness, data quality, standards, governance, data integration, clinical care, and clinical research.

Conclusion We provide here a set of perspectives on the challenges posed by the EHR to clinical and research users.

Keywords

electronic health records - user-computer interface - standards - medical informatics - systems integration

Supplementary Material



Publication History

Received: 28 December 2020

Accepted: 22 May 2021

Article published online:
19 July 2021

© 2021. Thieme. All rights reserved.

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

 

  • References

  • 1 Office of the National Coordinator for Health Information Technology. Office-based physician electronic health record adoption. Accessed December 8, 2020 at: dashboard.healthit.gov/quickstats/pages/physician-ehr-adoption-trends.php
  • 2 Office of the National Coordinator for Health Information Technology. Percent of hospitals, by type, that possess certified health IT. Accessed December 8, 2020 at: dashboard.healthit.gov/quickstats/pages/certified-electronic-health-record-technology-in-hospitals.php
  • 3 Artis KA, Bordley J, Mohan V, Gold JA. Data omission by physician trainees on ICU rounds. Crit Care Med 2019; 47 (03) 403-409
    CrossrefPubMedSearch in Google Scholar
  • 4 Shenvi EC, Feupe SF, Yang H, El-Kareh R. “Closing the loop”: a mixed-methods study about resident learning from outcome feedback after patient handoffs. Diagnosis (Berl) 2018; 5 (04) 235-242
    CrossrefPubMedSearch in Google Scholar
  • 5 Khairat S, Coleman C, Newlin T. et al. A mixed-methods evaluation framework for electronic health records usability studies. J Biomed Inform 2019; 94: 103175
    CrossrefPubMedSearch in Google Scholar
  • 6 Cohen DJ, Dorr DA, Knierim K. et al. Primary care practices' abilities and challenges in using electronic health record data for quality improvement. Health Aff (Millwood) 2018; 37 (04) 635-643
    CrossrefPubMedSearch in Google Scholar
  • 7 Bristol AA, Nibbelink CW, Gephart SM, Carrington JM. Nurses' use of positive deviance when encountering electronic health records-related unintended consequences. Nurs Adm Q 2018; 42 (01) E1-E11
    CrossrefPubMedSearch in Google Scholar
  • 8 Gephart S, Carrington JM, Finley B. A systematic review of nurses' experiences with unintended consequences when using the electronic health record. Nurs Adm Q 2015; 39 (04) 345-356
    CrossrefPubMedSearch in Google Scholar
  • 9 Friedman A, Crosson JC, Howard J. et al. A typology of electronic health record workarounds in small-to-medium size primary care practices. J Am Med Inform Assoc 2014; 21 (e1): e78-e83
    CrossrefPubMedSearch in Google Scholar
  • 10 Schiff GD, Zucker L. Medical scribes: salvation for primary care or workaround for poor EMR usability?. J Gen Intern Med 2016; 31 (09) 979-981
    CrossrefPubMedSearch in Google Scholar
  • 11 Flanagan ME, Saleem JJ, Millitello LG, Russ AL, Doebbeling BN. Paper- and computer-based workarounds to electronic health record use at three benchmark institutions. J Am Med Inform Assoc 2013; 20 (e1): e59-e66
    CrossrefPubMedSearch in Google Scholar
  • 12 Hysong SJ, Sawhney MK, Wilson L. et al. Provider management strategies of abnormal test result alerts: a cognitive task analysis. J Am Med Inform Assoc 2010; 17 (01) 71-77
    CrossrefPubMedSearch in Google Scholar
  • 13 Menon S, Murphy DR, Singh H, Meyer AND, Sittig DF. Workarounds and test results follow-up in electronic health record-based primary care. Appl Clin Inform 2016; 7 (02) 543-559
    Thieme ConnectPubMedSearch in Google Scholar
  • 14 Zahabi M, Kaber DB, Swangnetr M. Usability and safety in electronic medical records interface design: a review of recent literature and guideline formulation. Hum Factors 2015; 57 (05) 805-834
    CrossrefPubMedSearch in Google Scholar
  • 15 Roman LC, Ancker JS, Johnson SB, Senathirajah Y. Navigation in the electronic health record: a review of the safety and usability literature. J Biomed Inform 2017; 67: 69-79
    CrossrefPubMedSearch in Google Scholar
  • 16 Reid PP, Compton WD, Grossman JH, Fanjiang Geds. National Academy of Engineering (US) and Institute of Medicine (US) Committee on Engineering and the Health Care System. Building a Better Delivery System: A New Engineering/Health Care Partnership. Washington, DC: National Academies Press (US); 2005
    Search in Google Scholar
  • 17 Health information technology: standards, implementation specifications, and certification criteria for electronic health record technology. 2014 edition; revisions to the permanent certification program for health information technology. Accessed June 17, 2021 at: https://federalregister.gov/a/2012-20982
    Search in Google Scholar
  • 18 Ratwani RM, Fairbanks RJ, Hettinger AZ, Benda NC. Electronic health record usability: analysis of the user-centered design processes of eleven electronic health record vendors. J Am Med Inform Assoc 2015; 22 (06) 1179-1182
    CrossrefPubMedSearch in Google Scholar
  • 19 Gianfrancesco MA, Tamang S, Yazdany J, Schmajuk G. Potential biases in machine learning algorithms using electronic health record data. JAMA Intern Med 2018; 178 (11) 1544-1547
    CrossrefPubMedSearch in Google Scholar
  • 20 Kearns M, Neel S, Roth A, Wu ZS. An Empirical Study of Rich Subgroup Fairness for Machine Learning. In: Proceedings of the Conference on Fairness, Accountability, and Transparency. FAT* '19. Association for Computing Machinery; 2019 100-109
    Crossref
  • 21 Gijsberts CM, Groenewegen KA, Hoefer IE. et al. Race/ethnic differences in the associations of the framingham risk factors with carotid IMT and cardiovascular events. PLoS One 2015; 10 (07) e0132321
    CrossrefPubMedSearch in Google Scholar
  • 22 Eneanya ND, Yang W, Reese PP. Reconsidering the consequences of using race to estimate kidney function. JAMA 2019; 322 (02) 113-114
    CrossrefPubMedSearch in Google Scholar
  • 23 Rajkomar A, Hardt M, Howell MD, Corrado G, Chin MH. Ensuring fairness in machine learning to advance health equity. Ann Intern Med 2018; 169 (12) 866-872
    CrossrefPubMedSearch in Google Scholar
  • 24 Ehrmeyer SS, Laessig RH. Has compliance with CLIA requirements really improved quality in US clinical laboratories?. Clin Chim Acta 2004; 346 (01) 37-43
    CrossrefPubMedSearch in Google Scholar
  • 25 Greg Miller W, Myers GL, Lou Gantzer M. et al. Roadmap for harmonization of clinical laboratory measurement procedures. Clin Chem 2011; 57 (08) 1108-1117
    CrossrefPubMedSearch in Google Scholar
  • 26 Christenson RH, Jacobs E, Uettwiller-Geiger D. et al. Comparison of 13 commercially available cardiac troponin assays in a multicenter North American study. J Appl Lab Med 2017; 1 (05) 544-561
    CrossrefPubMedSearch in Google Scholar
  • 27 Huff SM, Rocha RA, McDonald CJ. et al. Development of the logical observation identifier names and codes (LOINC) vocabulary. J Am Med Inform Assoc 1998; 5 (03) 276-292
    CrossrefPubMedSearch in Google Scholar
  • 28 Cornet R, Chute CG. Health concept and knowledge management: twenty-five years of evolution. Yearb Med Inform 2016; 1 (9312666): (Suppl. 01) S32-S41
    PubMedSearch in Google Scholar
  • 29 Centers for Disease Control and Prevention. International Classification of Diseases, 10th Revision (ICD-10). Accessed June 17, 2021 at: https://www.cdc.gov/nchs/icd/icd10.htm
  • 30 Bodenreider O, Cornet R, Vreeman DJ. Recent Developments in Clinical Terminologies - SNOMED CT, LOINC, and RxNorm. Yearb Med Inform 2018; 27 (01) 129-139
    Thieme ConnectPubMedSearch in Google Scholar
  • 31 Hripcsak G, Duke JD, Shah NH. et al. Observational health data sciences and informatics (OHDSI): opportunities for observational researchers. Stud Health Technol Inform 2015; 216: 574-578
    PubMedSearch in Google Scholar
  • 32 World Health Organization. International Statistical Classification of Diseases and Related Health Problems (ICD). ICD-11. Accessed June 17, 2021 at: https://www.who.int/classifications/icd/en/
  • 33 Smith SW, Koppel R. Healthcare information technology's relativity problems: a typology of how patients' physical reality, clinicians' mental models, and healthcare information technology differ. J Am Med Inform Assoc 2014; 21 (01) 117-131
    CrossrefPubMedSearch in Google Scholar
  • 34 Woodfield R, Grant I, Sudlow CL. UK Biobank Stroke Outcomes Group, UK Biobank Follow-Up and Outcomes Working Group. Accuracy of electronic health record data for identifying stroke cases in large-scale epidemiological studies: a systematic review from the UK Biobank Stroke Outcomes Group. PLoS One 2015; 10 (10) e0140533
    CrossrefPubMedSearch in Google Scholar
  • 35 Liao KP, Cai T, Savova GK. et al. Development of phenotype algorithms using electronic medical records and incorporating natural language processing. BMJ 2015; 350: h1885
    CrossrefPubMedSearch in Google Scholar
  • 36 Schulz S, Jansen L. Formal ontologies in biomedical knowledge representation. Yearb Med Inform 2013; 8 (9312666): 132-146
    PubMedSearch in Google Scholar
  • 37 Smith B, Ashburner M, Rosse C. et al; OBI Consortium. The OBO Foundry: coordinated evolution of ontologies to support biomedical data integration. Nat Biotechnol 2007; 25 (11) 1251-1255
    CrossrefPubMedSearch in Google Scholar
  • 38 Zhang XA, Yates A, Vasilevsky N. et al. Semantic integration of clinical laboratory tests from electronic health records for deep phenotyping and biomarker discovery. NPJ Digit Med 2019; 2 (32) 32
    CrossrefPubMedSearch in Google Scholar
  • 39 Bona JP, Prior FW, Zozus MN, Brochhausen M. Enhancing clinical data and clinical research data with biomedical ontologies - insights from the knowledge representation perspective. Yearb Med Inform 2019; 28 (01) 140-151
    Thieme ConnectPubMedSearch in Google Scholar
  • 40 Brochhausen M, Bona J, Blobel B. The role of axiomatically-rich ontologies in transforming medical data to knowledge. Stud Health Technol Inform 2018; 249: 38-49
    PubMedSearch in Google Scholar
  • 41 Bowles KH, Potashnik S, Ratcliffe SJ. et al. Conducting research using the electronic health record across multi-hospital systems: semantic harmonization implications for administrators. J Nurs Adm 2013; 43 (06) 355-360
    CrossrefPubMedSearch in Google Scholar
  • 42 Nordo AH, Levaux HP, Becnel LB. et al. Use of EHRs data for clinical research: Historical progress and current applications. Learn Health Syst 2019; 3 (01) e10076
    CrossrefPubMedSearch in Google Scholar
  • 43 Garza M, Del Fiol G, Tenenbaum J, Walden A, Zozus MN. Evaluating common data models for use with a longitudinal community registry. J Biomed Inform 2016; 64: 333-341
    CrossrefPubMedSearch in Google Scholar
  • 44 Collins FS, Hudson KL, Briggs JP, Lauer MS. PCORnet: turning a dream into reality. J Am Med Inform Assoc 2014; 21 (04) 576-577
    CrossrefPubMedSearch in Google Scholar
  • 45 Murphy SN, Weber G, Mendis M. et al. Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2). J Am Med Inform Assoc 2010; 17 (02) 124-130
    CrossrefPubMedSearch in Google Scholar
  • 46 Murtagh MJ, Blell MT, Butters OW. et al. Better governance, better access: practising responsible data sharing in the METADAC governance infrastructure. Hum Genomics 2018; 12 (01) 24
    CrossrefPubMedSearch in Google Scholar
  • 47 Meinert E, Alturkistani A, Brindley D, Knight P, Wells G, de Pennington N. Weighing benefits and risks in aspects of security, privacy and adoption of technology in a value-based healthcare system. BMC Med Inform Decis Mak 2018; 18 (01) 100
    CrossrefPubMedSearch in Google Scholar
  • 48 Dankar FK, Ptitsyn A, Dankar SK. The development of large-scale de-identified biomedical databases in the age of genomics-principles and challenges. Hum Genomics 2018; 12 (01) 19
    CrossrefPubMedSearch in Google Scholar
  • 49 Stahl BC, Rainey S, Harris E, Fothergill BT. The role of ethics in data governance of large neuro-ICT projects. J Am Med Inform Assoc 2018; 25 (08) 1099-1107
    CrossrefPubMedSearch in Google Scholar
  • 50 Powles J, Hodson H. Google deepmind and healthcare in an age of algorithms. Health Technol (Berl) 2017; 7 (04) 351-367
    CrossrefPubMedSearch in Google Scholar
  • 51 Roland M. Linking physicians' pay to the quality of care--a major experiment in the United Kingdom. N Engl J Med 2004; 351 (14) 1448-1454
    CrossrefPubMedSearch in Google Scholar
  • 52 Roland M, Guthrie B. Quality and outcomes framework: what have we learnt?. BMJ 2016; 354: i4060
    CrossrefPubMedSearch in Google Scholar
  • 53 United States District Court Northern District of Illinois Eastern Division. MATT DINERSTEIN, individually and on behalf of all others similarly situated, Plaintiff, v. Google, LLC, a Delaware limited liability company, THE UNIVERSITY OF CHICAGO MEDICAL CENTER, an Illinois not-for-profit corporation, and THE UNIVERSITY OF CHICAGO, an Illinois not-for-profit corporation. Accessed June 17, 2021 at: https://www.courtlistener.com/recap/gov.uscourts.ilnd.366172/gov.uscourts.ilnd.366172.85.0.pdf
  • 54 Ohno-Machado L. To share or not to share: that is not the question. Sci Transl Med 2012; 4 (165) 165cm15
    CrossrefPubMedSearch in Google Scholar
  • 55 Wu Y, Jiang X, Kim J, Ohno-Machado L. Grid Binary LOgistic REgression (GLORE): building shared models without sharing data. J Am Med Inform Assoc 2012; 19 (05) 758-764
    CrossrefPubMedSearch in Google Scholar
  • 56 Ohno-Machado L, Agha Z, Bell DS. et al; pSCANNER team. pSCANNER: patient-centered scalable national network for effectiveness research. J Am Med Inform Assoc 2014; 21 (04) 621-626
    CrossrefPubMedSearch in Google Scholar
  • 57 Duan R, Boland MR, Liu Z. et al. Learning from electronic health records across multiple sites: a communication-efficient and privacy-preserving distributed algorithm. J Am Med Inform Assoc 2020; 27 (03) 376-385
    CrossrefPubMedSearch in Google Scholar
  • 58 Duan R, Luo C, Schuemie MJ. et al. Learning from local to global: an efficient distributed algorithm for modeling time-to-event data. J Am Med Inform Assoc 2020; 27 (07) 1028-1036
    CrossrefPubMedSearch in Google Scholar
  • 59 Deng Y, Jiang X, Long Q. Privacy-preserving methods for vertically partitioned incomplete data. Annu Symp Am Med Inform Assoc 2021; 2020: 348-357
    PubMedSearch in Google Scholar
  • 60 Chang C, Deng Y, Jiang X, Long Q. Multiple imputation for analysis of incomplete data in distributed health data networks. Nat Commun 2020; 11 (01) 5467
    CrossrefPubMedSearch in Google Scholar
  • 61 Health Information Privacy. Guidance regarding methods for de-identification of protected health information in accordance with the Health Insurance Portability and Accountability Act (HIPAA) privacy rule. Accessed August 3, 2020. Accessed August 3, 2020 at: https://www.hhs.gov/hipaa/for-professionals/privacy/special-topics/de-identification/index.html
  • 62 Kushida CA, Nichols DA, Jadrnicek R, Miller R, Walsh JK, Griffin K. Strategies for de-identification and anonymization of electronic health record data for use in multicenter research studies. Med Care 2012; 50: S82-S101
    CrossrefPubMedSearch in Google Scholar
  • 63 Rocher L, Hendrickx JM, de Montjoye Y-A. Estimating the success of re-identifications in incomplete datasets using generative models. Nat Commun 2019; 10 (01) 3069
    CrossrefPubMedSearch in Google Scholar
  • 64 Meystre SM, Friedlin FJ, South BR, Shen S, Samore MH. Automatic de-identification of textual documents in the electronic health record: a review of recent research. BMC Med Res Methodol 2010; 10 (100968545): 70
    CrossrefPubMedSearch in Google Scholar
  • 65 Hayden EC. Privacy loophole found in genetic databases. Accessed June 17, 2021 at: https://www.nature.com/articles/nature.2013.12237
  • 66 Pecoraro F, Luzi D, Ricci FL. Designing ETL tools to feed a data warehouse based on electronic healthcare record infrastructure. Stud Health Technol Inform 2015; 210: 929-933
    PubMedSearch in Google Scholar
  • 67 Pecoraro F, Luzi D, Ricci FL. Secondary uses of EHR systems: A feasibility study. Published in: E-Health and Bioengineering Conference (EHB). 21–23 November. 2013; Iasi, Romania; 2013: 1-6
    Search in Google Scholar
  • 68 Botsis T, Hartvigsen G, Chen F, Weng C. Secondary use of EHR: data quality issues and informatics opportunities. Summit On Translat Bioinforma 2010; 2010: 1-5
    PubMedSearch in Google Scholar
  • 69 Weiskopf NG, Hripcsak G, Swaminathan S, Weng C. Defining and measuring completeness of electronic health records for secondary use. J Biomed Inform 2013; 46 (05) 830-836
    CrossrefPubMedSearch in Google Scholar
  • 70 Kroth PJ, Morioka-Douglas N, Veres S. et al. Association of electronic health record design and use factors with clinician stress and burnout. JAMA Netw Open 2019; 2 (08) e199609-e199609
    Search in Google Scholar
  • 71 National Academy of Medicine, Board on Population Health and Public Health Practice, Institute of Medicine. Capturing Social and Behavioral Domains and Measures in Electronic Health Records: Phase 2. Washington, DC: National Academies Press (US); 2015
    Search in Google Scholar
  • 72 Bycroft C, Freeman C, Petkova D. et al. The UK Biobank resource with deep phenotyping and genomic data. Nature 2018; 562 (7726): 203-209
    CrossrefPubMedSearch in Google Scholar
  • 73 Hamilton CM, Strader LC, Pratt JG. et al. The PhenX Toolkit: get the most from your measures. Am J Epidemiol 2011; 174 (03) 253-260
    CrossrefPubMedSearch in Google Scholar
  • 74 McCarty CA, Berg R, Rottscheit CM. et al. Validation of PhenX measures in the personalized medicine research project for use in gene/environment studies. BMC Med Genomics 2014; 7: 3
    CrossrefPubMedSearch in Google Scholar
  • 75 Luzi D, Rocco I, Tamburis O, Corso B, Minicuci N, Pecoraro F. Variability in the assessment of children's primary healthcare in 30 European countries. Int J Qual Health Care 2021; 33 (01) mzab007
    CrossrefPubMedSearch in Google Scholar
  • 76 Deans KJ, Sabihi S, Forrest CB. Learning health systems. Semin Pediatr Surg 2018; 27 (06) 375-378
    CrossrefPubMedSearch in Google Scholar
  • 77 Sarafidis M, Tarousi M, Anastasiou A. et al. Data quality challenges in a learning health system. Stud Health Technol Inform 2020; 270: 143-147
    PubMedSearch in Google Scholar
  • 78 Chen JH, Alagappan M, Goldstein MK, Asch SM, Altman RB. Decaying relevance of clinical data towards future decisions in data-driven inpatient clinical order sets. Int J Med Inform 2017; 102: 71-79
    CrossrefPubMedSearch in Google Scholar
  • 79 Sculley D, Holt G, Golovin D. et al. Hidden technical debt in machine learning systems. Accessed June 17, 2021 at: https://papers.nips.cc/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf
  • 80 Mathes T, Buehn S, Prengel P, Pieper D. Registry-based randomized controlled trials merged the strength of randomized controlled trails and observational studies and give rise to more pragmatic trials. J Clin Epidemiol 2018; 93: 120-127
    CrossrefPubMedSearch in Google Scholar
  • 81 Workman TA. Engaging Patients in Information Sharing and Data Collection: The Role of Patient-Powered Registries and Research Networks. Rockville, MD: Agency for Healthcare Research and Quality; 2013
    Search in Google Scholar
  • 82 Wozniak L, Soprovich A, Rees S, Johnson ST, Majumdar SR, Johnson JA. Challenges in identifying patients with Type 2 Diabetes for quality-improvement interventions in primary care settings and the importance of valid disease registires. Can J Diabetes 2015; 39 (Suppl. 03) S77-S82
    CrossrefPubMedSearch in Google Scholar
  • 83 Hripcsak G, Albers DJ. Next-generation phenotyping of electronic health records. J Am Med Inform Assoc 2013; 20 (01) 117-121
    CrossrefPubMedSearch in Google Scholar
  • 84 Xie S, Himes BE. Approaches to link geospatially varying social, economic, and environmental factors with electronic health record data to better understand asthma exacerbations. AMIA Annu Symp Proc 2018; 2018: 1561-1570
    PubMedSearch in Google Scholar
  • 85 Xie S, Greenblatt R, Levy MZ, Himes BE. Enhancing electronic health record data with geospatial information. AMIA Jt Summits Transl Sci Proc 2017; 2017: 123-132
    PubMedSearch in Google Scholar
  • 86 Goldstein BA, Bhavsar NA, Phelan M, Pencina MJ. Controlling for informed presence bias due to the number of health encounters in an electronic health record. Am J Epidemiol 2016; 184 (11) 847-855
    CrossrefPubMedSearch in Google Scholar
  • 87 Rusanov A, Weiskopf NG, Wang S, Weng C. Hidden in plain sight: bias towards sick patients when sampling patients with sufficient electronic health record data for research. BMC Med Inform Decis Mak 2014; 14 (01) 51
    CrossrefPubMedSearch in Google Scholar
  • 88 Wei W-Q, Teixeira PL, Mo H, Cronin RM, Warner JL, Denny JC. Combining billing codes, clinical notes, and medications from electronic health records provides superior phenotyping performance. J Am Med Inform Assoc 2016; 23 (e1): e20-e27
    CrossrefPubMedSearch in Google Scholar
  • 89 Hubbard RA, Tong J, Duan R, Chen Y. Reducing bias due to outcome misclassification for epidemiologic studies using EHR-derived probabilistic phenotypes. Epidemiology 2020; 31 (04) 542-550
    CrossrefPubMedSearch in Google Scholar
  • 90 Shortreed SM, Cook AJ, Coley RY, Bobb JF, Nelson JC. Challenges and opportunities for using big health care data to advance medical science and public health. Am J Epidemiol 2019; 188 (05) 851-861
    CrossrefPubMedSearch in Google Scholar
  • 91 Agniel D, Kohane IS, Weber GM. Biases in electronic health record data due to processes within the healthcare system: retrospective observational study. BMJ 2018; 361: k1479
    CrossrefPubMedSearch in Google Scholar
  • 92 Haneuse S, Daniels M. A general framework for considering selection bias in EHR-based studies: what data are observed and why?. EGEMS (Wash DC) 2016; 4 (01) 1203
    PubMedSearch in Google Scholar
  • 93 Wu J, Roy J, Stewart WF. Prediction modeling using EHR data: challenges, strategies, and a comparison of machine learning approaches. Med Care 2010; 48 (06) S106-S113
    CrossrefPubMedSearch in Google Scholar
  • 94 Pathak J, Kho AN, Denny JC. Electronic health records-driven phenotyping: challenges, recent advances, and perspectives. J Am Med Inform Assoc 2013; 20 (e2): e206-e211
    CrossrefPubMedSearch in Google Scholar
  • 95 Robnik-Šikonja M, Kononenko I. Theoretical and empirical analysis of ReliefF and RReliefF. Mach Learn 2003; 53: 23-69
    CrossrefPubMedSearch in Google Scholar
  • 96 Madhavan S, Bastarache L, Brown JS. et al. Use of electronic health records to support a public health response to the COVID-19 pandemic in the United States: a perspective from 15 academic medical centers. J Am Med Inform Assoc 2021; 28 (02) 393-401
    CrossrefPubMedSearch in Google Scholar
  • 97 Haendel MA, Chute CG, Bennett TD. et al. The national COVID cohort collaborative (N3C): rationale, design, infrastructure, and deployment. J Am Med Inform Assoc 2021; 28 (03) 427-443
    CrossrefPubMedSearch in Google Scholar
  • 98 Brat GA, Weber GM, Gehlenborg N. et al. International electronic health record-derived COVID-19 clinical course profiles: the 4CE consortium. NPJ Digit Med 2020; 3: 109
    Search in Google Scholar
  • 99 Burn E, You SC, Sena AG. et al. Deep phenotyping of 34,128 adult patients hospitalised with COVID-19 in an international network study. Nat Commun 2020; 11 (01) 5009
    CrossrefPubMedSearch in Google Scholar
  • 100 Friedman C, Rubin J, Brown J. et al. Toward a science of learning systems: a research agenda for the high-functioning learning health system. J Am Med Inform Assoc 2015; 22 (01) 43-50
    CrossrefPubMedSearch in Google Scholar
  • 101 Hutter F, Kotthoff L, Vanschoren J. Automated Machine Learning: Methods, Systems, Challenges. 1st ed.. Switzerland: Springer Nature; 2019
    CrossrefSearch in Google Scholar