Analysis of Informatics Topics in Accreditation Council for Graduate Medical Education Program Requirements

Abstract

Background There is growing recognition of the need to incorporate informatics education in U.S. residencies. Medical residency training programs are critical in shaping system change and can play a pivotal role by incorporating clinical informatics (CI) based learning into their training requirements.

Objectives We searched the Accreditation Council for Graduate Medical Education (ACGME) Residency Program Requirements effective July 1, 2023, to assess the inclusion of CI topics for all medical residency specialties to assess the relative levels of CI knowledge expected by graduates.

Methods We performed independent full-text search queries of 23 informatics-related keywords (e.g., electronic health record, innovation, database) in the ACGME Residency Program Requirements of 24 medical specialties.

Results All specialties' requirements contained at least five different keywords, with the total count ranging from 25 to 42 (mean: 32.00; standard deviation: 5.09). Pathology contained the highest counts with 42, followed by internal medicine and family medicine with 41 each. Pathology included the most distinct keywords (11). The most common keywords were “leadership” (62%) and “electronic health record” (10%). There were no specific mentions of several keywords—including “analytics,” “artificial intelligence,” and “machine learning”—within any program requirements. Although the ACGME Residency Program Requirements state that residents must demonstrate competence in using information technology to optimize learning, the extent is not fully specified; only 10 programs mention the keyword “information technology” within their specialty guidelines.

Conclusion The integration of CI education varies across specialties and may be even more variable across programs. Our study highlights potential opportunities for further standardization and integration of CI into resident curriculum requirements in order to better prepare future physician workforces for a changing medical landscape. We encourage educators, residency review committees, and national specialty organizations to consider further exploring the incorporation of CI content into residency training program requirements.

Keywords

education - Accreditation Council for Graduate Medical Education - professional training - clinical informatics - workforce

Background and Significance

Clinical informatics (CI) is a young but growing medical subspecialty. The field focuses on transforming health care by analyzing, designing, implementing, and evaluating information and communication systems to enhance individual and population health outcomes, improve patient care, and strengthen the clinician–patient relationship.[1] National science academies have recognized that digital technology is increasingly being incorporated in every aspect of health care.[2] [3] [4] [5] All clinicians—including physicians of all medical specialties—will need to demonstrate proficiency in CI applications as a routine standard of medical care in the United States. Struggles including the transition to electronic health records (EHRs) and adoption of computerized provider order entry (CPOE) have contributed to physician burnout and clinicians leaving the practice of medicine.[6] While strategies are needed to help providers in all stages of clinical practice, early exposure to CI for current trainees is critical to develop the emerging physician workforce. Artificial intelligence (AI) and its subdisciplines like large language models (LLMs), and natural language processing (NLP) have rapidly become ubiquitous in mainstream society, and most clinicians use some form of clinical decision support (CDS) tools in routine practice.[7] Future generations of physicians will have trained in contexts dependent upon advanced informatics applications and will need to be ready to continue to incorporate new technologies.

There is growing recognition of the need to incorporate informatics education in U.S. residencies.[8] Resident physicians are uniquely positioned to contribute to the field of informatics given their daily use of technologies and vested interest in improving current technology for the duration of their careers. Medical residency training programs in the United States are critical in shaping system change and can play a pivotal role by incorporating CI-based learning into their training requirements; previous initiatives have demonstrated success in involving resident physicians in informatics training.[8]

Objectives

We searched the Accreditation Council for Graduate Medical Education (ACGME) Program Requirements effective July 1, 2023, to assess the inclusion of CI topics for all medical residency specialties to assess the relative level of CI knowledge that is expected of residents by the time they reach graduation. We chose to survey both surgical and medical residency programs given that individuals certified in any specialty are eligible for subsequent fellowship in CI.

Methods

Two authors performed independent full-text search queries of 28 informatics-related keywords (e.g., EHR, innovation, database) in the ACGME Program Requirements of 24 different medical specialties. The ACGME Program Requirements specify the core competencies and other standards of quality and education for each specialty and subspecialty, as opposed to the ACGME Common Program Requirements, which outline the basic standards in training and preparing all residents and fellow physicians regardless of specialization. We examined ACGME Program Requirements rather than the ACGME Common Program Requirements in order to analyze variation between CI training across specialties.

Keywords were selected based on concepts listed in the core content outline in the ACGME Program Requirements for Graduate Medical Education in Clinical Informatics as well as examination content for board certification in CI.[9] [10] Two senior authors (V.E.N. and E.K.) with experience in ACGME program leadership and both certified in CI reviewed CI certification examination content topics and the final list of keywords before searches were undertaken.

Concordance between the independent extractions was reviewed based on the preliminary searches, and further validated by senior authors. Contexts of terminology usage were reviewed to ensure relevance to informatics concepts. Descriptive statistics were performed using Microsoft Excel (Microsoft Corporation, Redmond, WA, United States).

Results

All 24 specialties' requirements contained at least five different keywords, with the total count of keywords ranging from 25 to 42 ([Table 1]). The mean number of keywords per specialty was 32.00 (standard deviation: 5.09) and the overall total number of keywords was 768. Pathology contained the highest count of keywords (42) followed by family medicine and internal medicine (each had a total of count of 41). Pathology included the most distinct keywords (each had 11 of the 28 keywords). The most common keywords were “leadership” (473/768, 62%) and “electronic health record” (75/768, 10%). There were no specific mentions of several keywords—including analytics, artificial intelligence, clinical translational research, computational biology, data governance, data organization, interoperability, machine learning, personal health records, and workflow—within any program requirements.

Table 1 Frequency of informatics-related keywords usage in ACGME program requirements for different specialties

	Algorithm	Bioinformatics	Change management	Clinical decision support (CDS)	Coding	Database	Decision-making	Decision support	Electronic health record (EHR)	Informatics	Information systems	Information technology (IT/HIT)	Innovation	Leadership	Technology/technologies	Telehealth/telemedicine	Virtual	Total keyword count
Allergy and immunology	0	0	0	0	0	1	0	0	3	0	0	0	2	21	2	1	0	30
Anesthesiology	1	0	1	0	0	1	1	0	3	0	0	1	2	22	4	1	0	37
Dermatology	0	0	0	0	0	1	0	0	3	0	0	0	2	18	0	1	0	25
Emergency medicine	1	0	0	0	0	2	2	0	3	0	1	2	3	22	3	1	0	40
Family medicine	0	0	0	0	1	1	1	0	4	0	0	0	2	21	4	4	0	42
General surgery	0	0	0	0	0	1	0	0	3	0	0	0	2	22	0	1	0	29
Internal medicine	0	0	0	1	0	1	0	2	3	0	0	0	3	21	6	4	0	41
Medical genetics and genomics	0	2	0	0	0	1	2	0	3	0	0	1	2	21	2	1	0	35
Neurological surgery	0	0	0	0	0	1	0	0	3	0	0	1	2	17	1	1	0	28
Neurology	0	0	0	0	0	1	0	0	3	0	0	0	2	18	2	1	0	27
Nuclear medicine	1	0	0	0	0	1	1	0	3	0	0	1	2	19	3	1	0	29
OBGYN	0	0	0	0	0	1	1	0	4	0	0	1	2	22	3	1	0	31
Ophthalmology	0	0	0	0	0	1	1	0	3	0	0	0	2	17	2	1	2	29
Orthopaedic surgery	0	0	0	0	0	1	0	0	3	0	0	0	2	18	0	1	0	25
Otolaryngology	0	0	0	0	0	1	0	0	3	0	0	1	2	21	4	3	0	31
Pathology	0	0	0	0	0	1	2	0	4	2	3	0	2	25	3	1	2	41
Pediatrics	0	0	0	0	1*	1	1	0	3	0	0	0	2	21	1	1	0	32
Physical medicine and rehabilitation	1	0	0	0	0	1	1	0	3	0	0	0	2	18	2	1	0	32
Plastic surgery	0	0	0	0	0	2	1	0	3	0	0	0	2	17	1	1	0	31
Preventive medicine	0	0	0	0	1*	2	1	0	3	0	1	1	2	21	2	2	1	35
Psychiatry	0	0	0	0	0	1	1	0	3	0	0	0	2	19	1	1	0	28
Radiation oncology	1	0	0	0	0	1	0	0	3	1	0	0	2	16	3	1	0	29
Radiology	0	0	0	0	0	1	0	0	3	3	0	1	2	19	2	1	0	34
Urology	0	0	0	0	0	1	0	0	3	0	0	0	2	17	2	1	1	27
Total	5	2	1	1	3	27	16	2	75	6	5	10	50	473	53	33	6	768

Abbreviations: ACGME, Accreditation Council for Graduate Medical Education; OBGYN, obstetrics and gynecology.

Note: Keywords with no results across all specialties are not shown: analytics, artificial intelligence (AI), clinical translational research, computational biology, data governance, data organization, interoperability, machine learning, personal health record, and workflow.

Discussion

We found several keywords relevant to the field of CI were mentioned in common program requirements for ACGME residency programs. Keywords were often contained in standardized sections shared across program requirements. For example, each ACGME program had a minimum of three mentions of “electronic health records” in regard to work from home requirements. Each program similarly had a statement about providing access to medical literature “databases” as well as “telemedicine” in reference to providing house staff with access to confidential, affordable mental health care. The most common keyword “leadership” was usually referring to residency program leadership, although all programs did acknowledge that “programs may place different emphasis on research, leadership, public health, etc. It is expected that the program aims will reflect the nuanced program-specific goals for it and its graduates[…].”[9]

There were also instances of variable usage of keywords across programs. The term “virtual” was included in four programs in reference to virtual study sets that augment residents' learning in pathology, virtual outbreak or cluster investigations in preventive medicine, virtual educational and interactive sessions in ophthalmology, and virtual curricula on harassment and implicit bias in urology, demonstrating differential application of the term. The term “coding” was included in plastic surgery, pathology, and radiology requirements, and was only included in reference to billing rather than writing computer programs. Similarly, the word “algorithm” was present in five program requirements, wherein it referred to creating a diagnostic algorithm for a scholarly approach to patient care.

We recognize that some of these terms refer to specific topics required to be learned by the end of fellowship but included them in our search to determine which specialties, if any, provide foundational knowledge in these areas. We acknowledge the limitation of our approach, which does not provide a detailed review of individual residency program curricula; however, given the role of program requirements guiding national approaches to residency education, we believe this snapshot provides a valuable baseline assessment of the minimum standards that undergird the current graduate medical education residency training landscape. Furthermore, even with the limitations, the study showed a concerning trend in medical education: despite advances in medical technology, the requirements guiding residency program curricula lag in the incorporation of these innovations into training. For example, the usage of telehealth and virtual care has surged following the COVID-19 pandemic.[11] Yet, when we count all the ways in which the word “virtual” was used, the total number was only six across all examined program requirements.

Although the ACGME residency program requirements state that residents must demonstrate competence in using information technology to optimize learning, the extent to which trainees must do so is not fully specified; only 10 specialties mention the keyword “information technology” within their specific guidelines.[9] Additionally, the integration of CI education varies across specialties and may be even more variable across programs. Based on program requirements, pathology in particular demonstrated high engagement with the field by specifically stating that residents' experiences must include education in CI and participation in the “application of clinical informatics, including hospital, laboratory, pathology information systems; and, quality assurance activities.”[12] Notably, pathology was among the first specialties to attempt creation of a CI subspecialty over 30 years ago.[13] [14] This may play a large role in our study's findings.

Conclusion

The establishment of the Clinical Informatics Subspecialty (CIS) in 2011 and the subsequent accreditation of the first CI fellowship programs in 2014 exemplified the growing importance of CI in medicine.[15] However, despite the creation of a CI subspecialty, foundational knowledge in CI remains essential to all physicians. Our study highlights potential opportunities for further standardization and integration of CI into resident curriculum requirements in order to better prepare future physician workforces for a changing medical landscape. We encourage educators, residency review committees, and national specialty organizations to consider further exploring the incorporation of CI content into residency training program requirements.

As the medical field continues to advance, we suggest future studies approach whether medical education is keeping up through a qualitative lens, for example, by surveying residents on the integration of CI into their education and the forms in which such integration has been accomplished. We have begun further investigation into this field with a deeper exploration of residency program curricula and incorporation of qualitative analyses to better inform educational leaders and policymakers alike in preparing the physician workforce of the future for integration of CI concepts into residency training.

Clinical Relevance Statement

Clinical training programs for physicians play a critical role in shaping the future workforce in preparation for careers that integrate CI. Our analysis of residency program requirements identifies areas of informatics concepts integrated into current requirements across clinical residencies and gaps that exist with regard to program content.

Multiple-Choice Questions

1. Which of the following CI keywords appears most frequently in residency program requirements for ACGME-accredited residency programs?

   • EHRs.

   • Leadership.

   • Telemedicine.

   • Decision support.

   Correct Answer: The correct answer is option b. In our analysis of 28 informatics keywords across all residency program requirements, leadership was the most frequently used keyword. The term accounted for over 60% of all informatics keywords used across programs.

2. Which of the following specialties demonstrated the incorporation of the most keywords relevant to CI in their residency program requirements?

   • Preventive medicine.

   • Anesthesia.

   • Pathology.

   • General surgery.

   Correct Answer: The correct answer is option c. In our analysis of 28 informatics keywords across all residency program requirements, pathology included the most frequent mentions of keywords relevant to CI (42), followed by internal medicine and family medicine (41 each).

Conflict of Interest

None declared.

Protection of Human and Animal Subjects

This study did not involve research with human subjects or animals.

• References

• 1 Gardner RM, Overhage JM, Steen EB. et al; AMIA Board of Directors. Core content for the subspecialty of clinical informatics. J Am Med Inform Assoc 2009; 16 (02) 153-157
• 2 Mai MV, Luo BT, Orenstein EW, Luberti AA. A model for clinical informatics education for residents: addressing an unmet need. Appl Clin Inform 2018; 9 (02) 261-267
• 3 Masters K. Artificial intelligence in medical education. Med Teach 2019; 41 (09) 976-980
• 4 Joint Science Academies. G-Science Academies Statement 2020: Digital Health and the Learning Health System—Executive Summary. 2020. Accessed August 6, 2024 at: https://www.nationalacademies.org/our-work/joint-science-academies-statements-on-global-issues#sl-three-columns-fdcca5c7-9e72-4381-9f8e-70fa4c07d389
• 5 Abernethy A, Adams L, Barrett M. et al. The promise of digital health: then, now, and the future. NAM Perspect 2022; 2022: 10.31478/202206e
• 6 Johnson KB, Neuss MJ, Detmer DE. Electronic health records and clinician burnout: a story of three eras. J Am Med Inform Assoc 2021; 28 (05) 967-973
• 7 Sutton RT, Pincock D, Baumgart DC, Sadowski DC, Fedorak RN, Kroeker KI. An overview of clinical decision support systems: benefits, risks, and strategies for success. NPJ Digit Med 2020; 3 (01) 17
• 8 You JG, Samal L, Leung TI. et al. A call to support informatics curricula in U.S.-based residency education. Appl Clin Inform 2023; 14 (05) 992-995
• 9 Accreditation Council for Graduate Medical Education (ACGME). ACGME Program Requirements for Graduate Medical Education in Anatomic Pathology and Clinical Pathology. 2023. Accessed September 27, 2024 at: https://www.acgme.org/globalassets/pfassets/programrequirements/300_pathology_2023.pdf
• 10 ACGME. Accreditation Council for Graduate Medical Education (ACGME) Common Program Requirements. Common Program Requirements. 2024. Accessed March 2, 2024 at: https://www.acgme.org/programs-and-institutions/programs/common-program-requirements/
• 11 Shaver J. The state of telehealth before and after the COVID-19 pandemic. Prim Care 2022; 49 (04) 517-530
• 12 American Board of Preventive Medicine. Clinical Informatics Content Outline. Accessed June 26, 2024 at: https://www.theabpm.org/become-certified/exam-content/clinical-informatics-content-outline/
• 13 Friedman BA. Informatics as a separate section within a department of pathology. Am J Clin Pathol 1990; 94 (4, suppl 1): S2-S6
• 14 Park S, Parwani AV, Aller RD. et al. The history of pathology informatics: a global perspective. J Pathol Inform 2013; 4: 7
• 15 Silverman HD, Steen EB, Carpenito JN, Ondrula CJ, Williamson JJ, Fridsma DB. Domains, tasks, and knowledge for clinical informatics subspecialty practice: results of a practice analysis. J Am Med Inform Assoc 2019; 26 (07) 586-593

Publication History

Received: 09 April 2024

Accepted: 19 September 2024

Article published online:
25 December 2024