Ophthalmology Residency Program Director Survey on Pass/Fail U.S. Medical Licensing Exam Step 1 Scoring

Abstract

Background Beginning January 26, 2022, the U.S. Medical Licensing Exam (USMLE) Step 1 changed from a numerical score to pass/fail (P/F). The purpose of this study was to determine the perspective of ophthalmology program directors regarding this change in evaluating applicants.

Methods After institutional review board approval, a survey was sent out to program directors of all 125 ophthalmology programs accredited by the Accreditation Council for Graduate Medical Education. Survey questions asked for program demographics, the utility of USMLE Step 1 and 2 Clinical Knowledge scores in assessing applicants, and the importance of 16 different applicant metrics before and after Step 1 becomes P/F. The metrics examined were: letters of recommendation; clerkship grades; class ranking; Alpha Omega Alpha Membership; Gold Humanism Honor Society Membership; Dean's Letter; involvement and leadership; personal statement; number of abstracts, presentations, and publications; mean number of research experiences in the specialty; Step 2 Clinical Knowledge score; volunteering; preclinical grades; away rotation in the specialty; the applicant having another graduate degree; and graduation from a top 40 National Institutes of Health-funded program. Data were analyzed using nonoverlapping 95% confidence intervals.

Results The survey was completed by 50 (40%) program directors. Sixty-eight percent of respondents stated a student's ranking would be considered more after USMLE Step 1 scores become P/F, and 60% stated medical schools should share clerkship shelf exam scores with residency programs. There were no significant differences in program directors' rankings of applicant metrics following the transition to P/F Step 1.

Conclusion Based on our data, program directors will likely not place a greater emphasis on Step 2 scores, despite it being the only remaining objective measure for all applicants following the switch to a P/F Step 1. Nevertheless, program directors expressed an interest in receiving other objective measures, such as shelf exam scores and class ranking, as part of the application process. Notably, we found no significant changes in the rankings of various applicant metrics before and after the transition to P/F Step 1, indicating that the metrics that were important to program directors prior to the change remain just as critical in the new era of admissions.

Keywords

ophthalmology residency - USMLE Step 1 - pass/fail

Background

The U.S. Medical Licensing Examination (USMLE) is the U.S. medical licensing program required of all physicians to receive their licenses and begin practicing. In February of 2020, the National Board of Medical Examiners (NBME) announced a change in USMLE Step 1 scoring from a 3-digit numeric score to pass/fail (P/F). Although the primary purpose of the USMLE Step 1 exam is to ensure medical licensing, it has been widely used by residency application committees as a primary screening tool. This is due to Step 1 scores being one of the few objective, nationally recognized metrics that allow for comparison of students from different medical schools. The heavy reliance on Step 1 scores in residency selection has driven a culture of prioritizing Step 1 to maximize a student's chances of matching. In the field of ophthalmology, 498 out of 677 total applicants matched into ophthalmology residency programs in 2021, a match rate of 74%. Matched applicants had an average USMLE Step 1 score of 245, whereas unmatched applicants had an average score of 238. In a study by Loh et al, which analyzed the characteristics of matched ophthalmology applicants between 2003 and 2008, it was found that a student's Step 1 score was a statistically significant predictor of matching, with each 10-unit increase in score associated with a 60% increase in odds of matching.[1] The increased significance of Step 1 scores has contributed to a general sense of heightened anxiety among medical students.[2]

While the USMLE Step 1 score assesses an applicant's ability to pass a standardized test, it may not necessarily reflect other qualities important for physicians. For instance, the exam is limited in its assessment of critical thinking skills, communication skills, or the ability to work well in a team. In fact, multiple studies have shown that Step 1 scores are not good predictors of clinical performance or residency success.[3] [4] Although there are a few studies that have found a correlation between Step 1 scores and board scores, the evidence for the strength of this correlation remains questionable.[5] [6] By contrast, there appears to be a stronger correlation between Step 2 scores and performance on boards exams.[7] However, even with USMLE Step 2 exams, the correlation between scores and residency performance remains uncertain.[8]

To address these concerns, the Federation of State Medical Boards (FSMB) and the NBME partnered with the American Medical Association, the Association of American Medical Colleges, and the Educational Commission for Foreign Medical Graduates to convene a summit, the Invitational Conference on USMLE Scoring. Following discussions at the summit and feedback solicited on a website, the FSMB and NBME made the decision to transition to P/F scoring. According to the Summary Report from the summit, the transition to P/F was made to “improve examinee and physician well-being, improve the reliability of assessments for the purpose of advancing the health of the public, maintain the quality and integrity of the U.S. medical licensure system, and promote holistic review of residency applicants, aligning with goals for diversity in medicine.”[9]

Although the rationale for this change is well intentioned, whether it achieves its aims remains unclear, and the impact of the change on the residency application process remains a larger mystery. With one of the major screening tools no longer available, residency programs must stratify hundreds of applicants with limited comparable data. Conversely, applicants have poor insight into their own attractiveness as an applicant without this measure. Thus, the present study assesses the perspectives of ophthalmology residency program directors in evaluating applicants with a P/F Step 1 score.

Methods

Prior to initiation of the study, institutional review board (IRB) approval was obtained (IRB-300007220). The current study is part of a larger, parent study examining the perspectives of allopathic residency program directors of 25 different specialties.[10] A 14-item survey was developed using Qualtrics and distributed via email. The anonymous survey consisted of five demographic questions, seven questions on the importance of USMLE Step 1 and Step 2 Clinical Knowledge (CK), and two questions asking participants to rank the importance of 16 different applicant metrics. The metrics examined were: letters of recommendation; clerkship grades; class ranking; Alpha Omega Alpha Membership; Gold Humanism Honor Society Membership; Dean's Letter; involvement and leadership; personal statement; number of abstracts, presentations, and publications; mean number of research experiences in the specialty; Step 2 CK score; volunteering; preclinical grades; away rotation in the specialty; the applicant having another graduate degree; and graduation from a top 40 National Institutes of Health (NIH)-funded program. The seven multiple choice questions included in the survey aim to gain a better understanding of program directors' perceptions of the predictiveness of the USMLE Step 1 and Step 2 CK exams, as well as the implications of the change to P/F Step 1 in assessing applicants. The full survey can be found in the supplementary materials ([Supplementary Material S1] [available in the online version]). After IRB approval was obtained, we distributed the survey to program directors and coordinators of all 125 ophthalmology residency programs accredited by the Accreditation Council for Graduate Medical Education. Contact information for program directors and program coordinators was generated from publicly available information. Over the course of 6 weeks, two requests for participation were sent via email. Data were analyzed using nonoverlapping 95% confidence intervals.

Results

The survey was completed by 50 of 125 (40%) participants. The majority (86%) were completed by program directors, 12% were from associate program directors, and 2% were from assistant program directors. Sixteen percent responded that they were from a top 15 NIH-funded program, 66% were not, and the remaining respondents did not know. The majority of respondents were part of an academic institution (90%), whereas 8% were from academic-affiliated institutions, and 2% were from community hospital systems.

Of all survey responses received, 68% (p < 0.05) of respondents stated that a student's medical school rank would be considered more after USMLE Step 1 scores are P/F, and 60% (p < 0.05) stated that medical schools should share clerkship NBME shelf exam scores with residency programs. Sixty-four percent (p < 0.05) believed Step 1 scores were able to adequately predict a resident's ability to pass the specialty's board exams, but 20% believed Step 1 could predict a resident's ability to perform clinically in ophthalmology. Similarly, 22% believed USMLE Step 2 CK scores accurately predict a resident's ability to perform in ophthalmology. Sixty-eight percent of respondents believe students will not be better prepared clinically following a transition to Step 1 becoming P/F (p < 0.05; [Table 1]).

Table 1 Ophthalmology program directors responses to pass/fail Step 1

	Yes	Neutral	No
Question	Percent (95% confidence interval)
After USMLE Step 1 becomes pass/fail, should medical schools share clerkship NBME shelf exam scores with residency programs?	60 (45.2–73.3)*	30 (18.3–44.8)	10 (3.7–22.6)
Do you believe that USMLE Step 1 scores adequately predict a resident's ability to pass your specialty's board exams?	64 (49.1–76.7)*	24 (13.5–38.5)	12 (5.0–25.0)
Do you believe that USMLE Step 2 CK scores adequately predict a resident's ability to pass your specialty's board exams?	32 (19.9–46.8)	46 (32.1–60.5)	22 (12.0–36.3)
Do you believe that USMLE Step 1 scores accurately predict a resident's ability to perform clinically in your specialty?	20 (10.5–34.1)	30 (18.3–44.8)	50 (35.7–64.3)
Do you believe that USMLE Step 2 CK scores accurately predict a resident's ability to perform clinically in your specialty?	22 (12.0–36.3)	48 (33.9–62.4)	30 (18.3–44.8)
Will a student's medical school rank be considered more after USMLE Step 1 becomes pass/fail?	68 (53.2–80.1)*	14 (6.2–27.4)	18 (9.0–31.9)
After USMLE Step 1 becomes pass/fail, do you believe students will be better prepared clinically?	4 (0.7–14.9)	28 (16.7–42.7)	68 (53.2–80.1)*

Abbreviations: CK, Clinical Knowledge; NBME, National Board of Medical Examiners; USMLE, U.S. Medical Licensing Examination.

^* denotes Statistical significance, p < 0.05.

Furthermore, the change to P/F Step 1 did not seem to affect the importance of different applicant metrics. The most important factor P/F Step 1 was letters of recommendation and after P/F Step 1 was clerkship grades. However, the results were not found to be statistically significant. The least important factor both before and after P/F Step 1 was graduation from a top 40 NIH-funded program ([Table 2]).

Table 2 Rankings of applicant metrics before and after pass/fail Step 1

	Ranking before Step 1 P/F	Ranking after Step 1 P/F
Variable	Percent (95% confidence interval)
Step 1 score	4.65 (3.66–5.64)	–
Letters of recommendation	4.50 (3.45–5.55)	4.25 (3.26–5.24)
Clerkship grades	4.85 (3.72–5.98)	4.20 (3.10–5.30)
Class rank	5.68 (4.50–6.85)	5.48 (4.39–6.56)
Alpha Omega Alpha Member	6.35 (5.27–7.43)	5.98 (4.93–7.02)
Gold Humanism Honor Society Member	7.85 (6.58–9.12)	7.03(5.85–8.20)
Dean's Letter	8.18 (6.84–9.51)	7.10 (5.79–8.41)
Involvement and leadership	8.28 (6.92–9.63)	8.08 (6.86–9.29)
Personal statement	8.73 (7.38–10.07)	8.08 (6.74–9.41)
Abstracts, presentations, and publications	9.55 (8.45–10.65)	9.05 (8.07–10.03)
Mean # research experiences in specialty	10.03 (9.29–10.76)	9.28 (8.51–10.04)
Step 2 CK score	10.10 (8.53–11.67)	7.28 (5.72–8.83)
Volunteering	10.28 (9.12–11.43)	9.33 (8.33–10.32)
Preclinical grades	12.40 (11.43–13.37)	11.58 (10.61–12.54)
Away rotation in specialty	13.13 (11.98–14.27)	12.45 (11.56–13.34)
Applicant has another graduate degree	14.10 (13.01–15.19)	13.38 (12.36–14.39)
Graduated from one of the top 40 NIH-funded programs	14.38 (12.94–15.81)	13.50 (12.14–14.86)

Abbreviations: CK, Clinical Knowledge; NIH, National Institutes of Health; P/F, pass/fail.

Discussion

Results from this survey suggest that ophthalmology program directors may prioritize other objective measures, such as NBME shelf exam scores and class rank, following the transition to P/F Step 1. Interestingly, the plurality believe that Step 1 scores do not accurately predict a resident's ability to perform in ophthalmology, but the majority believe that students will not be better prepared clinically. The majority also believe that Step 1 scores could accurately predict whether or not a resident will pass the ophthalmology board exams. Most were ambivalent as to whether or not USMLE Step 2 scores could predict a resident's ability to pass the ophthalmology board exams and perform clinically in ophthalmology. These data suggest that ophthalmology program directors believe preparation for Step 1 better prepared students for clinical rotations but not specifically for the field of ophthalmology. Our data illustrate the contrast between the practical application of medicine in clinical settings and the theoretical knowledge assessed on board exams.

According to our data, it also appears that most ophthalmology program directors responding to this survey are unlikely to alter their perceptions of the relative importance of various applicant metrics following the transition of Step 1 to P/F scoring. None of the 16 different metrics saw a significant change in their ranking of importance in evaluating applicants. Even Step 2 CK, the only remaining standardized measure, did not have a significant increase in importance before and after the transition to P/F Step 1.

Over the last decade, the mean number of applications submitted by each student applying into ophthalmology has dramatically increased from 52 in 2011 to 79 in 2021.[11] The increasing number of medical students applying to residencies poses a range of challenges for residency program directors and admissions committees in order to conduct a holistic review of each applicant. As the number of applicants grows, the time and resources required to review each application in detail become increasingly prohibitive. This can result in a more superficial evaluation process that relies heavily on objective metrics, such as board scores and grades, rather than a more comprehensive assessment of an applicant's clinical skills, personal qualities, and fit with the program.

Other factors that can influence the selection process include the reputation of the medical school, geographic location, and personal connections. These factors can create disparities in the selection process and make it more difficult for deserving applicants from less prestigious schools or underrepresented backgrounds to secure a residency position. Although there is a recent trend in medical schools withdrawing from the annual U.S. News and World Report rankings, many of the affiliated hospitals remain on the U.S. News “Best Hospitals” ranking. Thus, the effect of removing medical school ranking on students' decisions and program directors' opinions of students remains to be seen.

Additionally, medical students applying into ophthalmology face several challenges when it comes to evaluating their competitiveness for the specialty. One of the main challenges is the lack of standardized data and resources available for applicants. Unlike other specialties, ophthalmology is not a part of the Electronic Residency Application Service or the National Resident Matching Program (NRMP), two systems used in the match process for all other specialties except ophthalmology and urology. Instead, ophthalmology utilizes the San Francisco (SF) Match. One major advantage for students applying to specialties in the NRMP is the Program Director Survey. The program director survey queries program directors in each specialty as to what selection factors are important in selecting applicants for interviews. The NRMP Program Director Survey releases information on a wide range of applicant metrics, including but not limited to, board scores, class rank, clerkship grades, membership in national honor societies, letters of recommendation, personal statements, extracurricular activities, and research involvement. By contrast, the match data released by SF Match are limited to a few applicant characteristics and USMLE Step 1 scores. Thus, there is an inherent lack of transparency in what factors ophthalmology residency programs value in medical students.

Although the change to P/F Step 1 was made with the intention to reduce anxiety among medical students and improve well-being, there may be unintended effects of increased worry among students as they may feel uncertain about their competitiveness and likelihood of matching. There is also concern that the focus will now shift to Step 2 CK, which remains a scored assessment.

This study contains several limitations that must be considered. First, the study is limited by its response rate. Only 50 of the 125 ophthalmology program directors completed the survey. Second, participation in the survey was voluntary. Thus, program directors with stronger opinions with regard to the change to P/F reporting may have been more likely to complete the survey. Additionally, our study did not have any measures in place to prevent a participant from responding twice or to prevent a program director and associate program director from the same program from responding. However, given the uniqueness of each response received, we believe the likelihood of an individual responding twice is relatively low. Finally, our survey asks participants to rank factors that are provided and does not account for other possible factors that program directors may consider when selecting applicants. Despite these limitations, our study provides the first look into what ophthalmology program directors will prioritize in the era of binary Step scoring.

Conclusion

In the past, USMLE Step 1 scores have played a crucial role in the selection process for residency programs. However, with the recent shift to a P/F reporting system, there are significant implications for how medical students should approach the application process. Our study reveals that while many ophthalmology program directors may prefer additional objective measures, such as class rank and shelf exam scores, there is no significant increase in the emphasis placed on Step 2 CK scores. It remains to be seen whether this change will achieve its intended effects for medical students, and further research is needed to assess its impact.

Conflict of Interest

None declared.

• References

• 1 Loh AR, Joseph D, Keenan JD, Lietman TM, Naseri A. Predictors of matching in an ophthalmology residency program. Ophthalmology 2013; 120 (04) 865-870
• 2 Chaudhry HJ, Katsufrakis PJ, Tallia AF. The USMLE Step 1 Decision: An Opportunity for Medical Education and Training. JAMA 2020; 323 (20) 2017-2018
• 3 McGaghie WC, Cohen ER, Wayne DB. Are United States medical licensing exam Step 1 and 2 scores valid measures for postgraduate medical residency selection decisions?. Acad Med 2011; 86 (01) 48-52
• 4 Wagner JG, Schneberk T, Zobrist M. et al. What predicts performance? A multicenter study examining the association between resident performance, rank list position, and United States medical licensing examination Step 1 scores. J Emerg Med 2017; 52 (03) 332-340
• 5 Kay C, Jackson JL, Frank M. The relationship between internal medicine residency graduate performance on the ABIM certifying examination, yearly in-service training examinations, and the USMLE Step 1 examination. Acad Med 2015; 90 (01) 100-104
• 6 Harmouche E, Goyal N, Pinawin A, Nagarwala J, Bhat R. USMLE scores predict success in ABEM initial certification: a multicenter study. West J Emerg Med 2017; 18 (03) 544-549
• 7 Sharma A, Schauer DP, Kelleher M, Kinnear B, Sall D, Warm E. USMLE Step 2 CK: best predictor of multimodal performance in an internal medicine residency. J Grad Med Educ 2019; 11 (04) 412-419
• 8 Shirkhodaie C, Avila S, Seidel H, Gibbons RD, Arora VM, Farnan JM. The association between USMLE Step 2 Clinical Knowledge scores and residency performance: a systematic review and meta-analysis. Acad Med 2023; 98 (02) 264-273
• 9 Summary Report and Preliminary Recommendations from the Invitational Conference on USMLE Scoring (InCUS), March 11–12, 2019. Accessed December 12, 2022. https://www.usmle.org/sites/default/files/2021-08/incus_summary_report.pdf
• 10 Patel OU, Haynes WK, Burge KG. et al. Results of a national survey of program directors' perspectives on a pass/fail US medical licensing examination Step 1. JAMA Netw Open 2022; 5 (06) e2219212
• 11 Markle JC, Ahmed H, Pandya K. et al. Transparency in the ophthalmology residency match: background, study, and implications. Cureus 2021; 13 (11) e19826

Publication History

Received: 13 January 2023

Accepted: 12 June 2023

Article published online:
14 November 2023