Objective Resident Characteristics Associated with Performance on the Ophthalmic Knowledge Assessment Program Examination

Keywords

OKAP - Ophthalmic Knowledge Assessment Program - oral board examination - Accreditation Council for Graduate Medical Education - ACGME - ACGME milestones - ophthalmology residency - graduate medical education - United States Medical Licensing Examination - USMLE - ABO-WQE - written qualifying examination

Introduction

Attainment of medical knowledge (MK) is one of the primary goals of undergraduate and graduate medical education (GME), and the production of knowledgeable physicians is a central goal of medical schools and residency training programs. Physicians who have more MK are more likely to follow evidence-based care guidelines in both surgical[1] and primary care[2] settings, often leading to better outcomes for patients.[3] Accordingly, the Accreditation Council for Graduate Medical Education (ACGME) lists MK as one of the six core competencies that should be achieved by residency training programs in all fields.[4] Over the past several decades, standardized testing has become a central component of how GME programs assess educational performance among resident physicians of all fields.[5]

The Ophthalmic Knowledge Assessment Program (OKAP) serves as one of the tools by which ophthalmology resident physicians' MK is assessed. It is a 260-question multiple-choice computer-based examination administered by the American Academy of Ophthalmology (AAO) taken during each of the 3 years of ophthalmology residency training. It allows programs to assess residents' knowledge relative to their peers across the country, enabling ongoing assessment of program effectiveness and resident progress.[6] The OKAP also provides a method by which residents who need additional assistance can be identified early on in their training. Both the OKAP and written-qualifying examination (WQE) have been traditionally seen as markers of resident competence, with the WQE representing a significant part of the board certification process typically required for physician credentialing. Oral board examinations as administered by the American Board of Ophthalmology (ABO) are also a crucial part of the credentialing and self-regulation of new ophthalmologists.[7]

There are certain resident and program characteristics that have previously been associated with higher OKAP scores. In a survey of 183 residents from 19 ophthalmology programs by Zafar et al, factors such as increased United States Medical Licensing Examination (USMLE) Step 1 score, increased number of hours spent studying, and decreased number of hours spent doing research in the months preceding the examination were associated with increased OKAP performance.[8] The authors of that study acknowledged that there was a degree of recall and reporter bias in their study due to the fact that OKAP scores and other information was obtained via self-report from residents. Another study found that having a resident-run ophthalmology clinic did not have an impact on OKAP performance.[9] Passage of OKAP has been associated with higher rates of passage on the ABO-WQE in previous studies.[10] [11] [12] In contrast, no previous studies have assessed the association between OKAP performance and oral board exam pass rates.

The aim of the present investigation is to use program-level data from a single institution to investigate resident characteristics that correlate with objective OKAP performance as well as the relationship between OKAP performance and the ABO-WQE and oral examination pass rates. We hope to better delineate the correlation between these examinations as comparable assessments of resident knowledge for residents, program leadership, and national and local stakeholders.

Methods

This study was approved by the Institutional Review Board of the Johns Hopkins University School of Medicine (JHUSOM). Information was obtained via review of records kept by the ophthalmology residency training program at the Wilmer Eye Institute, JHUSOM, which was ranked number 2 in a recent survey published by Ophthalmology Times.[13] Information was gathered in a data sheet using Microsoft Excel (Microsoft Corporation, Redmond, WA) which was stored on a password-protected, encrypted institutional file hosting site. Each resident was provided a unique ID number for deidentification for statistical analysis.

Resident application data were obtained by review of San Francisco (SF) Match application information maintained by program leadership, and demographic information was collected from program registration records. Information on resident marital status and whether they had children was reported by residents to the program administrator at the beginning of residency. Numerical milestones data were extracted from the online Accreditation Data System on the ACGME Web site. Milestones were grouped together by category and numerical averages were taken for each individual category: patient care (PC), MK, systems-based practice (SBP), practice-based learning and improvement (PBLI), professionalism, and interpersonal and communication skills. ACGME milestone MK-1 was excluded from our analysis because it is a direct reflection of OKAP performance and thus would not provide additional value. OKAP percentile rankings, oral board passage rates, and WQE passage rates were also obtained for each individual resident.

Spearman's rank correlation coefficients were determined to quantify the association between OKAP percentile scores over all 3 years of training and continuous variables. Two-sample t-tests were performed between OKAP percentiles and categorical variables. A multivariate linear regression model of average OKAP scores was constructed, then a backward selection procedure using the R function step() was implemented to select an optimal model by Akaike information criterion. Statistical work was accomplished using R (R Foundation for Statistical Computing, Vienna, Austria). For the purposes of our statistical analysis, significance is defined as p < 0.05.

Results

Demographic factors and medical school academic performance metrics obtained from SF Match application and program registration records are demonstrated in [Table 1]. Data were collected from a total of 50 residents from the years 2012 to 2018. In our sample, 42% of residents were male, 48% were married, and 18% had children either before or during residency. The mean USMLE Step 1 score was 247 (standard deviation [SD] 12.23, range 222–273). The mean number of publications prior to residency was 7.33 (5.30, 0–22), with the mean number of first author publications prior to residency of 3.67 (SD 3.03, range 0–17).

Table 1 Demographic characteristics and medical school academic performance metrics of our ophthalmology resident physician cohort

Demographic variable		N (%)
Sex	Male Female	21 (42%) 29 (48%)
Marital status	Married Unmarried	24 (48%) 26 (52%)
Children during residency	Yes No	9 (18%) 41 (82%)
MD/PhD degree	Yes No	9 (18%) 41 (82%)
Other degree (i.e., MPH, MBA)	Yes No	11 (22%) 39 (78%)
Medical school rotation	Grade	N (%)
Surgery rotation grade	Honors High pass Pass	25 (57%) 17 (37%) 2 (5%)
Medicine rotation grade	Honors High pass Pass	24 (52%) 17 (37%) 5 (11%)
Academic performance variable (continuous)	Mean (SD)	Range
No. of publications prior to residency	7.33 (5.30)	0–22
No. of first author publications	3.67 (3.03)	0–17

Abbreviation: SD, standard deviation.

Annual trends in OKAP scores stratified by year of training are seen in [Fig. 1]. Mean (SD) OKAP percentile performance was 60.90 (27.51), 60.46 (28.12), and 60.55 (27.43) in the first, second, and third year examinations, respectively ([Table 2]). We stratified residents into three groups based on first-year OKAP scores (0–33rd, 34th–66th, and 67th–100th percentiles) and tracked how their OKAP performance varied over subsequent examinations ([Table 2]). We found that on average, those with lower first-year OKAP performance experienced the greatest absolute change in their scores in subsequent years. Those in our cohort who scored between 0 and 33rd percentiles during their first-year OKAP exam experienced an average increase of 9.6 percentile points in their second-year examination, and 18.3 percentile points between their first- and third-year examinations. In contrast, those who scored between 34th to 66th and 67th to 100th percentiles on their first-year examination experienced an average decline in their third-year examination scores (7.7 percentile decrease and 10.5 percentile decrease, respectively).

Fig. 1 Average Ophthalmic Knowledge Assessment Program (OKAP) percentile scores from 2012 to 2018, stratified by resident training year.

Table 2 Trends in yearly performance on OKAP examinations among our ophthalmology resident cohort

OKAP examination year	Percentile score		Year 1 percentile	Average absolute change in OKAP percentile
	Mean (SD)	Range		Year 1–year 2	Year 2–year 3	Year 1–year 3
Year 1	63.83 (26.24)	2–100	0–33rd	9.6	8.8	18.3
Year 2	58.07 (26.42)	6–99	34th–67th	−8.3	1.6	−7.7
Year 3	57.30 (26.97)	7–99	67th–100th	−7.9	-2.4	−10.5

Abbreviations: OKAP, Ophthalmic Knowledge Assessment Program; SD, standard deviation.

Note: Average OKAP scores are seen on the left panel, while average absolute change in OKAP score stratified by year 1 examination score is on the right panel.

OKAP scores were significantly associated with USMLE Step 1 scores in our univariate linear regression model (0.97 [0.73, 1.21], p = 0.008, [Supplementary Fig. S1]). On multivariate linear regression, USMLE Step 1 score was the only covariate that was significantly associated with OKAP scores (coefficient 0.88, 95% confidence interval [0.72, 1.22], p = 0.008). In this multivariate linear regression model, OKAP scores were associated with male sex (1.02 [−6.09, 8.13], p = 0.88), being married (1.85 [−5.64, 9.34], p = 0.81), and having a child (0.77 [−10.46, 11.98], p = 0.94), while having an advanced degree (−0.49 [−9.55, 8.57], p = 0.96) was negatively associated with OKAP performance; however, these observations did not achieve statistical significance ([Table 3]). Additionally, grades in the medicine (4.56 [−2.23, 11.35], p = 0.51) and surgery (0.34 [−6.33, 7.01], p = 0.96) clerkships were not significantly associated with OKAP performance in the multivariate model. This model selected by the stepwise algorithm was a simple linear regression model with USMLE Step 1 score as the only covariate.

Table 3 Associations between ophthalmology resident characteristics and continuous OKAP scores

Multivariate linear regression analysis
Characteristics	Coefficient (95% CI)	p-Value
Male (reference male)	1.02 (−6.09, 8.13)	0.888
Married (reference unmarried)	1.85 (−5.64, 9.34)	0.806
Has children (reference no children)	0.77 (−10.46, 11.98)	0.946
Other advanced degrees (reference no)	-0.49 (−9.55, 8.57)	0.957
USMLE step 1 score	0.88 (0.54, 1.18)	0.008
Medicine grade (reference P/HP)	4.56 (−2.23, 11.35)	0.506
Surgery grade (reference P/HP)	0.34 (−6.33, 7.01)	0.959

Abbreviations: CI, confidence interval; HP, high pass; OKAP, Ophthalmic Knowledge Assessment Program; P, pass; USMLE, United States Medical Licensing Examination.

Note: Bold indicates statistical significance.

Unadjusted linear regression analysis of OKAP scores and ACGME milestone data yielded a statistically significant association between higher first-year OKAP performance and mean PBLI score (odds ratio 5.08 [1.17–22.04], p = 0.030; [Supplementary Table S1]). There was also a significant difference in mean SBP score and higher mean OKAP performance in our univariate analysis when accounting for male sex (β coefficient 0.39 [0.002–0.79], p = 0.049); this difference lost statistical significance in our multivariate model (β coefficient 0.46, p = 0.055; [Supplementary Table S2]).

Spearman's rank correlation testing ([Table 4]) found that USMLE Step 1 scores were more strongly correlated with OKAP performance (r _s = 0.498, p = 0.0003) than other variables such as number of first author publications (r _s = 0.235, p = 0.108) and number of total publications (r _s = 0.172, p = 0.241).

Table 4 Associations between medical school academic performance variables and OKAP scores

Medical school research productivity
	Spearman's rank correlation coefficient	p-Value
Number of publications	0.172	0.241
Number of first author publications	0.235	0.108
USMLE Step 1 score
	Spearman's rank correlation coefficient	p-Value
	0.498	0.0003

Abbreviations: OKAP, Ophthalmic Knowledge Assessment Program; USMLE, United States Medical Licensing Examination.

OKAP scores were significantly correlated with passage of the WQE (r _s = 0.292, p = 0.049) and oral board examination (r _s = 0.49, p = 0.001).

Discussion

We identified specific objective resident factors that are associated with success on the OKAP examinations. Our study demonstrated statistically significant associations between OKAP performance and passage rates of the ABO oral board examination. This finding compliments associations between OKAP performance and WQE scores that have been demonstrated previously.[10] [11] [12]

The format of the oral board is unique in that it is not a multiple-choice examination such as the OKAP or WQE; rather, it is an in-person, timed evaluation comprising six 25-minute “miniexaminations” where patient management problems are presented by volunteer ophthalmologists and examinees are asked to describe their plan of care for that patient.[14] Certain residency training programs host some form of an internal mock oral board examination for residents to help them prepare. The fact that annual OKAP examination scores correlate with oral board examination scores is encouraging for continued use of the OKAP as an evaluation tool for ophthalmology residents, as it demonstrates that the examination likely prepares residents well for the medical knowledge expected of them during the oral board.

There were no significant associations observed in our cohort when analyzing various demographic factors such as additional degrees (PhD, MPH, MBA, etc.), parenthood, or sex. Regarding resident application factors that we studied, USMLE Step 1 score remains an important variable associated with performance on the OKAP examination on both an annual and average basis. Our findings reinforce the association demonstrated in a previous study by Zafar et al,[8] adding to this previous investigation in that our data included scores that were objectively reported in SF Match application data (as opposed to self-reported). The correlation between USMLE Step 1 score and residency examinations has been demonstrated in other specialties such internal medicine, where USMLE Step 1 performance was associated with success on annual in-service training examination scores as well as passage of the American Board of Internal Medicine certifying examination.[15] A similar association was demonstrated in a study of general surgery residents regarding first-time board pass rates and USMLE Step 1 scores.[16]

The USMLE's recent decision to transition from three-digit scores to pass/fail status on the Step 1 examination[17] will undoubtedly change the utility of this factor in identifying residents who may have more difficulty with the OKAP examinations. It has been posited in preliminary commentary that this decision will eventually lead to USMLE Step 2 clinical knowledge (CK) examination scores increasing in prominence for residency selection committees across the board.[18] Since ophthalmology is currently an early match process occurring in mid-January as opposed to mid-March, applicants often do not have Step 2 CK scores included in their initial SF Match application; a 2011 study found that 46% of applicants did not have this information on their application.[19] We collected data on USMLE Step 2 CK scores that were available in the historical SF Match application files of our study cohort. Only 15 of our 50 residents (30%) reported USMLE Step 2 CK scores on their SF Match application, and thus, we were unable to perform meaningful statistical analysis using this variable. Future investigations will need to analyze both the likely increase in prevalence of USMLE Step 2 CK scores on SF Match applications as well as how the results of that examination correlate with measures of resident performance.

Medical school grades were not significantly associated with OKAP performance. Differences in reporting of clerkship grades have been demonstrated previously,[20] [21] as they are composed of a variety of factors such as subjective clinical evaluations, in-house clinical skills performance examinations, as well as the end of clerkship shelf examination administered by the National Board of Medical Examiners (NBME). Medical schools place different weights on the importance of the NBME examination in determination of students' final clerkship grades[22]; some institutions simply require passage of the examination, while others require a certain national percentile cutoff score to be considered for the highest grade of honors. Furthermore, there is variability in the grading scale used by medical schools in assessment of students: some report letter grades (A, B, C, etc.), others use the honors/high pass/pass scale, others use a pass/fail only system, and other schools use numerical scores. Some of our residents were from medical schools outside of the United States, where the honors/high pass/pass system is not as common. Thus, a limitation of the present investigation is that we included residents who scored on the honors, high pass, pass system only as there exists no adequate way to exchange between the different grading scales to our knowledge.

There are several important limitations to consider with our study. First, it is a single-institution study of residents from an academic medical center setting. The average OKAP scores in our cohort were in the 60th percentile for each examination year, which is above the mean score of 50th percentile as determined in the AAO guidelines for the examination.[23] Thus, the trends we observed may not be representative of ophthalmology residents in all settings throughout the United States, although our cohort had acceptable balance by sex and an average USMLE Step 1 score (247) aligned with recent published averages for matched ophthalmology residents (247).[24] The cohort also had a first-time pass rate on the WQE and oral board examinations (100 and 93.8%, respectively) that is higher than the national averages,[25] providing a source of bias in our inferences regarding associations between OKAP scores and WQE and oral board pass rates. Both the oral board examination[26] and WQE[27] use criterion-referenced passing standard (i.e., are not graded on a curve), meaning that it is possible for all examinees to pass each examination if they score at or above that passing standard.

Information on the SF Match application—aside from USMLE scores and clerkship grades—are self-reported by residents at the time of their application to our program. This could also contribute to biases in the data (i.e., potential overreporting of publications in process at the time of application as opposed to already published). Another limitation is that our dataset included variables limited to the resident experience alone, so our measures cannot be extrapolated to how residents perform as practicing physicians after residency. Future directions for this work include expanding our data collection to additional academic centers to explore the relationship in a larger cohort.

Conclusion

OKAP scores are predictive for passing both written and oral board examinations on the first attempt. There is a correlation between higher OKAP scores and success on the oral board examination, and USMLE Step 1 scores are associated with success on the OKAP examination. As a whole, the findings from our study can be used to guide program directors as they seek to holistically evaluate residents' anticipated performance on the OKAP examinations, with the goal of proactively connecting residents to additional resources at an earlier stage to prepare for these examinations.

• References

• 1 Richardson DP, Porter GA, Johnson PM. Surgeon knowledge contributes to the relationship between surgeon volume and patient outcomes in rectal cancer. Ann Surg 2013; 257 (02) 295-301
• 2 Tamblyn R, Abrahamowicz M, Brailovsky C. et al. Association between licensing examination scores and resource use and quality of care in primary care practice. JAMA 1998; 280 (11) 989-996
• 3 Tamblyn R, Abrahamowicz M, Dauphinee WD. et al. Association between licensure examination scores and practice in primary care. JAMA 2002; 288 (23) 3019-3026
• 4 Holmboe ES, Edgar L, Hamstra S. The Milestones Guidebook. 2016 . Accessed March 18, 2020 at: https://www.acgme.org/Portals/0/MilestonesGuidebook.pdf?ver=2016-05-31-113245-103
• 5 Epstein RM. Assessment in medical education. N Engl J Med 2007; 356 (04) 387-396
• 6 American Academy of Ophthalmology. OKAP Exam. 2020 . Accessed March 18, 2020 at: https://www.aao.org/okap-exam
• 7 Hamming NA, Kline LB, Keltner JC, Orcutt JC, Farber MJ. History of the American Board of Ophthalmology Oral Examination. Ophthalmology 2016; 123 (9, Suppl): S20-S24
• 8 Zafar S, Wang X, Srikumaran D. et al. Resident and program characteristics that impact performance on the Ophthalmic Knowledge Assessment Program (OKAP). BMC Med Educ 2019; 19 (01) 190
• 9 Singman EL, Boland MV, Tian J, Green LK, Srikumaran D. Writing Committee of the Ophthalmology Program Directors' Study Group (OPDSG). Supervision and autonomy of ophthalmology residents in the outpatient clinic in the United States II: a survey of senior residents. BMC Med Educ 2019; 19 (01) 202
• 10 Lee AG, Oetting TA, Blomquist PH. et al. A multicenter analysis of the Ophthalmic Knowledge Assessment Program and American Board of Ophthalmology written qualifying examination performance. Ophthalmology 2012; 119 (10) 1949-1953
• 11 Johnson GA, Bloom JN, Szczotka-Flynn L, Zauner D, Tomsak RL. A comparative study of resident performance on standardized training examinations and the American Board of Ophthalmology written examination. Ophthalmology 2010; 117 (12) 2435-2439
• 12 Chen PP, Bhandari A. Resident performance on the Ophthalmic Knowledge Assessment Program and the written qualifying examination for board certification. J Acad Ophthalmol. 2010; 3 (01) 11-14
• 13 Ophthalmology Times. Bascom Palmer tops in annual program survey. October 4, 2019. Accessed August 11, 2020 at: https://www.ophthalmologytimes.com/view/bascom-palmer-tops-annual-program-survey
• 14 American Board of Ophthalmology. Oral Examination. 2020 . Accessed June 2, 2020 at: https://abop.org/become-certified/oral-exam/
• 15 Kay C, Jackson JL, Frank M. The relationship between internal medicine residency graduate performance on the ABIM certifying examination, yearly in-service training examinations, and the USMLE Step 1 examination. Acad Med 2015; 90 (01) 100-104
• 16 Sutton E, Richardson JD, Ziegler C, Bond J, Burke-Poole M, McMasters KM. Is USMLE Step 1 score a valid predictor of success in surgical residency?. Am J Surg 2014; 208 (06) 1029-1034 , discussion 1034
• 17 United States Medical Licensing Examination. “Change to pass/fail score reporting for Step 1. Accessed April 2, 2020 at: https://www.usmle.org/inCus/
• 18 Chaudhry HJ, Katsufrakis PJ, Tallia AF. The USMLE Step 1 decision: an opportunity for medical education and training. JAMA 2020; 323 (20) 2017-2018
• 19 Grubbs Jr JR, Mian SI. Advising students interested in ophthalmology: a summary of the evidence. Ophthalmology 2016; 123 (07) 1406-1410
• 20 Brownstein MR, Shen MR, Strassle PD, Meyers MO. Variation of surgery clerkship grades in US medical schools. Am J Surg 2019; 217 (02) 329-334
• 21 Westerman ME, Boe C, Bole R. et al. Evaluation of medical school grading variability in the United States: are all honors the same?. Acad Med 2019; 94 (12) 1939-1945
• 22 Schilling DC. Using the clerkship shelf exam score as a qualification for an overall clerkship grade of honors: a valid practice or unfair to students?. Acad Med 2019; 94 (03) 328-332
• 23 American Academy of Ophthalmology. OKAP User's Guide. 2020 . Accessed August 11, 2020 at: https://www.aao.org/Assets/d2fea240-4856-4025-92bb-52162866f5c3/637278171985530000/user-guide-2020-pdf
• 24 Association of University Professors of Ophthalmology. Ophthalmology Residency Match Summary Report. 2020 . Accessed March 23, 2020 at: https://sfmatch.org/PDFFilesDisplay/Ophthalmology_Residency_Stats_2020.pdf
• 25 American Board of Ophthalmology. Certification Statistics. Accessed August 11, 2020 at: https://abop.org/about/certification-statistics/
• 26 American Board of Ophthalmology. 1.5 Scoring Procedures. In: American Board of Ophthalmology Virtual Online Exam 2020 Candidate Procedures Manual. 2020 . Accessed June 2, 2020 at: https://www.manula.com/manuals/american-board-of-ophthal/abo-voe20-candidate-procedures-manual/1/en/topic/scoring-procedures
• 27 American Board of Ophthalmology. Written Qualifying Examination. In: Become Certified. 2020 . Accessed June 2, 2020 at: https://abop.org/become-certified/written-qualifying-exam/