The Bethesda ERCP Skills Assessment Tool (BESAT) can reliably differentiate endoscopists of different experience levels

Abstract

Background and study aims The Bethesda ERCP Skill Assessment Tool (BESAT) is a video-based assessment tool of technical endoscopic retrograde cholangiopancreatography (ERCP) skill with previously established validity evidence. We aimed to assess the discriminative validity of the BESAT in differentiating ERCP skill levels.

Methods Twelve experienced ERCP practitioners from tertiary academic centers were asked to blindly rate 43 ERCP videos using the BESAT. ERCP videos consisted of native biliary cannulation and sphincterotomy and were recorded from 10 unique endoscopists of various ERCP experience (from advanced endoscopy fellow to > 10 years of ERCP experience). Inter-rater reliability, discriminative validity, and internal structure validity were subsequently assessed.

Results The BESAT was found to reliably differentiate between endoscopists of varying levels of ERCP experience with experienced ERCPists scoring higher than novice ERCPists in 11 of 13 (85%) instrument items. Inter-rater reliability for BESAT items ranged from good to excellent (intraclass correlation range: 0.86 to 0.93). Internal structure validity was assessed with item-total correlations ranging from 0.53 to 0.83.

Conclusions Study findings demonstrate that the BESAT, a video-based ERCP skill assessment tool, has high inter-rater reliability and has discriminative validity in differentiating novice from expert ERCP skill. Further investigations are needed to determine the role of video-based assessment in improving trainee learning curves and patient outcomes.

Keywords

Pancreatobiliary (ERCP/PTCD) - ERC topics - Training - Quality and logistical aspects - Quality management

Publication History

Received: 20 March 2023

Accepted after revision: 22 August 2023

Accepted Manuscript online:
28 August 2023

Article published online:
28 February 2024

• References

• 1 Vandervoort J, Soetikno RM, Tham TC. et al. Risk factors for complications after performance of ERCP. Gastrointest Endosc 2002; 56: 652-656 DOI: 10.1067/mge.2002.129086. (PMID: 12397271)
• 2 Cotton PB, Garrow DA, Gallagher J. et al. Risk factors for complications after ERCP: a multivariate analysis of 11,497 procedures over 12 years. Gastrointest Endosc 2009; 70: 80-88 DOI: 10.1016/j.gie.2008.10.039. (PMID: 19286178)
• 3 Shahidi N, Ou G, Telford J. et al. Establishing the learning curve for achieving competency in performing colonoscopy: a systematic review. Gastrointest Endosc 2014; 80: 410-416 DOI: 10.1016/j.gie.2014.04.056. (PMID: 24973174)
• 4 Faulx AL, Lightdale JR. ASGE Standards of Practice Committee. et al. Guidelines for privileging, credentialing, and proctoring to perform GI endoscopy. Gastrointest Endosc 2017; 85: 273-281
• 5 Wani S, Keswani R, Hall M. et al. A prospective multicenter study evaluating learning curves and competence in endoscopic ultrasound and endoscopic retrograde cholangiopancreatography among advanced endoscopy trainees: The Rapid Assessment of Trainee Endoscopy Skills Study. Clin Gastroenterol Hepatol 2017; 15: 1758-1767
• 6 Wani S, Hall M, Wang AY. et al. Variation in learning curves and competence for ERCP among advanced endoscopy trainees by using cumulative sum analysis. Gastrointest Endosc 2016; 83: 711-719
• 7 Wani S, Keswani RN, Petersen B. et al. Training in EUS and ERCP: standardizing methods to assess competence. Gastrointest Endosc 2018; 87: 1371-1382 DOI: 10.1016/j.gie.2018.02.009. (PMID: 29709305)
• 8 Wani S, Cote GA, Keswani R. et al. Learning curves for EUS by using cumulative sum analysis: implications for American Society for Gastrointestinal Endoscopy recommendations for training. Gastrointest Endosc 2013; 77: 558-565 DOI: 10.1016/j.gie.2012.10.012. (PMID: 23260317)
• 9 Wani S, Hall M, Keswani RN. et al. Variation in Aptitude of Trainees in Endoscopic Ultrasonography, Based on Cumulative Sum Analysis. Clin Gastroenterol Hepatol 2015; 13: 1318-1325 e1312 DOI: 10.1016/j.cgh.2014.11.008. (PMID: 25460557)
• 10 McQueen S, McKinnon V, VanderBeek L. et al. Video-Based Assessment in Surgical Education: A Scoping Review. J Surg Educ 2019; 76: 1645-1654 DOI: 10.1016/j.jsurg.2019.05.013. (PMID: 31175065)
• 11 Augestad KM, Butt K, Ignjatovic D. et al. Video-based coaching in surgical education: a systematic review and meta-analysis. Surg Endosc 2020; 34: 521-535 DOI: 10.1007/s00464-019-07265-0. (PMID: 31748927)
• 12 Netter A, Schmitt A, Agostini A. et al. Video-based self-assessment enhances laparoscopic skills on a virtual reality simulator: a randomized controlled trial. Surg Endosc 2021; 35: 6679-6686 DOI: 10.1007/s00464-020-08170-7. (PMID: 33241429)
• 13 Mota P, Carvalho N, Carvalho-Dias E. et al. Video-based surgical learning: improving trainee education and preparation for surgery. J Surg Educ 2018; 75: 828-835 DOI: 10.1016/j.jsurg.2017.09.027. (PMID: 29031522)
• 14 Rindos NB, Wroble-Biglan M, Ecker A. et al. Impact of video coaching on gynecologic resident laparoscopic suturing: a randomized controlled trial. J Minim Invasive Gynecol 2017; 24: 426-431 DOI: 10.1016/j.jmig.2016.12.020. (PMID: 28063907)
• 15 Elmunzer BJ, Walsh CM, Guiton G. et al. Development and initial validation of an instrument for video-based assessment of technical skill in ERCP. Gastrointest Endosc 2021; 93: 914-923 DOI: 10.1016/j.gie.2020.07.055. (PMID: 32739484)
• 16 Scaffidi MA, Grover SC, Carnahan H. et al. A prospective comparison of live and video-based assessments of colonoscopy performance. Gastrointest Endosc 2018; 87: 766-775 DOI: 10.1016/j.gie.2017.08.020. (PMID: 28859953)
• 17 Jeyalingam T, Walsh CM. Video-based assessments: a promising step in improving polypectomy competency. Gastrointest Endosc 2019; 89: 1231-1233 DOI: 10.1016/j.gie.2019.04.203. (PMID: 31104751)
• 18 Messick S. Validity of psychological assessment: validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. Am Psychologist 1995; 50: 741-749
• 19 Messick S. Educational measurement 1989 3rd ed, p. 13–103.
• 20 Kane M. An argument-based approach to validity. Psychol Bull 1992; 112: 527-535
• 21 Downing S. Reliability: on the reproducability of assessment data. Med Educ 2004; DOI: 10.1111/j.1365-2929.2004.01932.x. (PMID: 15327684)
• 22 Elmunzer BJ, Walsh CM, Guiton G. et al. Development and initial validation of an instrument for video-based assessment of technical skill in ERCP. Gastrointestinal endoscopy 2021; 93: 914-923 DOI: 10.1016/j.gie.2020.07.055. (PMID: 32739484)
• 23 Watkins M. Foundations of clinical research: applications to practice. Pearson Education Inc.. 2009
• 24 Tavares W, Eva KW. Exploring the impact of mental workload on rater-based assessments. Adv Health Sci Educ Theory Pract 2013; 18: 291-303 DOI: 10.1007/s10459-012-9370-3. (PMID: 22484964)
• 25 Tavares W, Ginsburg S, Eva KW. Selecting and simplifying: rater performance and behavior when considering multiple competencies. Teach Learn Med 2016; 28: 41-51 DOI: 10.1080/10401334.2015.1107489. (PMID: 26787084)
• 26 Elmunzer BJ, Serrano J, Chak A. et al. Rectal indomethacin alone versus indomethacin and prophylactic pancreatic stent placement for preventing pancreatitis after ERCP: study protocol for a randomized controlled trial. Trials 2016; 17: 120 DOI: 10.1186/s13063-020-04458-0. (PMID: 32493506)
• 27 Scott DJ, Rege RV, Bergen PC. et al. Measuring operative performance after laparoscopic skills training: edited videotape versus direct observation. J Laparoendosc Adv Surg Tech A 2000; 10: 183-190 DOI: 10.1089/109264200421559. (PMID: 10997840)
• 28 Sawyer JM, Anton NE, Korndorffer JR. et al. Time crunch: increasing the efficiency of assessment of technical surgical skill via brief video clips. Surgery 2018; 163: 933-937 DOI: 10.1016/j.surg.2017.11.011. (PMID: 29373170)