Automatic and unbiased assessment of competence in colonoscopy: exploring validity of the Colonoscopy Progression Score (CoPS)

Background and aims: Colonoscopy is a difficult procedure to master. Increasing demands for colonoscopy, due to screening and surveillance programs, have highlighted the need for competent performers. Valid methods for assessing technical skills are pivotal for training and assessment. This study is the first clinical descriptive report of a novel colonoscopy assessment tool based on Magnetic Endoscopic Imaging (MEI) data and the aim was to gather validity evidence based on the data collected using the “Colonoscopy Progression Score” (CoPS).

Methods: We recorded 137 colonoscopy procedures performed by 31 endoscopists at three university hospitals. The participants performed more than two procedures each (range 2 – 12) and had an experience of 0 – 10 000 colonoscopies. The CoPS was calculated for each recording and validity was explored using a widely accepted contemporary framework. The following sources of validity evidence were explored: response process (data collection), internal structure (reliability), relationship to other variables (i. e. operator experience), and consequences of testing (pass/fail).

Results: Identical set-ups at all three locations ensured uniform data collection. The Generalizability coefficient (G-coefficient) was 0.80, and a Decision-study (D-study) revealed that four recordings were sufficient to ensure a G-coefficient above 0.80. We showed a positive correlation between CoPS and experience with Pearson’s r of 0.61 (P < 0.001). A pass/fail standard of 107 points was established using the contrasting group method to explore the consequences of testing.

Conclusion: This study provides evidence supporting the validity of the CoPS for use in assessing technical colonoscopy performance in the clinical setting.

Study registration: NCT01997177.

• References

• 1 Saunders BP, Phillips RK, Williams CB. Intraoperative measurement of colonic anatomy and attachments with relevance to colonoscopy. Br J Surg 1995; 82: 1491-1493
• 2 Bini EJ, Firoozi B, Choung RJ et al. Systematic evaluation of complications related to endoscopy in a training setting: A prospective 30-day outcomes study. Gastrointest Endosc 2003; 57: 8-16
• 3 Spier BJ, Durkin ET, Walker AJ et al. Surgical resident’s training in colonoscopy: numbers, competency, and perceptions. Surg Endosc 2010; 24: 2556-2561
• 4 Lee S-H, Chung I-K, Kim S-J et al. An adequate level of training for technical competence in screening and diagnostic colonoscopy: a prospective multicenter evaluation of the learning curve. Gastrointest Endosc 2008; 67: 683-689
• 5 Sedlack RE. Training to competency in colonoscopy: assessing and defining competency standards. Gastrointest Endosc 2011; 74: 355-366
• 6 Koch AD, Haringsma J, Schoon EJ et al. Competence measurement during colonoscopy training: the use of self-assessment of performance measures. Am J Gastroenterol 2012; 107: 971-975
• 7 Do A, Weinberg J, Kakkar A et al. Reliability of adenoma detection rate is based on procedural volume. Gastrointest Endosc 2013; 77: 376-380
• 8 Barton JR, Corbett S, Van der Vleuten CP. The validity and reliability of a Direct Observation of Procedural Skills assessment tool: assessing colonoscopic skills of senior endoscopists. Gastrointest Endosc 2012; 75: 591-597
• 9 Walsh CM, Ling SC, Khanna N et al. Gastrointestinal Endoscopy Competency Assessment Tool: development of a procedure-specific assessment tool for colonoscopy. Gastrointest Endosc 2014; 79: 798-807
• 10 Sedlack RE, Baron TH, Downing SM et al. Validation of a colonoscopy simulation model for skills assessment. Am J Gastroenterol 2007; 102: 64-74
• 11 Walsh CM, Sherlock ME, Ling SC, Carnahan H. Virtual reality simulation training for health professions trainees in gastrointestinal endoscopy. In: Walsh CM, ed. The Cochrane Collaboration. Cochrane Database of Systematic Reviews [Internet]. Chichester, UK: John Wiley & Sons, Ltd; 2012. [cited 2 June 2013]
• 12 Marshall JB. Technical proficiency of trainees performing colonoscopy: a learning curve. Gastrointest Endosc 1995; 42: 287-291
• 13 Haycock A, Bassett P, Bladen J et al. Validation of the second-generation Olympus colonoscopy simulator for skills assessment. Endoscopy 2009; 41: 952-958
• 14 Lüning TH, Keemers-Gels ME, Barendregt WB et al. Colonoscopic perforations: a review of 30,366 patients. Surg Endosc 2007; 21: 994-997
• 15 Cotton P, Williams C. Practical gastrointestinal endoscopy: the fundamentals. Oxford: John Wiley & Sons; 2008: 88-93
• 16 Shah SG, Thomas-Gibson S, Lockett M et al. Effect of real-time magnetic endoscope imaging on the teaching and acquisition of colonoscopy skills: results from a single trainee. Endoscopy 2003; 35: 421-425
• 17 Nerup N, Preisler L, Svendsen MBS et al. Assessment of colonoscopy by use of magnetic endoscopic imaging: design and validation of an automated tool. Gastrointest Endosc 2015; 81: 548-554
• 18 Messick SA. Validity. In: Linn RL, ed. Educational measurement. 3rd edn. New York: Macmillan; 1989: 13-103
• 19 Downing SM. Validity: on the meaningful interpretation of assessment data. Med Educ 2003; 37: 830-837
• 20 Brennan RL. Generalizability theory. New York: Springer; 2001: 190-207
• 21 Downing SM, Yudkowsky R. Assessment in health professions education. New York: Routledge; 2009: 119-148
• 22 Spier BJ, Benson M, Pfau PR et al. Colonoscopy training in gastroenterology fellowships: determining competence. Gastrointest Endosc 2010; 71: 319-324
• 23 Farthing MJ, Walt RP, Allan RN et al. A national training programme for gastroenterology and hepatology. Gut 1996; 38: 459-470
• 24 Downing SM. Reliability: on the reproducibility of assessment data. Med Educ 2004; 38: 1006-1012
• 25 Sedlack RE. The Mayo Colonoscopy Skills Assessment Tool: validation of a unique instrument to assess colonoscopy skills in trainees. Gastrointest Endosc 2010; 72: 1125-1133, 1133.e1 – 3
• 26 Dancey CP. Statistics without maths for psychology: using SPSS for Windows. 4th ed. Harlow, UK; New York: Pearson/Prentice Hall; 2007: 168-203
• 27 Chung YW, Han DS, Yoo K-S et al. Patient factors predictive of pain and difficulty during sedation-free colonoscopy: A prospective study in Korea. Dig Liver Dis 2007; 39: 872-876
• 28 Konge L, Larsen KR, Clementsen P et al. Reliable and valid assessment of clinical bronchoscopy performance. Respiration 2012; 83: 53-60
• 29 Manzone J, Tremblay L, You-Ten KE et al. Task- versus ego-oriented feedback delivered as numbers or comments during intubation training. Med Educ 2014; 48: 430-440
• 30 Korman LY. Skill and colonoscopy: even the pros take lessons. Gastrointest Endosc 2015; 81: 555-556