Rapid development of accurate artificial intelligence scoring for colitis disease activity using applied data science techniques

Abstract

Background and study aims Scoring endoscopic disease activity in colitis represents a complex task for artificial intelligence (AI), but is seen as a worthwhile goal for clinical and research use cases. To date, development attempts have relied on large datasets, achieving reasonable results when comparing normal to active inflammation, but not when generating subscores for the Mayo Endoscopic Score (MES) or ulcerative colitis endoscopic index of severity (UCEIS).

Patients and methods Using a multi-task learning framework, with frame-by-frame analysis, we developed a machine-learning algorithm (MLA) for UCEIS trained on just 38,124 frames (73 patients with biopsy-proven ulcerative colitis). Scores generated by the MLA were compared to consensus scores from three independent human reviewers.

Results Accuracy and agreement (kappa) were calculated for the following differentiation tasks: (1) normal mucosa vs active inflammation (UCEIS 0 vs ≥ 1; accuracy 0.90, κ = 0.90); (2) mild inflammation vs moderate-severe (UCEIS 0–3 vs ≥ 4; accuracy 0.98, κ = 0.96); (3) generating total UCEIS score (κ = 0.92). Agreement for UCEIS subdomains was also high (κ = 0.80, 0.83 and 0.88 for vascular pattern, bleeding and erosions respectively).

Conclusions We have demonstrated that, using modified data science techniques and a relatively smaller datasets, it is possible to achieve high levels of accuracy and agreement with human reviewers (in some cases near-perfect), for AI in colitis scoring. Further work will focus on refining this technique, but we hope that it can be used in other tasks to facilitate faster development.

Publication History

Received: 23 August 2021

Accepted after revision: 14 December 2021

Article published online:
14 April 2022

• References

• 1 Lamb CA, Kennedy NA, Raine T. et al. British Society of Gastroenterology consensus guidelines on the management of inflammatory bowel disease in adults. Gut 2019; 68: s1-106
• 2 Rubin DT, Ananthakrishnan AN, Siegel CA. et al. ACG Clinical Guideline: Ulcerative Colitis in Adults. Am J Gastroenterol 2019; 114: 384-413
• 3 Turner D, Ricciuto A, Lewis A. et al. STRIDE-II: An update on the Selecting Therapeutic Targets in Inflammatory Bowel Disease (STRIDE) Initiative of the International Organization for the Study of IBD (IOIBD): Determining therapeutic goals for treat-to-target strategies in IBD. Gastroenterology 2021; 160: 1570-1583
• 4 Xie T, Zhang T, Ding C. et al. Ulcerative Colitis Endoscopic Index of Severity (UCEIS) versus Mayo Endoscopic Score (MES) in guiding the need for colectomy in patients with acute severe colitis. Gastroenterol Rep 2018; 6: 38-44
• 5 Colombel JF, Rutgeerts P, Reinisch W. et al. Early mucosal healing with infliximab is associated with improved long-term clinical outcomes in ulcerative colitis. Gastroenterology 2011; 141: 1194-1201
• 6 Knowles SR, Keefer L, Wilding H. et al. Quality of life in inflammatory bowel disease: a systematic review and meta-analyses – Part II. Inflamm Bowel Dis 2018; 24: 966-976
• 7 Knowles SR, Graff LA, Wilding H. et al. Quality of Life in Inflammatory Bowel Disease: A Systematic Review and Meta-analyses – Part i. Inflamm Bowel Dis 2018; 24: 742-751
• 8 US Food and Drug Administration. Ulcerative Colitis: Clinical Trial Endpoints Guidance for Industry. 2016 https://www.fda.gov/regulatory-information/search-fda-guidance-documents/ulcerative-colitis-clinical-trial-endpoints-guidance-industry
• 9 European Medicines Agency. Guideline on the Development of New Medicinal Products for the Treatment of Crohn’s Disease. Published Online First: 2019. http://www.ema.europa.eu/ema/index.jsp?curl=pages/regulation/general/general_content_000375.jsp&mid=WC0b01ac0580032ec6%5Cnpapers2://publication/uuid/A052211A-E2B5-4084-9C22-29E99945A3C7
• 10 Vashisht M, Samaan M, Mosli M. et al. Endoscopic scoring indices for evaluation of disease activity in ulcerative colitis. Cochrane Database Sytst Rev 2018; 1: CD011450
• 11 Ikeya K, Hanai H, Sugimoto K. et al. The ulcerative colitis endoscopic index of severity more accurately reflects clinical outcomes and long-term prognosis than the Mayo Endoscopic Score. J Crohn Colitis 2016; 10: 286-295
• 12 Gutierrez Becker B, Arcadu F, Thalhammer A. et al. Training and deploying a deep learning model for endoscopic severity grading in ulcerative colitis using multicenter clinical trial data. Ther Adv Gastrointest Endosc Artif 2021; 14: 1-15
• 13 Bhambhvani HP, Zamora A. Deep learning enabled classification of Mayo endoscopic subscore in patients with ulcerative colitis. Eur J Gastroenterol Hepatol 2020; DOI: 10.1097/meg.0000000000001952.
• 14 Stidham RW, Liu W, Bishu S. et al. Performance of a deep learning model vs human reviewers in grading endoscopic disease severity of patients with ulcerative colitis. JAMA Net Open 2019; 2: e193963
• 15 Takenaka K, Ohtsuka K, Fujii T. et al. Development and Validation of a deep neural network for accurate evaluation of endoscopic images from patients with ulcerative colitis. Gastroenterology 2020; 158: 2150-2157
• 16 Gottlieb K, Requa J, Karnes W. et al. Central reading of ulcerative colitis clinical trial videos using neural networks. Gastroenterology 2021; 160: 710-719.e2
• 17 Hansen US, Landau E, Patel M. et al. Novel artificial intelligence-driven software significantly shortens the time required for annotation in computer vision projects. Endosc Int Open 2021; 09: E621-E626
• 18 Girshick R. Fast R-CNN. Proc IEEE Int Conf Comput Vis 2015; 1440-1448 DOI: 10.1109/ICCV.2015.169.
• 19 Travis SPL, Schnell D, Krzeski P. et al. Reliability and initial validation of the ulcerative colitis endoscopic index of severity. Gastroenterology 2013; 145: 987-995