Development of a novel endoscopic hemostasis-assisted navigation AI system in the standardization of post-ESD coagulation

Abstract

Background and study aims While gastric endoscopic submucosal dissection (ESD) has become a treatment with fewer complications, delayed bleeding remains a challenge. Post-ESD coagulation (PEC) is performed to prevent delayed bleeding. Therefore, we developed an artificial intelligence (AI) to detect vessels that require PEC in real time.

Materials and methods Training data were extracted from 153 gastric ESD videos with sufficient images taken with a second-look endoscopy (SLE) and annotated as follows: (1) vessels that showed bleeding during SLE without PEC; (2) vessels that did not bleed during SLE with PEC; and (3) vessels that did not bleed even without PEC. The training model was created using Google Cloud Vertex AI and a program was created to display the vessels requiring PEC in real time using a bounding box. The evaluation of this AI was verified with 12 unlearned test videos, including four cases that required additional coagulation during SLE.

Results The results of the test video validation indicated that 109 vessels on the ulcer required cauterization. Of these, 80 vessels (73.4%) were correctly determined as not requiring additional treatment. However, 25 vessels (22.9%), which did not require PEC, were overestimated. In the four videos that required additional coagulation in SLE, AI was able to detect all bleeding vessels.

Conclusions The effectiveness and safety of this endoscopic treatment-assisted AI system that identifies visible vessels requiring PEC should be confirmed in future studies.

Introduction

Artificial intelligence (AI)-based computer-aided systems include computer-aided detection (CADe), which assists endoscopists in the intraoperative detection of abnormal lesions, and computer-aided diagnosis (CADx), which assists endoscopists in diagnosing lesion characteristics (distinguishing between benign and malignant lesions, determining lesion progression, etc.). Most of the research on AI in endoscopy has focused on lesion detection and characterization. In the field of colonoscopy, several AI systems for colonoscopy have been produced and are available for clinical use, and many papers have been published, including randomized controlled trials and meta-analyses [1] [2]. However, while these AI systems exist as aids to endoscopic detection and diagnosis, AI has not yet been used in the field of endoscopic treatment.

For early gastric cancer, endoscopic submucosal dissection (ESD) is less invasive and more curative than surgery. However, ESD involves a 5.8% risk of delayed bleeding [3]. Takizawa et al. reported that post-ESD coagulation (PEC), which uniformly cauterizes all nonbleeding visible vessels (NBVVs) in post-ESD ulcers, was successful in reducing delayed bleeding [4]. However, the operator determines the criteria for selecting the vessels to be cauterized with PEC, and these criteria are not constant. Therefore, bleeding after PEC has not been adequately verified to determine if cauterization was inadequate or whether the operator failed to identify the vessels to be cauterized. The aim of this study was to develop an endoscopic treatment-assisted AI system to detect vessels that require PEC and alert the endoscopist.

Materials and methods

In this study, a treatment support AI that detects vessels requiring PEC in real time was developed with Google Cloud Vertex AI [5], and the effectiveness of the AI was evaluated using untrained videos.

PEC procedure and post-procedure management

PEC was performed in all cases after ESD. All NBVVs on the artificial ulcer were coagulated using a set of hemostatic forceps (Coagrasper; Olympus) in Soft-Coagulation mode (80-W, effect 5). Second-look endoscopy (SLE) was performed within 24 hours after ESD. Post-PEC bleeding was defined as the presence of an exposed vessel or active bleeding during SLE and additional cautery with coagulation forceps was performed at each operator's discretion. Delayed bleeding was defined as clinical evidence of bleeding (hematemesis and/or melena) after SLE.

Training image preparation

This was a single-center, retrospective study aimed at developing a computer-assisted therapy (named CATx) system to detect vessels requiring PEC. Training images were obtained from videos of 153 early gastric cancer ESD cases from June 2014 to August 2022. SLE was performed within 24 hours in each case, and no evidence of delayed bleeding was detected. These video images were captured using a high-resolution endoscope (GIF-H260Z) and recorded in full high definition. The following three vessel types were registered to help the AI recognize vessels requiring PEC ([Fig. 1]): 1) Vessels requiring PEC were defined as vessels that required hemostatic treatment during SLE because PEC was not performed during the ESD. These vessels were registered as requiring treatment; 2) PEC-successful vessels were defined as vessels that did not require hemostatic treatment during SLE because PEC was performed. These vessels were also registered as requiring treatment; and 3) Vessels not requiring PEC were defined as vessels that did not undergo PEC and did not require hemostatic treatment even during SLE. These vessels were registered as controls.

Fig. 1 Three types of blood vessels were registered in the training data. a Vessels that did not undergo post-endoscopic submucosal dissection (ESD) coagulation (PEC) during ESD (red box) but b required hemostatic treatment during second-look endoscopy (SLE) (arrow). c Vessels that underwent PEC during ESD (yellow box), and d did not require hemostatic treatment during SLE because of the use of PEC. e Vessels that did not require the use of PEC (blue box), and f did not require hemostatic treatment during SLE.

These vessels were compared with the movies taken during ESD and the pictures taken by SLE and video scenes showing the above three vessels were extracted as still images. These images were trimmed to the endoscopic image part and training images were prepared.

AI model development using Google Cloud Vertex AI

Although deep learning has become a mainstream method in recent years, the creation of AI models with algorithms developed specifically for each study requires expert deep learning engineers. Google Cloud Vertex AI is a cloud-hosted model generator and machine learning tool that allows users to automatically create AI models using their own datasets, without the need for technical skills or coding. Using this system requires the following steps: 1) register the image dataset in cloud storage; 2) annotate each image on Vertex AI; and 3) start the training. The images are automatically assigned to training, testing, and validation sets as follows: 80% to training images, 10% to test images, and 10% to validation images (Supplementary Material). The training results provide precision-recall curves when intersections over unions are specified, and precision and recall curves across all confidence thresholds are generated. The training results are output in TensorFlow Lite format and converted to EdgeTPU format using the EdgeTPU Compiler to introduce edge devices ([Fig. 2]).

Fig. 2 Flowchart showing steps in AI development. 1) still images were created from the endoscopic submucosal dissection (ESD) video and an image data set was prepared; 2) the image dataset was registered in Google cloud storage; 3) annotation and machine learning was performed on Vertex AI; and 4) the images are automatically assigned to training, testing, and validation sets as follows: 80% to training images, 10% to test images, and 10% to validation images. Training results were output from Vertex AI in TensorFlow Lite (TFLite) format.

Materials and system configuration

We employed Raspberry Pi 4B (RPi4), which enables low-cost development of real-time object detection applications via Python programming. To improve the speed and accuracy of the object detection model in RPi4, the architecture was simplified to match the hardware limitations of RPi4 and the TFlite model used Coral Edge TPU (Coral USB Accelerator, Google LLC, USA) to run the system [6]. The AI program running on the RPi4 was programmed in Python by two developers (H.F. and S.K.) ([Fig. 3]). AI recognition of a vessel on the video that required PEC was indicated by a yellow bounding box and a vessel that did not require PEC was indicated by a blue bounding box ([Fig. 4]).

Fig. 3 Deployment system hardware and operation image. a Test videos were input from a video recorder (also possible from an endoscopic video processor). b For high-speed operation on the Raspberry Pi, only the bounding box was rendered. c The final image was chroma-key synthesized from the input video image and AI images for real-time display during endoscopic treatment.

Fig. 4 Schematic diagram of AI operation. Vessels requiring PEC are indicated by a yellow bounding box and those not requiring PEC are indicated by a blue one.

Ethics committee approval

The research protocol was approved by the Ethics Committee of Toyama University Hospital (approval number R2021032). All procedures were conducted according to the relevant guidelines and regulations and the principles of the Declaration of Helsinki. The study design was approved by the Ethics Committee with a declaration on the University of Toyama Hospital website of an opt-out policy allowing potential patients or their relatives to refuse to participate in the study.

Results

AI model construction and validation

In total, 810 images were annotated and labeled with 790 hemostasis-requiring vessels (106 posthemorrhagic vessels and 684 hemostasis-requiring vessels) and 141 hemostasis-free vessels. All images were randomly and automatically assigned as follows: 744 images for training, 94 images for validation, and 93 images for testing. When the confidence threshold was set at 0.60, the precision was 70.4% and the recall was 40.2% ([Fig. 5]) (Supplementary Material). The training model was deployed on the RPi4 device and the ESD video recorded in full high definition (1920 × 1080, 30 frames per second) was input to the validation data. The AI processing capacity using this system was 16 to 24 images per second and the yellow bounding boxes denoted the vessels requiring PEC in real time ([Video 1]).

Fig. 5 AutoML model evaluation. a The area under precision-recall curve. b The relationship between the precision-recall curve and the confidence threshold.

The AI detects vessels located at the post-ESD ulcer. The yellow bounding boxes denote the vessels requiring PEC in real-time. After coagulation of the target vessel, the yellow bounding box no longer appears in the detection field.Video 1

Video-based validation

To test the performance of the real-time hemostatic assistance, we prepared 12 untrained ESD videos of early gastric cancer. In these ESD videos, an alert was considered positive if it was issued against the same vessel ≥ 2 times on the video. When the AI was run from the end of dissection to the completion of hemostasis, 109 blood vessels were detected in real time from 12 videos. Of the 109 vessels for which the AI ordered PEC, 80 vessels (73.4%) in which PEC was performed did not require additional treatment and the instructions by AI were correct. However, 25 vessels (23.1%) did not undergo PEC during ESD and did not require additional treatment during SLE and were thus overestimated (Table 1). Furthermore, 21 vessels that did not require PEC were identified, avoiding unnecessary coagulation. The 12 test videos included four cases that required hemostasis at the second examination; the AI identified the four vessels that required treatment but did not receive PEC ([Fig. 6]).

Fig. 6 Four examples of AI-predicted post-PEC bleeding. a,c The AI identified untreated vessels for which no post-ESD coagulation was performed, and b,d these vessels were coagulated during second-look endoscopy (SLE). e The AI identified a vessel that was recognized as a slightly bluish node. f This portion was recognized as an exposed vessel. g No blood vessels were clearly visible at the margins of the ulcer noted by the AI; however, during SLE, h bleeding was observed from the same site.

Discussion

This is the first endoscopic treatment-assisted AI reported to recognize vessels requiring PEC. PEC is a procedure that reduces posterior hemorrhage by cauterizing all visible NBBVs. The criterion for requiring PEC was the need for an additional procedure during SLE. Thus, the AI identified vessels requiring PEC; bounding boxes were displayed around vessels where PEC had been performed and no additional treatment was required during SLE and vessels that were missed during ESD and required additional procedures during SLE. Half of all post-endoscopic bleeding occurs within 24 hours of ESD [7]. Thus, performing appropriate PEC is important. However, a possible cause of delayed bleeding after PEC is undetected vessels requiring PEC. Vessels requiring cauterization are missed owing to operator experience, skill level, and fatigue. In the video validation conducted in this study, AI detected vessels on which the operator did not perform PEC and hemostatic treatment was required during SLE. In addition, some vessels that require PEC cannot be detected by endoscopy alone. Uedo et al. used Doppler ultrasound (DOP-US) to assess blood flow in NBVVs and cauterized DOP-US-positive vessels until the signal disappeared, and no delayed hemorrhage occurred [8]. Adding vascular images assessed using DOP-US to the dataset is necessary to verify whether AI can identify blood vessels that cannot be judged by the human eye.

There are several limitations to this study. It was conducted at a single center as a pilot study to explore the potential of the new field of endoscopy-assisted AI. To control delayed bleeding, vessels with delayed bleeding need to be registered in the dataset. However, in our center, the number of delayed hemorrhage cases was small; thus, data for the study were insufficient. Second, the size of the training images was reduced due to the creation of a lightweight training model, resulting in low resolution of the vessels targeted for PEC. Thus, the accuracy of the model did not improve. To address this problem, transition learning with YOLOX-nano, Efficient Net-Lite, and MOBILENET_MULTI_AVG_I384, which have large image sizes, and data augmentation of the dataset were performed. Accuracy exceeding the Vertex AI could not be achieved (Supplementary Material and Table S1).

Conclusions

Integrating this tool into endoscopic systems to support PEC in real time is currently possible. However, several obstacles to integrating this system into real-world clinical practice remain. These issues should be resolved to verify the effectiveness of PEC support in clinical applications.

Conflict of Interest

The authors declare that they have no conflict of interest.

Acknowledgement

We thank the members of Medical Engineering Technicians, Kenji Ogawa, Masaya Hiraki, Sayaka Shimizu, and Miho Takagi. The authors also thank Enago (www.enago.com) for the English language review.

• References

• 1 Hassan C, Spadaccini M, Iannone A. et al. Performance of artificial intelligence in colonoscopy for adenoma and polyp detection: a systematic review and meta-analysis. Gastrointest Endosc 2021; 93: 77-85 DOI: 10.1016/j.gie.2020.06.059. (PMID: 32598963)
• 2 Repici A, Badalamenti M, Maselli R. et al. Efficacy of real-time computer-aided detection of colorectal neoplasia in a randomized trial. Gastroenterology 2020; 159: 512-520
• 3 Oda I, Gotoda T, Hamanaka H. et al. Endoscopic submucosal dissection for early gastric cancer: technical feasibility, operation time and complications from a large consecutive series. Dig Endosc 2005; 17: 54-58
• 4 Takizawa K, Oda I, Gotoda T. et al. Routine coagulation of visible vessels may prevent delayed bleeding after endoscopic submucosal dissection – An analysis of risk factors. Endoscopy 2008; 40: 179-183 DOI: 10.1055/s-2007-995530. (PMID: 18322872)
• 5 Google Cloud, Vertex AI beginner’s guides. https://cloud.google.com/vertex-ai/docs/beginner/beginners-guide?hl=en
• 6 Peter S, Ayman A, Thangarajah A. et al. Intelligent real-time face-mask detection system with hardware acceleration for COVID-19 mitigation. Healthcare 2022; 10: 873
• 7 Fumiaki T, Naoto Y, Takashi A. et al. A second-look endoscopy may not reduce the bleeding after endoscopic submucosal dissection for gastric epithelial neoplasm. BMC Gastroenterology 2014; 14: 152
• 8 Uedo N, Takeuchi Y, Ishihara R. et al. Endoscopic Doppler US for the prevention of ulcer bleeding after endoscopic submucosal dissection for early gastric cancer: a preliminary study (with video). Gastrointest Endosc 2010; 72: 444-448

Publication History

Received: 28 June 2023

Accepted after revision: 04 January 2024

Accepted Manuscript online:
08 January 2024

Article published online:
15 April 2024