Subscribe to RSS
DOI: 10.1055/a-0761-9742
Nomenclature and semantic description of vascular lesions in small bowel capsule endoscopy: an international Delphi consensus statement[*]
Corresponding author
Publication History
submitted 07 May 2018
accepted after revision 09 August 2018
Publication Date:
28 February 2019 (online)
Abstract
Background and study aims Nomenclature and descriptions of small bowel (SB) vascular lesions in capsule endoscopy (CE) are scarce in the medical literature. They are mostly based on the reader's opinion and thus differ between experts, with a potential negative impact on clinical care, teaching and research regarding SBCE. Our aim was to better define a nomenclature and to give a description of the most frequent vascular lesions in SBCE.
Methods A panel of 18 European expert SBCE readers was formed during the UEGW 2016 meeting. Three experts constructed an Internet-based four-round Delphi consensus, but did not participate in the voting process. They built questionnaires that included various still frames of vascular lesions obtained with a third-generation SBCE system. The 15 remaining participants were asked to rate different proposals and description of the most common SB vascular lesions. A 6-point rating scale (varying from ‘strongly disagree’ to ‘strongly agree’) was used successive rounds. The consensus was reached when at least 80 % voting members scored the statement within the ‘agree’ or ‘strongly agree’.
Results Consensual terms and descriptions were reached for angiectasia/angiodysplasia, erythematous patch, red spot/dot, and phlebectasia. A consensual description was reached for more subtle vascular lesions tentatively named “diminutive angiectasia” but no consensus was reached for this term.
Conclusion An international group has reached a consensus on the nomenclature and descriptions of the most frequent and relevant SB vascular lesions in CE. These terms and descriptions are useful in daily practice, for teaching and for medical research purposes.
#
Introduction
Capsule endoscopy (CE) is a minimally invasive technique that allows complete visualization of the small bowel (SB) [1]. During its development in the 1990s, CE was immediately perceived as a disruptive technology. Since its clinical inception in 2000, CE has rapidly become the first-line diagnostic tool to explore the SB, with a diagnostic yield of approximately 60 % in the setting of obscure gastrointestinal bleeding (OGIB), and 50 % in suspected Crohn’s disease [2].
Because CE offers enhanced imaging of the SB mucosa, an important effort has been made during the first few years of its clinical use to design a structured lexicon for use in training, clinical practice, and for research purposes [3] [4] [5]. The CE Structured Terminology (CEST) emerged through this work, but it did not provide definitions nor full descriptions of the most common SB lesions. As a matter of fact, no clear definitions/descriptions of vascular lesions have been widely accepted yet. A typical example is “angiodysplasia” (AGD) that remains the most common vascular and obvious lesion found in SBCE in the setting of OGIB [6]; however, there are controversies regarding the name, description, and interobserver agreement of this diagnosis. Some experts argue that the suffix “dysplasia” is inappropriate from a pathological point of view, and that the term “ectasia” should be preferred [7]. Nevertheless, in clinical practice, more than half of reports use other terms, such as, “angioma,” “ectasia,” “arteriovenous malformation,” “vascular ectasia,” or “vascular malformation” [4].
In the medical literature, semantic descriptions of AGD are scarce [6] [8], and are even more rare for more subtle vascular lesions. In addition, in most studies aiming to evaluate interobserver in describing CE findings, the “gold standard” for diagnosis is based on the experience of expert investigators, without any description or definition of vascular lesions [9] [10]. Overall, names, descriptions, and diagnosis of vascular SB lesions remain subjective. Still, when a putative diagnosis of SB vascular lesion is made with CE, ablation therapy is first-line treatment in most cases, pharmacological treatments are discussed, but endoscopic sampling or surgery followed by pathological examination are rarely performed to confirm the diagnosis [6]. Overall, interobserver agreement for all lesions seen in CE is moderate (kappa coefficient = 0.6) in the medical literature [10], and seems even lower for more subtle lesions.
Universal nomenclature and descriptions of SB vascular are needed for CE reporting [11], medical education, and research. The aim of this study was to achieve consensus on the nomenclature and descriptions of SB vascular lesions seen in CE.
#
Materials and methods
Design
The consensus process used a Delphi method, an established method for obtaining answers to questions that cause uncertainty even between experts [12] [13]. The study design was based on alternative feedback rounds of a core group (CG) and an expert group (EG) until consensus was achieved ([Fig. 1]). Several Internet-based illustrated questionnaires were thus from the CG to the EG for rating and comments. All questionnaires were structured in a similar way, with proposals for a nomenclature and a semantic description, and calling for free comments. Nomenclature and descriptions were revised by the CG following each feedback round based on rating and of the EG.
#
Working groups and feedback rounds
Working groups were formed aside the 24th United European Gastroenterology Week (October 15 – 19, 2016, Vienna, Austria). The three investigators of the CG worked in Sorbonne University & APHP, Saint Antoine Hospital, Paris, France. The 15 international investigators of the EG were based in England (n = 1), France (n = 3), Israel (n = 2), Italy (n = 3), Scotland (n = 3), Spain (n = 1) and Sweden (n = 2). Mean age of the EG was 47 years. Mean CE reading experience of the EG was 13 years, with a mean number of 100 CEs read annually.
#
Proposals and revisions of names and semantic descriptions
The multicenter French database CAD-CAP (Computer-assisted Diagnosis for Capsule endoscopy) was used. CAD-CAP is composed of 20,000 normal CE still frames and 6013 still frames of abnormal CE findings from third-generation SBCE system videos (Pillcam SB3 system, Medtronic), including 2946 vascular lesions [14]. The CG selected an initial set of 100 still frames of most common vascular lesion images seen in SBCE. Based on this image dataset and on their knowledge, the members of the CG categorized 5 different types of SB vascular lesions. For each type of SB vascular lesion, the CG selected 4 typical frames (with adequate mucosal visualization and brightness) and proposed a putative name and a semantic description, to be sent to the EG through an Internet-based questionnaire. At each round, and for each type of lesion, members of the EG were asked to rate proposals of the CG, on a numerical scale from 1 (strongly disagree) to 6 (strongly agree) ([Table 1]), and to provide comments and suggestions for substantive revisions. After each round, all suggestions from EG respondents were used by the CG to improve names/descriptions.
1 |
Strongly disagree |
2 |
Disagree |
3 |
Moderately disagree |
4 |
Moderately agree |
5 |
Agree |
6 |
Strongly agree |
#
Analysis
Consensus was reached when at least 80 % of voting members rated with a score of 5 (agree) or 6 (strongly agree) within four rounds. Rounds were stopped for any lesion for which consensual name and description were found. For any other lesion, votes and comments were analyzed and discussed during several face-to-face meetings by the three participants of the CG in between the three first rounds. When consensual name and description were not achieved after the third round, a teleconference was held between the CG and the EG. Then, the CG made final revisions and provided a fourth questionnaire to the EG, for a final vote. The process was stopped thereafter, whether a consensus was reach or not.
#
#
Results
The CG selected five types of common SB vascular lesions from the CAD-CAP database. For each type of vascular lesions, a set of 4 still frames ([Fig. 2], [Fig. 3], [Fig. 4], [Fig. 5], [Fig. 6]) was sent, using Internet-based questionnaires, to the EG with tentative names and descriptions. Further feedback rounds between the CG and the EG were scheduled until consensus was achieved or until round 4 was reached. Consensual names and definitions of “angiectasia/angiodysplasia” ([Fig. 2]), “erythematous patch” ([Fig. 3]), “red spot/dot” ([Fig. 4]), and “phlebectasia” ([Fig. 5]) were achieved within 3 rounds ([Table 2]). However, a conference call and a final fourth round did not result in consensus on the name of more subtle vascular mucosal abnormalities, tentatively named “diminutive angiectasia” ([Fig. 6]), although a consensus was reached on their semantic description. Numbers of voting experts for names and descriptions are described more precisely for the last round in [Table 3] and [Table 4]. Details on the votes for nomenclatures are given in [Supplementary Table 1].
Nomenclature |
Semantic Description |
Nomenclature/Description % of agreement or strong agreement |
Angiectasia/angiodysplasia |
A clearly demarcated, bright-red, flat lesion, consisting of tortuous and clustered capillary dilatations, within the mucosal layer (surrounded by intestinal villi). Can be small (few mm) to large (few cm). |
100 %/93 % |
Erythematous patch |
A small (few mm) and flat reddish area, without any vessel appearance, within the mucosal layer (surrounded by intestinal villi). |
87 %/80 % |
Red spot/dot |
A miniscule (less than 1 mm), punctuate, and flat lesion with a bright-red area, without linear or vessel appearance, within the mucosal layer (surrounded by intestinal villi). |
93 %/80 % |
Phlebectasia |
A small (few mm), flat to slightly elevated, bluish venous dilatation running below the mucosa (covered by intestinal villi). |
93 %/87 % |
Diminutive angiectasia[*] |
A clearly demarcated, linear, bright-red lesion, consisting of tiny non-clustered capillary dilatations, within the mucosal layer (surrounded by intestinal villi). |
73 %/87 % |
* Consensus was not reached
Nomenclature |
Numerical scale/Number of voting expert |
% of agreement or strong agreement |
Number of rounds passed |
|||||
1 |
2 |
3 |
4 |
5 |
6 |
|||
Angiectasia/angiodysplasia |
0 |
0 |
0 |
0 |
1 |
14 |
100 % |
1 |
Erythematous Patch |
1 |
0 |
1 |
0 |
3 |
10 |
87 % |
2 |
Red spot/dot |
1 |
0 |
0 |
1 |
4 |
9 |
93 % |
1 |
Phlebectasia |
0 |
0 |
0 |
1 |
3 |
11 |
93 % |
1 |
“Diminutive angiectasia” |
0 |
3 |
0 |
1 |
8 |
3 |
73 %[*] |
4 |
* Consensus was not reached
Description |
Numerical scale/Number of voting expert |
% of agreement or strong agreement |
Number of rounds passed |
|||||
1 |
2 |
3 |
4 |
5 |
6 |
|||
Angiectasia/angiodysplasia |
0 |
0 |
0 |
1 |
5 |
9 |
93 % |
3 |
Erythematous Patch |
2 |
0 |
0 |
1 |
9 |
3 |
80 % |
2 |
Red spot/dot |
1 |
1 |
1 |
1 |
6 |
6 |
80 % |
1 |
Phlebectasia |
0 |
0 |
1 |
1 |
4 |
9 |
87 % |
1 |
“Diminutive angiectasia”[*] |
0 |
1 |
0 |
1 |
8 |
5 |
87 % |
4 |
* Consensus was not reached
Vascular lesions images |
Round 1 |
Round 2 |
Round 3 |
Round 4 |
[Figure 2] |
Angiodysplasia |
Angiodysplasia |
/ |
/ |
[Figure 3] |
Red spot |
Erythematous patch |
/ |
/ |
[Figure 4] |
Red dot |
Red dot |
/ |
/ |
[Figure 5] |
Phlebectasia |
/ |
/ |
/ |
[Figure 6] |
Telangiectasia |
Angiectasia |
Small angiodysplasia |
Diminutive angiectasia |
* AVM = Arterial vascular malformation
Angiectasia/angiodysplasia
Regarding [Fig. 2], the terminology voted for after one round with a 100.0 % rate of agreement was an “angiodysplasia.” A 93.3 % rate of agreement was found after Round 3 on the following description: “a clearly demarcated, bright-red, flat lesion, consisting of tortuous and clustered capillary dilatations within the mucosal layer (surrounded by intestinal villi). Can be small (few mm) to large (few cm).” However, there were controversies regarding the name “angiodysplasia” versus “angiectasia.” During post-Round 1 analysis, five experts mentioned that they would have preferred the term “angiectasia.” Thus, the core group decided to put the question to the vote during the second round. After Round 2, no agreement was found as 60.0 % of the experts preferred the term “angiectasia,” whereas 40.0 % preferred the term “angiodysplasia.”
#
Erythematous patch
Regarding [Fig. 3], the terminology voted for after Round 2 with an 86.6 % rate of agreement was an “erythematous patch.” An 80.0 % rate of agreement was found on the following description: “small (few mm) and flat reddish area, without any vessel appearance within the mucosal layer (surrounded by intestinal villi).”
#
Red spot/dot
Regarding [Fig. 4], the terminology voted for after Round 1 with a 93.3 % rate of agreement was a “Red spot/dot.” An 80.0 % rate of agreement was found on the following description: “a miniscule (less than 1 mm), punctuate, and flat lesion with a bright-red area, without linear or vessel appearance within the mucosal layer (surrounded by intestinal villi).” However, there were controversies regarding the term “red dot” versus “red spot” and the CG decided to put this question to a vote. After Round 2, no agreement was found, as 53.3 % preferred the name “red spot,” whereas 46.7 % preferred the term “red dot.”
#
Phlebectasia
Regarding [Fig. 5], the terminology voted for after Round 1 with a 93.3 % rate of agreement was a “phlebectasia.” An 86.6 % rate of agreement was found in Round 1 for the following description: “a small (few mm), flat to slightly-elevated, bluish venous dilatation running below the mucosa (covered by intestinal villi).”
#
‘Diminutive angiectasia’
Regarding [Fig. 6], the terminology voted for after Round 4 with only 73.3 % rate of agreement was a “diminutive angiectasia.” This lesion was the most difficult to name and to describe. Several proposals for terminology were discussed during the three first rounds, among them “dilated capillaries,” “small angiodysplasia,” “telangiectasia,” “engorged capillaries,” “small arterial vascular malformation (AVM),” “atypical AVM,” and “atypical AGD.” Experts failed to achieve a consensus after Round 3. According to protocol, a conference call was set up, for names and descriptions to be debated before a final vote. At Round 4, there was an 86.6 % rate of agreement on the following description: “a clearly demarcated, linear, bright-red lesion, consisting of tiny non-clustered capillary dilatations, within the mucosal layer (surrounded by intestinal villi).”
#
#
Discussion
This study established nomenclature and descriptions for some of the most common vascular lesions seen in SB-CE. Through a Delphi consensus conducted with a significant number of CE expert readers, four names and their semantic descriptions were validated with more than an 80 % rate of agreement: “angiectasia/angiodysplasia” (93.3 %), “erythematous patch” (80.0 %), “red spot/dot” (80.0 %), and “phlebectasia” (86.6 %) ([Table 2]).
Since the early 2000s, CE offers a new insight into SB pathology. Although an effort has been made initially to create a lexicon to describe SBCE findings [4], there has been no dictionary (with an entry name and a corresponding description) of SB vascular lesions since. This makes it difficult to attempt to standardize SBCE reading, reports, and research. For example, to date, most studies aiming to evaluate interobserver agreement in the description of SBCE findings have used the expert investigators’ opinion as gold standard for diagnosis [10], rather than any validated name or description of SB vascular lesions.
Consensus methods are being used increasingly to solve problems in medicine. Their main purpose is to define levels of agreement on controversial subjects. As a mean of achieving consensus on an issue, the Delphi methodology has become widely used in healthcare research [15] [16]. The literature on this technique is expanding, mainly addressing what it is and how it should be used [13] [17]. Given the need for a SB vascular lesions dictionary, we considered that Delphi methodology a good approach for large-scale research. However, Delphi methodology has some limitations. Therefore, we paid special attention to clearly separate the CG from the EG, with different settings and different tasks. The CG was a small group of three physicians from the same center who had experience in SB-CE reading, teaching, and research. Members of the CG were able to attend consensus meetings and aimed to make proposals and revisions based on the EG votes and comments but they did not vote. The EG comprised 15 European opinion leaders in SB-CE. They were chosen based on their long experience (mean 13 years) and on their high volume of SB-CE reading (mean 100 cases a year). This study has other strengths. All Internet-based answers were anonymous, to allow the experts to express any contradictory view or even to change their mind during the voting rounds. The participation rate for all questionnaires reached 100 % in all rounds.
Some limitations should be mentioned as well. First, the CG of three physicians from the same institution made the initial image dataset and proposals for names and descriptions, possibly misleading the EG from the very start. Other names, terms, and definitions could have emerged from a larger or more heterogeneous group. Second, questionnaires and votes were Internet-based, except one conference call. It is likely that a conference call or physical meeting would have created more debate, controversy, and enrichment of vocabulary for names and descriptions than comments on a website without any live interaction. However, free comments were still possible through the Internet-based questionnaire, and at the time of the conference call after Round 3. Third, some rare, protruding or syndrome-related vascular lesions were not addressed in this study (hemangioma, varices, Kaposi sarcoma, blue rubber bleb syndrome, for example). We believe that there was an urge to build a consensus on the most frequent vascular lesions seen in SB-CE. However, many other less frequent vascular lesions will also need to be better defined. Fourth, some differences in descriptions of vascular lesions are based on the lesion size. Although a team has proposed measuring the size of SB polyps using reference granules in patients with Peutz-Jeghers syndrome [18], there is currently no validated system for size measurement in SBCE. Fifth, this consensus aimed to brush up the nomenclature and semantic descriptions of SB vascular lesions, not to provide their clinical relevance in terms of bleeding. Our group is now strongly considering doing another Delphi consensus on the significance of the SB vascular lesions named and described herein.
Some issues have not been fully solved by this study. Although their description was consensual, there were some controversies regarding the choice for a name between “angiodysplasia” and “angiectasia.” The core group decided to put that question to the vote during Round 2 but no agreement was found as 60.0 % of the experts preferred the term “angiectasia,” whereas 40.0 % preferred the term “angiodysplasia.” Thus, in the experts’ view, there would be a slight preference for the term “angiectasia” for daily practice. For some experts, the term “angiodysplasia” is widely used in clinical practice and in research, and refers adequately to the above-described vascular lesion. However, some other experts argue that the suffix “dysplasia” is inappropriate because the lesion is not an epithelial anomaly of growth and differentiation, which for pathologists usually indicates a precancerous lesion. However, in Ancient Greek, “dys” means difficult and “plasso” means formed, and “angiodysplasia” would therefore etymologically mean “vascular malformation,” with no reference to precancerous change in cells and tissues [19]. Indeed, the term “dysplasia” is used in other benign conditions, such as hip, kidney or bone dysplasia. On the other hand, “ektasis” means “dilation” [19], and seems very appropriate to describe mucosal capillary dilations seen by pathologists in the rare cases in which such lesions are sampled. Overall, some slight confusion may still ensue with the different terms “angiectasia” (also sometimes written “angioectasia”) and “angiodysplasia,” although they relate to the very same type of lesion and may be considered as synonyms. Indeed, in the study validating the Capsule Endoscopy Standardized Terminology, the term “angiectasia” was used as an “obvious synonym” for the term “angiodysplasia” in 435 out of 486 occurrences for the experts [4]. For even more subtle lesions, such as so-called “diminutive angiectasia,” we were not able find a consensus name (only a 73.3 % agreement was reached), although a consensus description was made.
#
Conclusion
In conclusion, the effort carried out by an international group through this consensus statement on nomenclature and descriptions of the most frequent SB vascular lesions represents a step forward in standardization of SB-CE reading, teaching, and medical research on this topic.
#
#
Competing interests
Dr. Dray has acted as a consultant for Boston Scientific, Fujifilm, Medtronic, and Pentax.
Acknowledgements
The Computer-Assisted Diagnosis for Capsule Endoscopy database is endorsed by the Société Française d’Endoscopie Digestive. It has been made possible with the support of the Société Nationale Française de Gastroentérologie and with the support of MSD France.
* Meeting presentations: 25th United European Gastroenterology Week 2017
-
References
- 1 Iddan G, Meron G, Glukhovsky A. et al. Wireless capsule endoscopy. Nature 2000; 405: 417
- 2 Pennazio M, Spada C, Eliakim R. et al. Small-bowel capsule endoscopy and device-assisted enteroscopy for diagnosis and treatment of small-bowel disorders: European Society of Gastrointestinal Endoscopy (ESGE) Clinical Guideline. Endoscopy 2015; 47: 352-376
- 3 Korman LY, Delvaux M, Gay G. et al. Capsule endoscopy structured terminology (CEST): proposal of a standardized and structured terminology for reporting capsule endoscopy procedures. Endoscopy 2005; 37: 951-959
- 4 Delvaux M, Friedman S, Keuchel M. et al. Structured terminology for capsule endoscopy: results of retrospective testing and validation in 766 small-bowel investigations. Endoscopy 2005; 37: 945-950
- 5 Aabakken L, Rembacken B, LeMoine O. et al. Minimal standard terminology for gastrointestinal endoscopy – MST 3.0. Endoscopy 2009; 41: 727-728
- 6 Becq A, Rahmi G, Perrod G. et al. Hemorrhagic angiodysplasia of the digestive tract: pathogenesis, diagnosis, and management. Gastrointest Endosc 2017; 86: 792-806
- 7 Wassef M, Blei F, Adams D. et al. Vascular anomalies classification: recommendations from the International Society for the Study of Vascular Anomalies. Pediatrics 2015; 136: e203-214
- 8 Yano T, Yamamoto H, Sunada K. et al. Endoscopic classification of vascular lesions of the small intestine (with videos). Gastrointest Endosc 2008; 67: 169-172
- 9 Pezzoli A, Cannizzaro R, Pennazio M. et al. Interobserver agreement in describing video capsule endoscopy findings: a multicentre prospective study. Dig Liver Dis 2011; 43: 126-131
- 10 Jang BI, Lee SH, Moon J-S. et al. Inter-observer agreement on the interpretation of capsule endoscopy findings based on capsule endoscopy structured terminology: a multicenter study by the Korean Gut Image Study Group. Scand J Gastroenterol 2010; 45: 370-374
- 11 Micic D, Semrad CE. Small bowel endoscopy. Curr Treat Options Gastroenterol 2016; 14: 220-235
- 12 Dalkey N, Helmer O. An experimental application of the DELPHI Method to the use of experts. Management Science 1963; 9: 458-467
- 13 Hsu C-C, Sandford BA. The Delphi technique: making sense of consensus. Pract Assess Res Eval 2007; 12: 1-8
- 14 Leenhardt R, Li C, Le Mouel JP. et al. CAD-CAP: a 26000 images database serving the development of artificial intelligence for capsule endoscopy. United European Gastroenterology Week, Vienna, 22–24 oct 2018
- 15 Katragadda C, Finnane A, Soyer HP. Technique standards for skin lesion imaging: a delphi consensus statement. JAMA Dermatol 2017; 153: 207-213
- 16 Collins S, Yen P-Y, Phillips A. et al. Nursing informatics competency assessment for the nurse leader: The Delphi Study. J Nurs Adm 2017; 47: 212-218
- 17 Okoli C, Pawlowski SD. The Delphi method as a research tool: an example, design considerations and applications. Info Manag 2004; 42: 15-29
- 18 Rácz I, Jánoki M, Kovács V. Measurement of small-bowel polyp size in patients with Peutz-Jeghers syndrome by using reference granules during video capsule endoscopy. Endoscopy 2007; 1: E41
- 19 Brandt LJ. Terminology for vascular lesions of the GI tract. Gastrointest Endosc 2018; 87: 1595-1596 comment to Becq A, et al. Terminology for vascular lesions of the GI tract. Gastrointest Endosc 2018; 87: 1596
Corresponding author
-
References
- 1 Iddan G, Meron G, Glukhovsky A. et al. Wireless capsule endoscopy. Nature 2000; 405: 417
- 2 Pennazio M, Spada C, Eliakim R. et al. Small-bowel capsule endoscopy and device-assisted enteroscopy for diagnosis and treatment of small-bowel disorders: European Society of Gastrointestinal Endoscopy (ESGE) Clinical Guideline. Endoscopy 2015; 47: 352-376
- 3 Korman LY, Delvaux M, Gay G. et al. Capsule endoscopy structured terminology (CEST): proposal of a standardized and structured terminology for reporting capsule endoscopy procedures. Endoscopy 2005; 37: 951-959
- 4 Delvaux M, Friedman S, Keuchel M. et al. Structured terminology for capsule endoscopy: results of retrospective testing and validation in 766 small-bowel investigations. Endoscopy 2005; 37: 945-950
- 5 Aabakken L, Rembacken B, LeMoine O. et al. Minimal standard terminology for gastrointestinal endoscopy – MST 3.0. Endoscopy 2009; 41: 727-728
- 6 Becq A, Rahmi G, Perrod G. et al. Hemorrhagic angiodysplasia of the digestive tract: pathogenesis, diagnosis, and management. Gastrointest Endosc 2017; 86: 792-806
- 7 Wassef M, Blei F, Adams D. et al. Vascular anomalies classification: recommendations from the International Society for the Study of Vascular Anomalies. Pediatrics 2015; 136: e203-214
- 8 Yano T, Yamamoto H, Sunada K. et al. Endoscopic classification of vascular lesions of the small intestine (with videos). Gastrointest Endosc 2008; 67: 169-172
- 9 Pezzoli A, Cannizzaro R, Pennazio M. et al. Interobserver agreement in describing video capsule endoscopy findings: a multicentre prospective study. Dig Liver Dis 2011; 43: 126-131
- 10 Jang BI, Lee SH, Moon J-S. et al. Inter-observer agreement on the interpretation of capsule endoscopy findings based on capsule endoscopy structured terminology: a multicenter study by the Korean Gut Image Study Group. Scand J Gastroenterol 2010; 45: 370-374
- 11 Micic D, Semrad CE. Small bowel endoscopy. Curr Treat Options Gastroenterol 2016; 14: 220-235
- 12 Dalkey N, Helmer O. An experimental application of the DELPHI Method to the use of experts. Management Science 1963; 9: 458-467
- 13 Hsu C-C, Sandford BA. The Delphi technique: making sense of consensus. Pract Assess Res Eval 2007; 12: 1-8
- 14 Leenhardt R, Li C, Le Mouel JP. et al. CAD-CAP: a 26000 images database serving the development of artificial intelligence for capsule endoscopy. United European Gastroenterology Week, Vienna, 22–24 oct 2018
- 15 Katragadda C, Finnane A, Soyer HP. Technique standards for skin lesion imaging: a delphi consensus statement. JAMA Dermatol 2017; 153: 207-213
- 16 Collins S, Yen P-Y, Phillips A. et al. Nursing informatics competency assessment for the nurse leader: The Delphi Study. J Nurs Adm 2017; 47: 212-218
- 17 Okoli C, Pawlowski SD. The Delphi method as a research tool: an example, design considerations and applications. Info Manag 2004; 42: 15-29
- 18 Rácz I, Jánoki M, Kovács V. Measurement of small-bowel polyp size in patients with Peutz-Jeghers syndrome by using reference granules during video capsule endoscopy. Endoscopy 2007; 1: E41
- 19 Brandt LJ. Terminology for vascular lesions of the GI tract. Gastrointest Endosc 2018; 87: 1595-1596 comment to Becq A, et al. Terminology for vascular lesions of the GI tract. Gastrointest Endosc 2018; 87: 1596