Accuracy of Coronary Plaque Detection and Assessment of Interobserver Agreement for Plaque Quantification Using Automatic Coronary Plaque Analysis Software on Coronary CT Angiography

Abstract

Purpose: To evaluate the accuracy of automatic plaque detection and the interobserver agreement of automatic versus manually adjusted quantification of coronary plaques on coronary CT angiography (cCTA) using commercially available software.

Materials and Methods: 10 cCTA datasets were evaluated using plaque software. First, the automatically detected plaques were verified. Second, two observers independently performed plaque quantification without revising the automatically constructed plaque contours (automatic approach). Then, each observer adjusted the plaque contours according to plaque delineation (adjusted approach). The interobserver agreement of both approaches was analyzed.

Results: 32 of 114 automatically identified findings were true-positive plaques, while 82 (72 %) were false-positive. 20 of 52 plaques (38 %) were missed by the software (false-negative). The automatic approach provided good interobserver agreement with relative differences of 0.9 ± 16.0 % for plaque area and –3.3 ± 33.8 % for plaque volume. Both observers independently adjusted all contours because they did not represent the plaque delineation. Interobserver agreement decreased for the adjusted approach with relative differences of 25.0 ± 24.8 % for plaque area and 20.0 ± 40.4 % for plaque volume.

Conclusion: The automatic plaque analysis software is of limited value due to high numbers of false-positive and false-negative plaque findings. The automatic approach was reproducible but it necessitated adjustment of all constructed plaque contours resulting in deterioration of the interobserver agreement.

Key points:

• Automatic plaque detection is limited due to high false-positive and false-negative findings.

• Automatic plaque quantification was reproducible in the few accurately detected plaques.

• The automatically constructed contours did not represent the plaque delineation.

• Both observers independently adjusted the plaque contours.

• Manual adjustment of plaque contours reduced the interobserver agreement.

Citation Format:

• Laqmani A, Klink T, Quitzke M et al. Accuracy of Coronary Plaque Detection and Assessment of Interobserver Agreement for Plaque Quantification Using Automatic Coronary Plaque Analysis Software on Coronary CT Angiography. Fortschr Röntgenstr 2016; 188: 933 – 939

Zusammenfassung

Ziel: Evaluation der Detektionsgenauigkeit und der Interobserver-Übereinstimmung von automatischer gegenüber manuell adjustierter Quantifizierung von Koronarplaques mit der Koronar-CT-Angiografie (cCTA) unter Verwendung einer kommerziell erhältlichen Software.

Material und Methoden: 10 cCTA-Datensätze wurden unter Verwendung einer Plaqueanalyse-Software evaluiert. Als erstes wurden die automatisch detektierten Plaques verifiziert. Danach führten 2 Untersucher unabhängig voneinander die automatische Plaque-Quantifizierung durch ohne die automatisch konstruierten Plaquekonturen zu revidieren (automatische Vorgehensweise). Anschließend konnten die Untersucher die Konturen gemäß der Plaquegrenzen anpassen (adjustierte Vorgehensweise). Die Interobserver-Übereinstimmung beider Vorgehensweisen wurde überprüft.

Ergebnisse: 32 der 114 automatisch identifizierten Befunde waren richtig-positiv, während 82 (72 %) der Befunde falsch-positiv waren. 20 der 52 (38 %) Plaques wurden nicht von der Software detektiert (falsch-negativ). Die automatische Vorgehensweise bot eine gute Interobserver-Übereinstimmung mit relativen Unterschieden von 0,9 ± 16,0 % für Plaquefläche und –3,3 ± 33,8 % für Plaquevolumen. Beide Untersucher passten unabhängig von einander alle Plaquekonturen an, da die automatisch konstruierten Konturen nicht den Plaquegrenzen entsprachen. Die anschließende adjustierte Vorgehensweise verschlechterte die Interobserver-Übereinstimmung mit relativen Unterschieden von 25.0 ± 24.8 % für Plaquefläche und 20,0 ± 40,4 % für Plaquevolumen.

Schlussfolgerung: Die untersuchte Software ist wegen vieler falsch-positiven und falsch-negativen Plaque-Detektionen von begrenztem Nutzen. Die automatische Vorgehensweise lieferte reproduzierbare Ergebnisse, benötigte jedoch eine Adjustierung sämtlicher Plaquekonturen, die zu einer Verschlechterung der Interobserver-Übereinstimmung führte.

Kernaussagen:

• Automatische Plaquedetektion ist durch hohe Raten an falsch-positiven und falsch-negativen Befunden limitiert.

• Bei den wenigen korrekt detektierten Plaques ist die automatische Plaquequantifizierung reproduzierbar.

• Die automatisch konstruierten Konturen entsprachen nicht den Plaquegrenzen.

• Folglich passten beide Untersucher unabhängig von einander die Plaquekonturen an.

• Diese Anpassung der Plaquekonturen führte zu einer Herabsetzung der Interobserver-Übereinstimmung.

Key words

atherosclerotic plaque - coronary computed tomography angiography - coronary artery disease - software tools - automatic quantification

Introduction

Coronary CT angiography (cCTA) has become a valuable diagnostic tool for the noninvasive detection of coronary artery disease. This technique has been widely implemented in clinical practice during the last decade [1] [2]. Besides the detection of coronary stenosis, cCTA also allows the assessment of morphologic and geometric characteristics of coronary artery plaques [3] [4] [5] [6]. Reproducible quantification of plaque dimensions by cCTA has great importance for individual risk stratification and therapeutic monitoring [7] [8]. Recent developments of dedicated post-processing tools enable automatic segmentation of the coronary artery tree with automatic plaque detection and quantification of the plaques [9] [10] [11] [12]. Current studies suggest that these tools provide reproducible, observer-independent plaque detection and quantification [9] [10] [11] [12] [13]. However, these studies only evaluated the automatic approach for analyzing plaque dimensions. The accuracy of automatic plaque detection has not been evaluated so far. Furthermore, the effect of a potentially necessary adjustment of the automatically generated incorrect plaque contours on the interobserver agreement was not assessed. Assessment of both approaches is of importance because the automatic approach does not always represent the true plaque dimension, thus requiring manual adjustment of the plaque contours.

The aim of this study was to evaluate the accuracy of automatic plaque detection and the interobserver agreement of automatic versus manually adjusted quantification of coronary plaque dimensions on cCTA using commercially available software.

Materials and Methods

Patient population

Written informed consent was waived for this retrospective study by the institutional review board. In 10 patients (6 men, 4 women, mean age: 63 ± 10 years, range: 46 – 75 years), clinically indicated cCTA was performed for the diagnostic workup of suspected coronary artery disease. The inclusion criterion was high image quality of the coronary arteries in all segments. The exclusion criteria were cCTA scans with artifacts related to motion, heart rhythm disturbances, a metallic foreign material such as pacemakers, and patients with stents or with aortocoronary bypass.

Patient preparation

Patients with heart rates greater than 65 bpm and no contraindications received up to 100 mg of atenolol (Atenolol-ratiopharm^®, ratiopharm GmbH, Ulm, Germany) per os 30 min. before the scan. Immediately prior to the scan, all patients received a single dose of 0.8 mg glycerol nitrate (Nitrolingual^® N-Spray, Pohl-Boskamp, Hohenlockstedt, Germany) sublingually.

cCTA protocol

All examinations were performed with a 256-MSCT scanner (Brilliance iCT, Philips, Best, The Netherlands) using a prospectively ECG-triggered, sequential-axial CT scan protocol (Step & Shoot Cardiac, Philips Healthcare). A patient weight-adjusted volume of 70 – 80 mL of iodinated contrast agent (Imeron 400, Bracco Altana, Dresden, Germany) was injected at a flow rate of 4 ml/s followed by 100 mL of saline solution through an antecubital vein via an 18-gauge catheter. Automatic bolus tracking was employed using a region of interest placed in the descending aorta. The acquisition of cCTA data was started 5 sec. after a preset threshold of 110 Hounsfield units was reached. Scans were performed using the following protocol: detector collimation 128 × 0.625 mm, 256 overlapping slices of 0.625 mm thickness with dynamic z-focal spot, gantry rotation time 270 ms, tube potential 120 kV and tube current-time product of 250 mAs. X-rays were prospectively triggered at 75 % of the R–R interval with use of an X-ray acquisition buffer of ± 90 ms to ensure sufficient data for reconstruction in the case of heart rate variation.

Image analysis

Images were analyzed by commercially available plaque analysis software (Comprehensive Cardiac Analysis, Extended Brilliance Workspace, V4.0; Philips Healthcare, Best, The Netherlands) supplied by the vendor of our CT system. This software automatically performed model-based whole heart segmentation, coronary artery segmentation including luminal and vessel wall contouring, and detection of coronary plaques without any user interference. Two board-certified radiologists with 12 and 6 years of experience in cCTA, who were blinded to the cCTA results, evaluated the accuracy of the automatic plaque detection in consensus reading. The software finding was classified as a true positive result when both observers considered the detected lesion to be plaque. In the event of a false-positive finding, the supposed reason for the software misinterpretation was noted. Plaques that were not detected by the software were classified as false-negative. Plaques that were correctly detected by the software were included in the automatic quantitative analysis. Subsequently, the two observers separately and independently performed automatic plaque segmentation and quantification (automatic approach). Both observers analyzed the plaque area (mm²), plaque volume (mm³), and plaque burden (%). In the case of inaccurate plaque delineation by the software, the observers were requested in a further step to adjust the plaque contours according to their personal assessment of the plaque delineation.

Statistical analysis

Continuous data are expressed as mean ± standard deviation (SD). Interobserver agreement was determined for the automatic and the manually adjusted approach. Absolute and relative differences between observers were calculated as described by Bland and Altman [14]. Relative differences were calculated as difference of two measurements divided by the mean of both measurements multiplied by 100 to give percentages. Variance was calculated as the square of the standard deviation of the relative differences. Unpaired t-tests were used to compare the relative differences, and F-tests were used to compare the variances of the results of the automatic and the manually adjusted approach. Differences between mean plaque dimensions of the automatic and the manual approach were compared using paired t-tests. P < 0.05 was considered statistically significant.

Results

Automatic plaque detection

The software identified 114 findings along the coronary arteries as plaques. Consensus reading revealed that 32 (28 %) of these lesions were true-positive, while 82 (72 %) were false-positive findings ([Table 1]). The software did not detect 20 (38 %) plaques of 52 true coronary artery plaques. There were classified as false-negative findings ([Table 1]).

Table 1 Results of the automatic plaque detection using the plaque analysis software.
Tab. 1 Ergebnisse der automatischen Plaquedetektion unter Verwendung der Plaqueanalysen-Software.

software human readers	plaque present n (%)	plaque absent n (%)	sum n (%)
plaque present n (%)	32 (61 %)  32 (28 %)	20 (39 %)	52 (100 %)
plaque absent n (%)	82 (72 %)	not applicable
sum	114 (100 %)

Reasons for false-positive findings

The majority of false-positive findings (n = 48, 59 %) were related to an intermediate signal in the pericoronary epicardial fat ([Table 2, ] [Fig. 1a]). 20 false-positive findings (24 %) were related to vessel branching ([Fig. 1b]). 6 false-positive findings (7 %) were due to false vessel contouring with placement of the outer border and inner lumen border significantly off the course of the coronary artery ([Fig. 1c]). Contrast in the adjacent vein resulted in 5 false-positive findings (6 %, [Fig. 1 d]). The software marked a kinking of the coronary artery in 3 lesions (4 %) as plaque ([Fig. 1e]).

Table 2 Reasons for false-positive and false-negative plaque analysis software findings.
Tab. 2 Gründe für falsch-positive und falsch-negative Befunde der Plaqueanalysen-Software.

reasons for false-positive plaque findings	n = 82 (100 %)
intermediate signal in pericoronary fat	48 (59 %)
vessel branching	20 (24 %)
false vessel contouring	6 (7 %)
contrast in adjacent vein	5 (6 %)
kinking of coronary artery	3 (4 %)
reasons for false-negative plaque findings	n = 20 (100 %)
small plaque size	16 (80 %)
adjacent plaques, detected as one plaque	4 (20 %)

Fig. 1 Representative examples for false-positive findings (red arrows). The blue lines represent the vessel wall and the red lines represent the border of the vessel lumen generated by the software. The software identified findings between these two lines as plaques. False-positive findings were related to an intermediate signal in the pericoronary epicardial fat a, vessel branching b, false contouring of the coronary artery c, contrast in an adjacent vein d and kinking of the coronary artery e.

Abb. 1 Repräsentative Beispiele für falsch-positive Befunde (rote Pfeilmarkierungen). Die blauen Linien repräsentieren die durch die Software generierten Grenzen der Gefäßwand und die roten Linien repräsentieren die Grenzen des Gefäßlumens. Befunde zwischen diesen beiden Linien wurden von der Software als Plaques identifiziert. Falsch-positive Befunde waren mit perikoronarem epikardialem Fettgewebe a, Gefäßabgängen b, falschen Gefäßkonturierungen c, Kontrast in der angrenzenden Vene d und Gefäßkinking e assoziiert.

Reasons for false-negative findings

With 80 %, a small plaque size was the most common reason for a plaque not being detected with the automatic approach ([Table 2, ] [Fig. 2a]). The mean plaque area of the undetected plaques (3.7 ± 1.1 mm²) was significantly smaller than that of the detected plaques (5.7 ± 2.1 mm²) (P< 0.05). 4 false-negative findings (20 %) were related to two adjacent plaques, which were identified as one plaque by the software ([Fig. 2b]).

Fig. 2 Representative examples for false-negative findings (red arrows). a shows a small plaque, which was missed by the software. b shows two adjacent plaques, which were identified by the software as one plaque.

Abb. 2 Repräsentative Beispiele für falsch-negative Befunde (rote Pfeilmarkierungen). a zeigt einen kleinen Plaque, welcher von der Software nicht detektiert wurde. b zeigt zwei benachbart lokalisierte Plaques, welche von der Software als eine Plaque identifiziert wurde.

Manual adjustment of automatically generated plaque contours

Both observers independently adjusted the automatically generated plaque contours of all 32 plaques because of inadequate plaque delineation by the software. In general, the automatic plaque delineation resulted in insufficient plaque dimensions due to truncation of the plaque borders by the software ([Fig. 3, ] [Table 3]).

Fig. 3 Example of the automatic and manually adjusted approach. a displays the automatic vessel delineation with the vessel wall in blue and the vessel lumen in red. b displays the plaque area marked by the software in yellow. c displays the vessel contour and the resulting plaque area after adjustment of the contours by observer 1.

Abb. 3 Beispiel für die automatische and manuell adjustierte Vorgehensweise. a zeigt die automatisch erstellte Gefäßkontur mit der Gefäßwand in blau und dem Gefäßlumen in rot. b zeigt die Plaquefläche, welche durch die Software gelb markiert wurde. c bildet die Gefäßkontur und die resultierende Plaquefläche nach Adjustierung der Konturen durch Untersucher 1 ab.

Table 3 Interobserver agreement of plaque dimensions using the automatic and the manually adjusted approach.
Tab. 3 Interobserver-Übereinstimmung der Plaquedimensionen für die automatische und die manuell adjustierte Vorgehensweise.

automatic approach	plaque area (mm2)	plaque volume (mm3)	plaque burden (%)
mean ± SD	5.7 ± 2.1	24.8 ± 13.4	35.3 ± 10.5
absolute difference ± SD	0.07 ± 0.8	–0.9 ± 6.8	1.7 ± 5.0
relative difference ± SD (%)	0.9 ± 16.0	–3.3 ± 33.8	5.0 ± 16.1
relative limits of agreement (%)	–30 and 32	–69 and 63	–27 and 37
variance (%2)	256	1142.4	259.2
manually adjusted approach	plaque area (mm2)	plaque volume (mm3)	plaque burden (%)
mean ± SD	6.2 ± 2.4	27.9 ± 16.8	45.2 ± 14.2
absolute difference ± SD	1.5 ± 1.8	3.2 ± 8.4	6.4 ± 13.3
relative difference ± SD (%)	25.0 ± 24.8	20.0 ± 40.4	13.9 ± 27.8
relative limits of agreement (%)	–24 and 74	–59 and 99	–41 and 68
variance (%2)	615	1632.2	772.8
p-value (T-test, (automatic vs. manually adjusted)	< 0.001	< 0.001	= 0.08
p-value (F-test, (automatic vs. manually adjusted)	< 0.01	= 0.32	< 0.001

Interobserver agreement between the automatic and manual approach

The automatic approach resulted in good interobserver agreement for all plaque dimensions shown by low absolute and relative differences between both observers and low limits of agreement ([Table 3]). Manual adjustment of the plaque contours resulted in a significant increase in the differences between both observers for plaque area (P< 0.001) and plaque volume (P< 0.001) and in a significant increase in the limits of agreement for plaque area (P< 0.01) and plaque burden (P< 0.001). For example, the good interobserver agreement for plaque area with a relative difference of 0.9 ± 16.0 % decreased after contour adjustment to 25.0 ± 24.8 % (P< 0.001), indicating a systematic overestimation of plaque area by observer 1 compared to observer 2 ([Table 3, ] [Fig. 4]). The corresponding variance increased from 256 %² to 615 %² (P< 0.01, [Table 3]).

Fig. 4 Bland-Altman plots show the relative differences and limits of agreement of plaque dimensions comparing the automatic and manually adjusted analysis approach.

Abb. 4 Bland-Altman Plots zeigen die relativen Unterschiede und die Grenzen der Übereinstimmung der Plaque-Dimensionen für die automatische und die manuell adjustierte Vorgehensweise.

Discussion

We assessed the diagnostic performance of commercially available plaque analysis software with respect to the accuracy of automatic plaque detection and interobserver agreement of plaque quantification using an automatic approach and a manually adjusted approach. The major findings of our study were: First, the software provided a high number of false-positive plaque findings (72 %) and false-negative findings (38 %). Second, the automatic approach for plaque quantification resulted in good interobserver agreement. However, both observers independently adjusted the automatic contours in all plaques because of inadequate plaque delineation by the software. Third, the good interobserver agreement significantly decreased after the necessary contour adjustment.

The high false-positive plaque detection rate of 72 % represented a major limitation and required an intensive review process to check the results of the software. Our review process revealed that an intermediate signal in the pericoronary epicardial fat was with 59 % the main reason for false-positive findings. This finding indicates that the software has a low signal threshold to detect plaques. Despite this low threshold, the software failed to detect 38 % of plaques, with small plaque sizes being the main reason for false-negative findings with 80 %. Our data show that the low threshold does not guarantee a high sensitivity for detecting plaques. Other false-positive findings are related to wrong segmentation of the coronary artery. With 24 %, vessel branching represented the second most common reason for false-positive plaque findings.

The good interobserver agreement with low limits of agreement of the automatic approach was decreased by the necessity for contour adjustment in all detected plaques. Both observers independently rated that the software contours did not correspond with the true plaque delineation. The necessary adjustment of the plaque contours resulted in significant worsening of the interobserver agreement. This observed deterioration of the interobserver agreement is most likely related to the fact that no precise rules for adjusting the contours were predefined and that the observers were not specially trained in the tracing of plaque contours. Previous work has shown that training and adherence to predefined rules substantially improves the interobserver agreement of data analysis based on contour tracing [15] [16].

Previous studies have evaluated the reproducibility of plaque quantification using the currently analyzed software [3] [9] [17]. These studies assessed the manual approach after adjustment of the automatically generated contours. Similar to our data, Klass et al. [17] found low reproducibility for plaque quantification with a relative interobserver agreement for plaque volume of – 44 ± 46 %. They reported that automatic plaque contouring was less successful at delineating plaques with greater calcium load, thus more manual correction of calcified lesions was required. This result supports our observation of the need for manual adjustment of the automatic plaque contours using this software. Korosoglou et al. [3] assessed the same software version and reported a fairly good interobserver variability of 13 % for the assessment of plaque volume. However, the standard deviation of their measurements was not reported, so it is possible that a substantial variability in their measurements was present. Concerning the accuracy of plaque segmentation, Korosoglou et al. [3] stated that plaque contours were manually edited if necessary. However, the contour correction frequency was not reported. Therefore, it is possible that the reported low interobserver variability was related to a primary automatic approach with only little human interaction.

Study limitations

At the first glance the small number of patients may represent a limitation of our study. However, it has to be emphasized that the software detected a high number of 114 plaques, of which 82 (72 %) were false-positive findings. Additionally, the software missed 20 of 52 plaques resulting in 38 % false-negative findings. In the 32 correctly identified plaques, both observers independently adjusted the automatically generated plaque contours, because the contours were not accurate. Both findings represent a limitation of the software and not of the study design. The inclusion of more patients only would have resulted in a duplication of the reported results with identical conclusions, since none of patients had reduced image quality as a potential explanation for the low performance of the software. Inclusion of cCTA examinations with reduced image quality most likely would have further decreased the performance of the software.

Finally, it has to be underlined that our results only relate to the investigated software version and cannot be applied to software versions that have since been released or to software tools from other vendors.

Conclusion

Use of the investigated plaque analysis software is of limited value because the software identified a high number of false-positive and false-negative plaques, thus requiring careful revision of the automatically detected plaques. Uncritical clinical use of the software would result in an erroneous plaque detection, consequently resulting in potential over- or underdiagnosis of the presence or absence of coronary artery disease. Therefore, we recommend users of this automatic plaque analysis software to critically review their software results before applying them in a routine clinical setting. The fact that both observers independently adjusted the automatically generated plaques contours represents another software limitation, resulting in a significantly reduced interobserver agreement for the investigated plaque dimensions. This low interobserver agreement highlights the difficulty regarding serial measurement of coronary plaque development, e. g. under lipid lowering therapy, using the studied software.

• References

• 1 Taylor AJ, Cerqueira M, Hodgson JM et al. ACCF/SCCT/ACR/AHA/ASE/ASNC/NASCI/SCAI/SCMR 2010 Appropriate Use Criteria for Cardiac Computed Tomography. A Report of the American College of Cardiology Foundation Appropriate Use Criteria Task Force, the Society of Cardiovascular Computed Tomography, the American College of Radiology, the American Heart Association, the American Society of Echocardiography, the American Society of Nuclear Cardiology, the North American Society for Cardiovascular Imaging, the Society for Cardiovascular Angiography and Interventions, and the Society for Cardiovas- cular Magnetic Resonance. Circulation 2010; 122: e525-e555
• 2 Andreini D, Pontone G, Mushtaq S et al. A long-term prognostic value of coronary CT angiography in suspected coronary artery disease. JACC Cardiovasc Imaging 2012; 5: 690-701
• 3 Korosoglou G, Mueller D, Lehrke S et al. Quantitative assessment of stenosis severity and atherosclerotic plaque composition using 256-slice computed tomography. Eur Radiol 2010; 20: 1841-1850
• 4 Hoffmann H, Frieler K, Hamm B et al. Intra- and interobserver variability in detection and assessment of calcified and noncalcified coronary artery plaques using 64-slice computed tomography: variability in coronary plaque measurement using MSCT. Int J Cardiovasc Imaging 2008; 24: 735-742
• 5 Knollmann F, Ducke F, Krist L et al. Quantification of atherosclerotic coronary plaque components by submillimeter computed tomography. Int J Cardiovasc Imaging 2008; 24: 301-310
• 6 Pundziute G, Schuijf JD, Jukema JW et al. Head-to-head comparison of coronary plaque evaluation between multislice computed tomography and intravascular ultrasound radiofrequency data analysis. JACC Cardiovasc Interv 2008; 1: 176-182
• 7 Burgstahler C, Reimann A, Beck T et al. Influence of a lipid-lowering therapy on calcified and noncalcified coronary plaques monitored by multislice detector computed tomography: results of the New Age II Pilot Study. Invest Radiol 2007; 42: 189-195
• 8 Dey D, Schuhbaeck A, Min JK et al. Non-invasive measurement of coronary plaque from coronary CT angiography and its clinical implications. Expert Rev Cardiovasc Ther 2013; 11: 1067-1077
• 9 Lee MS, Chun EJ, Kim KJ et al. Reproducibility in the assessment of noncalcified coronary plaque with 256-slice multi-detector CT and automated plaque analysis software. Int J Cardiovasc Imaging 2010; 26: 237-244
• 10 Busch S, Johnson TR, Nikolaou K et al. Visual and automatic grading of coronary artery stenoses with 64-slice CT angiography in reference to invasive angiography. Eur Radiol 2007; 17: 1445-1451
• 11 Oberoi S, Meinel FG, Schoepf UJ et al. Reproducibility of noncalcified coronary artery plaque burden quantification from coronary CT angiography across different image analysis platforms. Am J Roentgenol 2014; 202: W43-W49
• 12 Versteylen MO, Kietselaer BL, Dagnelie PC et al. Additive value of semiautomated quantification of coronary artery disease using cardiac computed tomographic angiography to predict future acute coronary syndrome. J Am Coll Cardiol 2013; 61: 2296-2305
• 13 Dey D, Schepis T, Marwan M et al. Automated three-dimensional quantification of noncalcified coronary plaque from coronary CT angiography: comparison with intravascular US. Radiology 2010; 257: 516-522
• 14 Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986; 1: 307-310
• 15 Groth M, Muellerleile K, Klink T et al. Improved agreement between experienced and inexperienced observers using a standardized evaluation protocol for cardiac volumetry and infarct size measurement. Rofo 2012; 184: 1131-1137
• 16 Karamitsos TD, Hudsmith LE, Selvanayagam JB et al. Operator induced variability in left ventricular measurements with cardiovascular magnetic resonance is improved after training. J Cardiovasc Magn Reson 2007; 9: 777-783
• 17 Klass O, Kleinhans S, Walker MJ et al. Coronary plaque imaging with 256-slice multidetector computed tomography: interobserver variability of volumetric lesion parameters with semiautomatic plaque analysis software. Int J Cardiovasc Imaging 2010; 26: 711-720