Applications of Augmented Reality in Neuro-Oncology: A Case Series

Abstract

Augmented reality (AR) is a technological tool that superimposes two-dimensional virtual images onto three-dimensional real-world scenarios through the integration of neuronavigation and a surgical microscope. The aim of this study was to demonstrate our initial experience with AR and to assess its application in oncological neurosurgery. This is a case series with 31 patients who underwent surgery at Santa Casa BH for the treatment of intracranial tumors in the period from March 4, 2022, to July 14, 2023. The application of AR was evaluated in each case through three parameters: whether the virtual images auxiliated in the incision and craniotomy and whether the virtual images aided in intraoperative microsurgery decisions. Of the 31 patients, 5 patients developed new neurological deficits postoperatively. One patient died, with a mortality rate of 3.0%. Complete tumor resection was achieved in 22 patients, and partial resection was achieved in 6 patients. In all patients, AR was used to guide the incision and craniotomy in each case, leading to improved and precise surgical approaches. As intraoperative microsurgery guidance, it proved to be useful in 29 cases. The application of AR seems to enhance surgical safety for both the patient and the surgeon. It allows a more refined immediate operative planning, from head positioning to skin incision and craniotomy. Additionally, it helps decision-making in the intraoperative microsurgery phase with a potentially positive impact on surgical outcomes.

Keywords

neuro-oncology - augmented reality - image-guided surgery - neuronavigation

Introduction

Augmented reality (AR) is a useful tool for technological interaction between real and virtual worlds, and its use in medicine has increased in the last decades, including its application in neurosurgery.[1] [2] [3] [4] While virtual reality (VR) creates a virtual scenario that helps simulate situations for educational or preoperative purposes, AR is based on visual two-dimensional (2D) resources applied to three-dimensional (3D) reality.[5] [6] [7]

Technically, AR is projected over the real world and applied on a surgeon-patient interface, helping preoperative planning and surgical decisions.[1] [8] The aim of this study was to demonstrate our initial experience with AR overlay integrated into the microscope oculars and to assess its application in oncological neurosurgery.

Methods

Patient Population

This is a case series of 31 patients who underwent surgery at Santa Casa BH for the treatment of intracranial tumors from March 4, 2022, to July 14, 2023. Out of these, 10 were female and 21 were male. The age ranged from 15 to 75, with a mean of 47 years. Patient information was collected from operative videos and medical records. Approval for this study was obtained from the Institutional Medical Ethics Committee.

Applications of Augmented Reality

The application of AR was evaluated in each case through three parameters: Whether the virtual images aided in the incision and craniotomy, and whether the virtual images aided in intraoperative microsurgery decisions.

Surgical Technique

Instrumentation

Patients were positioned on the operating table and their heads fixed in a radiolucent head-holder (Mayfield; Integra LifeScience, Plainsboro, New Jersey, United States). Patients were registered to the neuronavigation station (StealthStation S7; Medtronic, United States) using face surface-matching systems, relative to the reference star fixed to the head-holder. A separate, dedicated reference star was fixed to the microscope, and the microscope (Pentero 800, Carl Zeiss Meditec, Germany) was then calibrated to the patient's neuronavigation reference star.

Augmented Reality

3D models of patients' heads, skulls, and tumors were preoperatively segmented in a single 3D matrix from the 3D image using the StealthStation S7. These segmentations were intraoperatively injected into the microscope's eyepiece after microscope calibration was performed. First, 3D models of patients' heads were injected to evaluate the accuracy of neuronavigation coregistration visually through the degree of superposition between the patients' heads and their virtual projections. If a mismatch was encountered, the microscope was recalibrated on the reference star, and the accuracy of the neuronavigational data was reassessed.

Surgical Planning

Surgical planning was optimized by the corresponding author according to the relationship between the 2D-visualized tumor and structures. Using microscope-based neuronavigation, a reasonable incision line was delineated. Additionally, bone flap, extent of cortical incision, and white matter trajectory were determined based on the reconstructed structures visualized through the microscope eyepiece.

Intraoperative Guidance Surgical Resection

Virtual images can be visualized as 2D sections, with the structures represented by different colors. In the 2D image scene, reconstructed structures in the focal plane are delineated with solid lines. The above information allows the neurosurgeon to determine the site of surgical manipulation and assess its safety.

Results

Clinical Outcome

A total of 31 patients were included in this study. Demographic and clinical data are presented in [Table 1]. Of the 31 patients, 5 patients developed new neurological deficits postoperatively: 2 patients due to hemorrhage, and 2 due to infection. One patient died, with a mortality rate of 3.0%.

Table 1 Demographic and clinical data of the 31 patients

Data	n (%)
Age (y)
≤ 60	11 (35.5)
> 60	20 (64.5)
Sex
Male	25 (80.7)
Female	06 (19.3)
Preoperative KPS
≥ 70	25 (80.6)
< 70	06 (19.4)
New neurological deficits
Yes	5 (16.1)
No	26 (83.8)
Mortality
Yes	01 (3.2)
No	30 (93.5)

Abbreviation: KPS,Karnofsky Performance Scale.

Gross total resection was achieved in 22 patients (70.9%), while partial resection occurred in 6 cases of glioma, including 2 unplanned cases. For the remaining cases (four patients), gross total resection was not feasible due to their proximity to eloquent structures. Regarding the assessment of the extent of resection, an immediate postoperative magnetic resonance imaging (MRI) was performed. However, in three patients, this was not feasible due to their clinical condition.

Applications of Augmented Reality

In the 31 operated patients, AR was used to guide the incision and craniotomy in each case, leading to improved and precise surgical approaches. As intraoperative guidance ([Table 2]), it proved to be useful in 29 cases: in dysembryoplastic neuroepithelial tumors (DNTs), high-grade and low-grade gliomas (case 1) for defining tumor limits; in metastases, for subcortical tumor localization and defining tumor limits (case 2); in tumors adjacent eloquent areas (Case 3), for defining the area to be resected in relation to the eloquent area; and in deep-seated tumors, for tumor localization and defining tumor limits (case 4).

Table 2 Applications of augmented reality

Tumor classification	Surgical goal (augmented reality)	Localization	Sample size
High-grade glioma	Surgical planning (incision + craniotomy placement) and intraoperative guidance	Right parietal (3) Left parietal (1) Right frontal (1) Left frontal (2) Right temporal (6) Left temporal (6) Left thalamus (1)	20
Low-grade glioma	Surgical planning (incision + craniotomy placement) and intraoperative guidance	Left parietal (1) Left frontal (1) Right temporal (1) Left temporal (1)	4
Metastasis	Surgical planning (incision + craniotomy placement) and intraoperative guidance	Right parietal (1) Left parietal (2) Right frontal (1)	3
Meningioma	Surgical planning (incision + craniotomy placement)	Left frontal (2)	2
Dysembryoplastic neuroepithelial tumor	Surgical planning (incision + craniotomy placement) and intraoperative guidance	Left temporal (1)	1
Teratoma	Surgical planning (incision + craniotomy placement) and intraoperative guidance	Pineal (1)	1

Illustrative Cases

Case 1 (Dysembryoplastic Neuroepithelial Tumor)

A 16-year-old patient with difficult-to-control seizure episodes. T1-weighted MRI showed a hypointense left temporal tumor without contrast enhancement. The patient underwent left temporal craniotomy with a transmiddle temporal gyrus approach guided by AR. Neoplasia diagnosis was DNT. This case is depicted in [Fig. 1].

Fig. 1 Augmented reality (AR) neuronavigation system for computer-assisted volumetric resection of temporal tumor. (A) Surgical planning (StealthStation S7 Surgical Navigation System) delineating tumor (green). (B) Augmented reality holographic projection for guided craniotomy and incision (tumor: yellow). (C) Microsurgical resection guided by holographic projection of the tumor (yellow). (D) T2-weighted magnetic resonance imaging (MRI) revealing the complete resection of the tumor.

Case 2 (Subcortical Tumor)

A 55-year-old patient with a history of renal cell tumor presented with seizure episodes. MRI revealed a right frontal subcortical tumor with significant adjacent edema. The patient underwent right frontal craniotomy and microsurgery guided by AR. Neoplasia diagnosis was kidney neoplasm metastasis. This case is depicted in [Fig. 2].

Fig. 2 Augmented reality (AR) neuronavigation system for computer-assisted volumetric resection of subcortical tumor. (A) Surgical planning (StealthStation S7 Surgical Navigation System) delineating tumor (red). (B) Augmented reality holographic projection for guided craniotomy and incision. (C) Surgical field without AR hologram projection. (D) Surgical field with AR hologram projection of the tumor (yellow and red). (E) Microsurgical resection guided by holographic projection (tumor: yellow). (F) Complete resection confirmed by holographic projection.

Case 3 (Tumors Adjacent Eloquent Areas)

A 33-year-old patient with focal seizures in the right upper limb that are difficult to control. T1-weighted MRI showed a hypointense left frontal tumor without contrast enhancement adjacent to the motor area. The patient underwent surgical treatment guided by AR and brain mapping. Neoplasia diagnosis was isocitrate dehydrogenase mutant astrocytoma, central nervous system World Health Organization grade II. This case is depicted in [Fig. 3].

Fig. 3 Augmented reality (AR) neuronavigation system for computer-assisted volumetric resection of left frontal tumor adjacent to the motor area. (A) Surgical planning (StealthStation S7 Surgical Navigation System), delineating tumor (green) and motor cortex (red). (B) Surgical field with augmented reality hologram projection (tumor: green and cortex motor: red). (C) Brain mapping confirming the position of the motor cortex, corresponding to the holographic projection of the cortex motor (red). (D) Microsurgical resection guided by cerebral mapping and augmented reality.

Case 4 (Deep-Seated Tumors)

An 18-year-old patient with difficult-to-control seizure episodes. MRI revealed a left mesial temporal tumor. Surgical treatment is planned through a supracerebellar transtentorial approach guided by AR. Neoplasia diagnosis was glioneural neoplasm. This case is depicted in [Fig. 4].

Fig. 4 Augmented reality (AR) neuronavigation system for computer-assisted volumetric resection of left mesial temporal tumor. (A) Surgical planning (StealthStation S7 Surgical Navigation System), delineating tumor (green) and motor tract (blue). (B) Tentorium (arrow white) exposure through the supracerebellar approach. (C) Definition of the tentorium opening point (arrow white) under the tumor, guided by augmented reality, with the assistance of the hologram from neuronavigation (arrow yellow). (D) Location of the tumor confirmed by the hologram of the tumor (yellow) and neuronavigation. (E) Microsurgical resection guided by augmented reality. (F) T2-weighted magnetic resonance imaging (MRI) revealing the gross total resection of the tumor.

Discussion

AR holds substantial value for procedures in neuro-oncology, particularly in relation to incisions and craniotomies. Furthermore, virtual images can serve as intraoperative guidance.[8] [9]

A learning curve required for proper and optimized use of resources is based on manipulating technological tools in conjunction with basic anatomical and surgical knowledge,[10] essential for every neurosurgeon. Expert neurosurgeons and trainee neurosurgery residents are capable of acquiring the necessary knowledge to apply a kind of protocol for constructing 3D projection scenarios from 2D neuroimaging exams.[11]

The benefits from AR use in neurosurgical cases for detailed preoperative planning and image-guided surgery could be smaller and more aesthetic incisions, safer and strategic surgical approaches, intraoperative decision-making, and enlargement of neoplasm resections allowing subtotal or even total resections with potential positive prognostic impact.[2] [6] [12] [13] Currently, in the literature, there are few studies showing the applications of AR in oncological neurosurgery. Watanabe et al published a study involving six patients who underwent tumor resection surgery. The primary advantage of AR lies in its ability to provide volumetric navigation, in stark contrast to the traditional point-to-point navigation method. It extends AR images directly onto real surgical images, thus helping the surgeon to integrate these two dimensions intuitively.[14]

Yang et al conducted a prospective randomized clinical trial for AR use in the presurgical planning of skull base tumor resection. The technology was not only found to be useful for localizing lesions and designing patient-specific trajectories but also resulted in significantly reduced postoperative stay and complications due to cerebrovascular injury.[15]

Sun et al,[13] in a study involving 79 glioma patients and 55 control subjects, demonstrated that AR protocols incorporating functional neuronavigation and intraoperative MRI offered valuable insights for performing tailored and optimized surgeries. Complete resection was accomplished in 55 out of 79 (69.6%) glioma patients and 20 out of 55 (36.4%) control subjects, with average resection rates of 95.2 ± 8.5% and 84.9 ± 15.7%, respectively. The researchers concluded that integrating VR and AR through functional neuronavigation and intraoperative MRI could facilitate the resection of gliomas involving eloquent areas.

In the present study, we demonstrate that AR is useful in various surgical steps, including incision, craniotomy, and intraoperative microsurgery, for both superficial and deep-seated tumors ([Table 2]; [Figs. 1] [2] [3] [4]). Complete tumor resection was achieved in 22 (70.9%) cases. In relation to intraoperative microsurgery guidance, AR was useful in finding subcortical lesions, defining lesion limits in relation to eloquent areas, and aiding in the approach of deep lesions. In addition, AR facilitated the guidance of microsurgical techniques without disrupting workflow and contributed to improvements in efficiency and ergonomics.

The critics of the AR tools highlight the increased surgical time associated with difficulties in managing such technologies.[5] [6] However, a progressive reduction in the additional time required for surgical setup can be noted as the specialist's proficiency with the procedure increases. One of the benefits may be an increment of surgeon's attention, as there is no need to look away from the microscope to take visual information related to neuroimage once it is applied in the surgical field in real time.[16]

Limitations

Intraoperative guidance is a major limitation of AR, attributed to the failure to account for brain shift during surgery, which can lead to potential inaccuracies. This may be remedied through the use of intraoperative anatomic and functional imaging. One example is the Intraoperative Brain Imaging System, an open-source neurosurgery platform that improves surgical navigation accuracy by using intraoperative ultrasound to update patient models through an AR view.[17]

Conclusion

The application of AR seems to enhance surgical safety for both the patient and the surgeon. It allows for more refined immediate operative planning, from head positioning to skin incision and craniotomy. Additionally, it helps decision-making in the intraoperative microsurgery phase with a potentially positive impact on surgical outcomes. AR is an applicable tool with promising prospects in neurosurgery, and clinical trials should be increasingly encouraged.

Conflict of Interest

None declared.

• References

• 1 Cannizzaro D, Zaed I, Safa A. et al. Augmented reality in neurosurgery, state of art and future projections. A systematic review. Front Surg 2022; 9: 864792
• 2 Mikhail M, Mithani K, Ibrahim GM. Presurgical and intraoperative augmented reality in neuro-oncologic surgery: clinical experiences and limitations. World Neurosurg 2019; 128: 268-276
• 3 Shuhaiber JH. Augmented reality in surgery. Arch Surg 2004; 139 (02) 170-174
• 4 Scherschinski L, McNeill IT, Schlachter L. et al. Augmented reality-assisted microsurgical resection of brain arteriovenous malformations: illustrative case. J Neurosurg Case Lessons 2022; 3 (25) CASE21135
• 5 Li CR, Shen CC, Yang MY, Tsuei YS, Lee CH. Intraoperative augmented reality in microsurgery for intracranial arteriovenous malformation: a case report and literature review. Brain Sci 2023; 13 (04) 653
• 6 Cabrilo I, Bijlenga P, Schaller K. Augmented reality in the surgery of cerebral arteriovenous malformations: technique assessment and considerations. Acta Neurochir (Wien) 2014; 156 (09) 1769-1774
• 7 Besharati Tabrizi L, Mahvash M. Augmented reality-guided neurosurgery: accuracy and intraoperative application of an image projection technique. J Neurosurg 2015; 123 (01) 206-211
• 8 Guha D, Alotaibi NM, Nguyen N, Gupta S, McFaul C, Yang VXD. Augmented reality in neurosurgery: a review of current concepts and emerging applications. Can J Neurol Sci 2017; 44 (03) 235-245
• 9 Mazur T, Mansour TR, Mugge L, Medhkour A. Virtual reality-based simulators for cranial tumor surgery: a systematic review. World Neurosurg 2018; 110: 414-422
• 10 Chou B, Handa VL. Simulators and virtual reality in surgical education. Obstet Gynecol Clin North Am 2006; 33 (02) 283-296 , viii–ix
• 11 Orringer DA, Golby A, Jolesz F. Neuronavigation in the surgical management of brain tumors: current and future trends. Expert Rev Med Devices 2012; 9 (05) 491-500
• 12 Ivan ME, Eichberg DG, Di L. et al. Augmented reality head-mounted display-based incision planning in cranial neurosurgery: a prospective pilot study. Neurosurg Focus 2021; 51 (02) E3
• 13 Sun GC, Wang F, Chen XL. et al. Impact of virtual and augmented reality based on intraoperative magnetic resonance imaging and functional neuronavigation in glioma surgery involving eloquent areas. World Neurosurg 2016; 96: 375-382
• 14 Watanabe E, Satoh M, Konno T, Hirai M, Yamaguchi T. The trans-visible navigator: a see-through neuronavigation system using augmented reality. World Neurosurg 2016; 87: 399-405
• 15 Yang DL, Xu QW, Che XM, Wu JS, Sun B. Clinical evaluation and follow-up outcome of presurgical plan by Dextroscope: a prospective controlled study in patients with skull base tumors. Surg Neurol 2009; 72 (06) 682-689 , discussion 689
• 16 Shan Q, Doyle TE, Samavi R, Al-Rei M. Augmented reality based brain tumor 3D visualization. Procedia Comput Sci 2017; 113: 400-407
• 17 Drouin S, Kochanowska A, Kersten-Oertel M. et al. IBIS: an OR ready open-source platform for image-guided neurosurgery. Int J CARS 2017; 12 (03) 363-378

Publication History

Article published online:
08 July 2024