Streaming 2D-Endoscopic Video into an Augmented Reality Headset Display: A Feasibility Study

Objective: Augmented reality (AR) applications in neurosurgery have rapidly grown over the last 6 years with most efforts concentrated in surgical guidance. To date, there have been no studies in streaming video feeds in the operating room (OR) to an AR heads-up display, such as that from the endoscope used in endonasal approaches. The primary objective of this study was to determine if endoscopic footage could be streamed into an AR heads-up display with minimal latency to allow for use in the operating room. A secondary objective was to assess hologram coloration and resolution.

Methods: This feasibility study investigated whether a head-mounted AR device could not only display 2D-endoscope video but also with low enough latency for quick completion of fine motor tasks. Two setups were used. In the first, the endoscope video was accessed on an online streaming application used for remote viewing, available in all ORs. Once the video stream was opened on a desktop with teleconferencing capability, a teleconference call was initiated between the desktop and AR headset, then the desktop's screen with the endoscope video was shared. In the subsequent construct, the online streaming application was bypassed using a DVI to HDMI cable linked directly to the endoscope's camera box, such as the Image 1S (Karl Storz, El Segunda, California, United States). Then an HDMI to USB-C capture card connected this output to a laptop, creating a live display with negligible latency. This laptop teleconferenced with the AR headset. Overall latency was measured in both constructs. Each construct's effect on a surgeon's ability to suture and manipulate cadaveric tissue was observed. Coloration and resolution were juxtaposed to the traditional monitor display to determine subjective hologram quality, both in the cadaver lab and OR.

Results: The initial construct created multilayered streaming that resulted in 700 milliseconds (ms) of latency. These conditions led to error-filled, slow suturing from dyssynchronous visual-tactile feedback. The second construct reduced latency by 57% to approximately 300 ms which improved speed and coordination of the surgeon's suturing. Visual-tactile synchrony was still an issue. The second construct was used in the cadaver lab. Tissue was manipulated fluidly without overshooting structures. The sphenoid sinus was drilled without unintentional disruption of anatomy. Lastly, coloration and resolution of the hologram was comparable to the standard monitored display both in the lab and OR. Shine through of background objects was not distracting in either environment.

Conclusion: Streaming 2D endoscopic video into an AR headset was done with low enough latency to allow for slow but safe fine motor task completion. Latency is not low enough for intraoperative use; however, since there is still a noticeable separation of visual and tactile feedback. Target latency to achieve perceived visual-tactile feedback synchrony is 150 ms. Future directions will investigate new hardware constructs. The effect on latency of incorporating additional video feeds into the AR display, such as the feed from patient vital signs, will be explored.

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Publication History

Article published online:
15 February 2022