Augmented Reality in Neurosurgery: A Review of Current Concepts and Emerging Applications

Abstract

Augmented reality (AR) superimposes computer-generated virtual objects onto the user’s view of the real world. Among medical disciplines, neurosurgery has long been at the forefront of image-guided surgery, and it continues to push the frontiers of AR technology in the operating room. In this systematic review, we explore the history of AR in neurosurgery and examine the literature on current neurosurgical applications of AR. Significant challenges to surgical AR exist, including compounded sources of registration error, impaired depth perception, visual and tactile temporal asynchrony, and operator inattentional blindness. Nevertheless, the ability to accurately display multiple three-dimensional datasets congruently over the area where they are most useful, coupled with future advances in imaging, registration, display technology, and robotic actuation, portend a promising role for AR in the neurosurgical operating room.

Résumé

Réalité augmentée en neurochirurgie : revue des concepts actuels et applications émergeantes. La réalité augmentée (RA) superpose des objets virtuels générés par ordinateur à la vision du monde réel de l’utilisateur. Parmi les disciplines médicales, la neurochirurgie a longtemps été à l’avant-garde de la chirurgie guidée par imagerie et continue de repousser les frontières de la technologie de RA en salle d’opération. Nous avons procédé à une revue systématique afin d’explorer l’histoire de la technologie de RA en neurochirurgie et nous avons examiné la littérature portant sur les applications neurochirurgicales actuelles de la RA. Il existe des défis importants dans ce domaine dont l’erreur d’alignement, la perception altérée de la profondeur, l’asynchronie temporelle visuelle et tactile et l’aveuglement due à l’inattention de l’opérateur. Néanmoins, la capacité de permettre une visualisation précise de multiples ensembles de données tridimensionnelles de façon congruente sur la zone où elles sont le plus utiles, couplée à des progrès qui seront réalisés en imagerie, en inscription, en technologie d’affichage et en actionnement robotique laisse entrevoir un rôle prometteur de la RA en salle d’opération neurochirurgicale.

Introduction

Intraoperative image guidance has been used in multiple surgical disciplines over the past two decades for localizing subsurface targets that cannot be visualized directly. Although significant advances have been made, current navigation paradigms require surgeons to mentally transform two-dimensional (2D) patient-specific images (e.g. computed tomography [CT] or magnetic resonance imaging [MRI]) into three-dimensional (3D) anatomy, as well as 3D computer-rendered anatomy to the patient, and subsequently manipulate instruments in the surgical field while looking at a separated display. The promise of congruent virtual and physical realms is held by augmented reality (AR) systems, in which computer-generated 2D or 3D images are superimposed onto a user’s vision of the real world. ¹ This contrasts with virtual reality (VR), in which the user is fully immersed in a computer-generated environment without real-world input, ² impractical in an operating room but useful in simulation exercises. ³

Although not computer-generated, the first system overlaying a virtual image registered to a hidden object was described in 1938 in Austria, using a system of x-ray tubes and mirrors to reveal the position of a hidden bullet. ⁴ Head-up displays were developed in the 1940s to display radar information and artificial horizons in military aircraft; however, it was not until 1968 that a tracked head-mounted display (HMD) was developed by Sutherland. ⁵ With a mechanical ceiling-mounted head position tracking mechanism, this device allowed the overlay of analog line drawings onto the user’s vision of the real world (Figure 1). Medical applications of AR first began in the mid-1980s, with the augmentation of a neurosurgical monoscopic operating microscope with CT images ⁶ and the development of a video see-through HMD in the early 1990s for the augmentation of ultrasound images. ⁴

Figure 1 The first described head-mounted display, with ceiling-mounted mechanical head position tracking mechanism. ⁵

Surgical applications of AR have been reviewed extensively over the past decade, with several recent systematic reviews. ^{2 , 4 , 7 , 8} Reviews of AR applications specifically in neurosurgery, however, are limited, with recent articles relatively minimal in scope. ^{9 , 10} Here, we review the history of AR particularly as it pertains to neurosurgery, detail the modern paradigms of AR setups as well as their current neurosurgical applications, and explore challenges and future directions in the field.

Components of AR

Surgical AR systems comprise three core components. First, a virtual image or environment must be modelled. In modern AR systems, using neurosurgery as an example, this typically involves a computer-generated 3D reconstruction of a subsurface target, often sourced from segmented cross-sectional imaging (CT or MRI), with color or texture-coded differentiation between anatomic structures. Virtual images are overlaid on the user’s vision of the real world classically by solid or wire-mesh overlays (Figure 2). Nonphotorealistic, or “inverse-realism,” augmentation techniques may improve visualization and depth perception. ¹¹ In contrast with earlier systems, modern AR devices use on-demand augmentation whereby virtual image layers may be removed when desired.

Figure 2 Current methods of overlaying virtual content. The example shown is a minimally invasive lumbar hemilaminectomy captured with a head-mounted camera. (A) No augmentation; (B) solid overlay; and (C) wire-mesh overlay.

The second requirement for AR systems is the registration of virtual environments with real space. This is particularly critical in AR because our perceptual systems are more sensitive to visual misalignments than to the kinetic errors common in VR. ¹² Registration may be accomplished through a number of means and is the subject of significant ongoing research. Frame-based techniques create a rigid 3D Cartesian system in which the position and pose of imaging devices may be determined, allowing registration of a virtual environment as well as rapid updates as the real-world viewing position changes. More commonly, frameless registration methods using the point-matching of virtual and real spaces using known or rigid anatomic landmarks, including bony landmarks for cranial and spinal surgery, and (relatively) stationary “vessel signatures” for vascular procedures are used. ^{2 , 13} This is often enhanced with surface-mapping using infrared light-emitting diode (IR-LED)-tracked instruments, or laser range scanners. ¹⁴

The final requirement for functional AR is a display technology to combine the virtual and real environments. Display techniques may be categorized broadly as HMDs, augmented external monitors, augmented optics, augmented windows, and image projections (Figure 3). ⁴ Virtual environments may be projected onto an HMD, overlaid onto either the user’s vision of the real world (optical see-through), or onto a video feed of the real environment (video see-through). Augmented monitors are simply standalone screens displaying virtual content overlaid onto a video feed from the real world. Augmented optics involve direct augmentation of the oculars of an operating microscope or binoculars. Augmented windows are an emerging technology in which a semitransparent screen is placed directly over the surgical site, allowing the display of virtual objects (on the screen) directly over the real object underneath. Last, virtual environments may be projected directly onto the patient using a standard computer projector, without a separate display. ¹⁵

Figure 3 Examples of current AR display methods. (A) Video see-through HMD, with head-mounted video camera ³⁵ ; (B) user’s view of output from video-pass through HMD, with augmentation calibration marker (gray) and overlaid vertebroplasty needle trajectories (red, yellow) ³⁵ ; and (C) image projection of cortex and deep lesion (red) onto skin surface for incision planning. ¹⁵

AR in Neurosurgery

Neurosurgery has long been at the forefront of image-guided surgery, with the first frameless stereotactic navigation systems being developed for intracranial tumor localization in the early 1990s. It is unsurprising that many surgical applications of AR were pioneered for neurosurgery (Figure 4). The first augmented operating microscope was developed in 1985 at Dartmouth for cranial surgery. ⁶ Segmented 2D preoperative CT slices were displayed monoscopically into the optics of a standard operating microscope, which was registered to the operating table using an acoustic localizer system. Real-time tool tracking was not possible however, because repositioning of the microscope necessitated reregistration with the operating table, taking ~20 seconds. It was not until 1995 that the first augmented stereo microscope, offering accurate depth perception, was developed in the United Kingdom. ¹⁶ This system allowed for the multicolor display of segmented 3D cross-sectional imaging data directly into the microscope oculars, as solid or wire-mesh overlays. Intraoperative registration accuracy of 2mm to 3 mm was reported.

Figure 4 Timeline of neurosurgical applications of augmented reality.

The first video augmentation devices in neurosurgery were developed in 1994. ^{17 , 18} In both systems, a video camera was mounted on a stereotactic frame, allowing registration to the operating table, and trained on the patient from the surgeon’s presumed perspective. Multicolor 3D reconstructions of segmented CT or MRI data were overlaid onto the video feed on an external display. The surgery was subsequently performed either under direct vision or via the external screen.

AR for endovascular applications was demonstrated first in 1998, overlaying reconstructed preoperative vascular anatomy, from CT or MR angiography, onto a virtual screen displaying real-time x-ray fluoroscopy data. ¹⁹ With registration accuracies of 2 to 3 mm, this system was intended to obviate the additional contrast load required to generate angiographic roadmaps.

Although endoscopes have been in use in general surgery since the 1980s, the first augmented neurosurgical endoscope was developed in 2002 for endonasal transsphenoidal approaches. ²⁰ Volumetric 3D reconstructions of preoperative CT or MRI data were overlaid onto the endoscope video feed on an external display. IR-LEDs were used to track the endoscope relative to the patient, allowing display of the endoscope trajectory relative to delicate neurovascular structures.

Current Applications

A comprehensive review was performed on the recent literature pertaining to AR for human clinical neurosurgical applications. MEDLINE, Web of Science, and Scopus were searched for English-language literature from 2000 through 2015 using the search terms (augment* AND reality AND (neurosurgery OR spine OR endovascular)). The search was conducted in August 2016. Nonduplicated, peer-reviewed original investigations encompassing in vivo, human phantom, or human cadaveric specimens were included. Of 126 screened abstracts, 44 were either not relevant to AR or neurosurgery or were commentaries on other primary investigations; 14 were reports on VR devices. The full texts of the remaining 68 articles were reviewed independently by two authors (DG, NMA). From these were excluded 15 reviews of previously published literature and 20 technical/engineering papers without clinical translation, leaving 33 primary manuscripts (Table 1). ^{13 , 15} Given the significant heterogeneity in reported outcomes, pooled statistics were not computed.

Table 1 Summary of studies on neurosurgical applications of AR

* All values are presented as means or percentages.

AVM=arteriovenous malformation; CTA=computed tomography angiography; fps=frames per second; MCA=middle cerebral artery; mRS=modified Rankin score; NIR=near-infrared; OA=occipital artery; PICA=posterior inferior cerebellar artery; STA=superficial temporal artery; US=ultrasound.

Of the 33 articles on neurosurgical AR, the majority were for applications in tumor resection (16 articles, 48%), open neurovascular surgery (9 articles, 27%), or spinal procedures (7 articles, 21%). Four articles pertained to the stereotactic localization of ventriculostomy catheters or simply a tracked probe (12%) and one pertained to cortical resection in epilepsy. Notably, there were no recent publications on AR for endovascular procedures. Of the 33 total studies, four assessed the role of AR for trainee simulation (12%), with the remainder devoted to intraoperative applications. Nineteen studies were conducted with some in vivo human clinical testing (58%), whereas 14 were exclusively cadaveric or phantom studies. AR stereomicroscopes were assessed in five studies (15%), although three were from the same center. AR HMDs were investigated in three articles (9%), image projection techniques in four (12%; two from the same center), and AR windows in four (12%; two from the same center). All other studies used external AR displays, either standalone or tablets/smartphones.

Evaluation and reporting of outcomes from the use of AR devices were highly heterogeneous across studies. A summary of reported outcomes for each study is presented in Table 1. Subjective feedback of operator comfort, usability, and/or depth perception were reported by most studies, often dichotomized as “satisfactory/unsatisfactory.” Studies investigating AR simulators typically quantified accuracy for the specific simulated task, for instance, the translational deviation of a virtual ventriculostomy catheter from its ideal target or the percentage of pedicle screws placed in satisfactory position. ^{29 , 38 , 39} For clinical studies, the most commonly quantified metrics included setup time and overall registration error. Overall registration errors were calculated differently between studies, unsurprising given the variety of augmentation techniques used and hence the types of errors introduced. For instance, camera calibration errors apply to any systems using video imaging of the real world, but are obviated in optical see-through HMDs. Error in tracking and transforming eye movements, however, applies primarily to optical HMDs. Errors in virtual image overlay or reprojection, occurring to various extents with each type of augmented display, were typically not reported separately. Nonetheless, overall registration errors for cranial AR ranged from 0.3mm to 4.2 mm, with most studies reporting 2mm to3 mm. This is well within the range of accuracy achieved by current neuronavigation systems.

AR in Vascular Neurosurgery

With the exception of one study in which AR was used by a remote surgeon to guide a carotid endarterectomy ³¹ and one in which volumetric intracranial CT angiography (CTA) data were overlaid onto a real-world video feed, ⁴⁷ AR for vascular neurosurgery has focused on the augmentation of stereomicroscopes. Microscope overlays include either fluorescence images from intraoperative indocyanine green (ICG) angiography or segmented preoperative CTA/magnetic resonance angiography/digital subtraction angiography (DSA). ^{13 , 40 , 41 , 46} Registration of the virtual environment to the patient is done by tracking each component using a standard IR-LED–based neuronavigation system, which would be used typically as standard of care. Verification of registration is performed at each step of skin incision, craniotomy (using standard anatomic landmarks), and arachnoid dissection (by registering the ‘vessel signature’ of the exposed cortex to the preoperative CTA/DSA). ¹³

Overlay of the target vasculature optimizes both skin incision and craniotomy, with a smaller craniotomy fashioned for 63% of AR-guided cases versus without AR in one series of aneurysm clipping. ⁴¹ In this series, AR guidance was felt to be most useful for aneurysms requiring an unusual trajectory or with limited exposition and hidden branches; cases done with AR showed no difference in intraoperative clip correction rates, or 3-month patient functional outcomes, relative to procedures without AR.

For extracranial-intracranial bypass procedures, AR overlays offered the additional advantages of readily identifying donor vessels on the skin surface, facilitating skin incision and vessel harvest. AR guidance proved superior to manual pulse palpation and comparable to Doppler ultrasound or intraoperative DSA-guided donor vessel identification. ¹³ Craniotomy size was also minimized because of AR display of the preoperatively identified recipient vessel sites.

The role of AR in arteriovenous malformation (AVM) resection may be more limited; in the few series to date, although vessel augmentation was useful for skin incision, craniotomy, and resection planning, the complexity of arterial feeders in most AVMs was not resolvable with current systems, particularly in the context of surrounding hemorrhage from preoperative rupture. ⁴⁰ Identification of the depth of feeding arteries with AR views was also problematic, despite the use of manually identified markers on deep feeding arteries. ⁴⁷

Intraoperative setup of the AR stereomicroscope requires approximately 20 additional minutes beyond the registration of a standard optical navigation system: 10 minutes for registration of the microscope and 10 minutes for verification of registration accuracy. ⁴¹ Therefore, there is minimal disruption of the surgical workflow, particularly once the procedure is under way. Segmentation and merging of preoperative cross-sectional imaging as well as DSA, however, does entail additional time before surgery.

AR in Skull Base/Tumor Surgery

As with neurovascular applications, AR guidance is particularly useful in the initial stages of surgery for planning skin incisions and minimizing the extent of craniotomy. When tumor boundaries and planned resection margins are segmented preoperatively, along with adjacent neurovascular structures to be preserved, AR overlays of these targets facilitate maximal safe resection, particularly for gliomas. In one series of 74 patients, 64 with primary or recurrent gliomas, AR overlay of volumetric CT/MRI data was achieved with no additional surgical time or complications and reduced both intensive care unit and hospital length of stay by 40% to 50% relative to non-AR cases. ²³ AR also offers direct visualization of superficial and deep venous structures, which is particularly useful in the resection of large convexity, parasagittal, and parafalcine meningiomas. ²⁶ However, as with any neuronavigation system guided by preoperative imaging, current AR devices are unable to account for brain shift during cranial surgery, which may represent a significant source of registration error once large volumes of tumor or cerebrospinal fluid have been removed. ⁵¹ Recent advancements in surgical navigation include real-time registration updates from intraoperative 3D ultrasound, accounting for brain shift on a time scale on the order of minutes. ⁵² Ultrasound-based registration updates are now beginning to be applied to AR views for tumor surgery. ⁵³

For endoscopic endonasal transsphenoidal approaches to the skull base, although anatomic landmarks are typically sufficient to target midline and avoid injury to the carotid arteries and optic apparatus, these landmarks are absent in reoperations. In the one series of augmented neuroendoscopes to date, AR overlays of both endoscope trajectory and neurovascular anatomy were highly valuable in reaching the sellar floor safely in redo procedures, with no additional operative time or hardware setup required. ²⁰

AR in Stereotactic Localization and Functional Neurosurgery

The advantages of AR overlays in providing “x-ray vision” to identify deep intracranial structures, may be extended to ventriculostomy insertion. AR ventriculostomy simulators providing haptic feedback and 3D visualization of intracranial catheter trajectory have been instructional for junior residents in appreciating not only a proper target, the foramen of Monro, but also an appropriate trajectory and the adjacent nuclei to be avoided. AR simulators have also allowed for the real-time quantification of trainee accuracy, revealing trends of improvement with multiple attempts as well as with seniority in training. ^{38 , 39}

AR in Spinal Surgery

Although the literature on spinal applications of AR in open procedures is relatively sparse, there is promise in the ability of AR to provide real-time trajectory guidance for percutaneous instrumentation at or superficial to skin level. ³⁵ Current spinal stereotactic navigation systems are able to guide hardware trajectory relative to bony anatomy, but leave the skin projection of these trajectories at the discretion of the surgeon. The literature on spinal AR has largely focused on percutaneous vertebroplasty/kyphoplasty, although these are readily adaptable to percutaneous pedicle screw placement through identical transpedicular approaches.

C-arm intraoperative fluoroscopy is classically used to guide percutaneous instrumentation. One augmentation technique has involved the placement of a video camera in-line with the x-ray axis, allowing overlay of x-ray and real-world images. In a small cadaver study, although AR decreased radiation exposure compared to a C-arm–only technique, breach rates of pedicle instrumentation were 40%, far greater than the 5% to 15% accepted by most practicing surgeons. ^{28 , 54}

Overlay of 3D-reconstructed MRI or CT data is an alternative technique. ³² In one cadaveric study overlaying intraoperative MRI for vertebroplasty guidance, needle-tip target errors averaged 6.1 mm; however with a mean of six intraoperative MRI scans required per level, this is cumbersome for human clinical application. ⁴⁴ In a study projecting 3D-reconstructed spine CT imaging onto cadaveric torsos for transpedicular approaches, although the AR projection facilitated positioning of the C-arm appropriately for initial targeting, with an entry point error of 4.4 mm, AR alone was unable to accurately allow targeting of the final needle position because of lack of angular information, with a target error of 9.1 mm. ⁴⁵

Although potentially very useful in percutaneous applications, relevant given the emerging indications for minimally invasive procedures, AR for percutaneous spinal surgery remains insufficiently accurate for clinical application. The overlay of multiplanar cross-sectional imaging rather than 3D reconstructions only, similar to the displays of current stereotactic navigation systems, may provide the angular information required for more accurate targeting of implants to their final transpedicular position.

Challenges in AR

Display of 3D virtual objects onto real-world images presents multiple challenges, some specific to certain display techniques. A basic requirement for AR is the accurate registration of real and virtual spaces, which requires knowledge of the pose and optical characteristics of both real and virtual cameras. ⁵⁵ Registration errors in video see-through systems, in which the real world is imaged through a video camera, are constituted by errors in camera calibration, image distortion, and object-to-patient registration. ¹² Optical see-through systems, although eliminating the need for camera calibration, require tracking of head and eye movements for synchronization of real and virtual content from varying perspectives, introducing additional error. ⁵⁵ Eye tracking is unnecessary for image projection techniques and AR windows. Unfortunately, projection of 2D light onto 3D surfaces becomes inaccurate with highly curved surfaces and is less useful once direct line of sight to the patient is unavailable, for instance, with the introduction of a microscope or other equipment adjacent to the operating table. AR windows, although able to display content from any perspective without eye tracking, must be placed over the area to be imaged and thus obstruct the surgical field. However, in endovascular and other procedures where the site of manipulation is distant from the target, AR windows may be appropriate.

Even with geometrically correct positional registration, problems with impaired depth perception may arise. In viewing a native scene, the human eye converges on a particular 3D point and accommodates onto that plane to view the image clearly. These focus cues are combined with numerous monocular and binocular depth cues, including shading, texture, stereopsis, motion parallax, and occlusion, to generate 3D perception in the brain. Discrepancy in accommodation and divergence impairs depth perception, most evident in optical see-through AR in which the focal plane of the virtual image is at the level of the display panel, whereas the eye must accommodate at a longer distance onto the real-world target to see it clearly. ⁵⁶ Accommodation-divergence discrepancy may also lead to visual fatigue, headaches, and diplopia, particularly after prolonged use. ⁵⁷ Injection of 3D images into the oculars of stereomicroscopes somewhat alleviates this, but the focal plane of the virtual image remains incongruent with that of the target. Video see-through displays, either HMDs or external displays, minimize perceptual discrepancies between real and virtual environments by having full control of both. ²⁵ Unfortunately, they are hampered by limited resolution relative to the native eye. Recent work on multifocal plane stereoscopic displays, either via spatial or time-multiplexing, shows promise for the proper display of depth and focus cues in AR. ⁵⁸ Occlusion, the partial blockage of an object’s view by another nearer object, is an important monocular depth cue for the perception of relative depth. ⁵⁹ A well-documented challenge with AR environments is the occlusion of operator’s hands or instruments by superimposed virtual images, leading to misperceptions of relative proximity. Multiple techniques of occlusion handling have been described, for instance, the detection of edges and color-specific surfaces in the camera feed and retention and display of these features over the virtual object. ^{11 , 60}

Temporal synchronization of virtual and real environments is an additional challenge with all AR systems, particularly with rapid perspective changes. This has been most apparent with optical see-through systems, in which even slight delays in remapping of the virtual environment to the real world, following a change in position, are visually jarring to the surgeon. ⁴ By controlling both real and virtual “cameras,” video see-through systems eliminate relative visual lag; however, lag between visual and tactile feedback cannot be avoided by this technique and should ideally be less than 80 ms for the accurate manipulation of delicate structures. ⁶¹

Finally, though image fusion in AR offers the benefit of visualizing multiple 3D datasets congruently, extraneous information may distract surgeons from unpredictable findings in the operative field. Termed “inattentional blindness,” this phenomenon has been extensively studied in the aviation industry, but only recently so in surgery. ⁶² The predominant driver of inattentional blindness appears to be greater cognitive load of the primary task, ^{62 , 63} although augmentation of the visual field has also been implicated in multiple studies. ^{64 - 66} Use of wire-mesh and “inverse-realism” overlay techniques, rather than solid overlay, has been suggested to potentially reduce inattentional blindness. ⁶⁶

Future Directions

Although AR has proven useful in the overlay of multiple 3D imaging modalities onto the working area of interest, much work needs to be done to improve utility, streamline workflow, and minimize surgeon distraction and visual fatigue. Improvements in registration techniques, currently under significant research for nonaugmented navigation technologies, will allow automatic intraoperative reregistration to account for soft-tissue deformation or changes in patient positioning. Improvements in the range of content that can be overlaid—for instance computational fluid dynamics quantifications of blood flow from intraoperative angiograms—will broaden the scope of AR to AVM resections and other procedures where pure anatomic data is not necessarily useful.

Advances in display technology, particularly in augmented optics for microscopes, will streamline AR integration into existing hardware as well as improve visualization and depth perception. The intent of AR is to display contextually relevant content over the area where it is required, theoretically minimizing the current ergonomic hindrances requiring surgeons to look simultaneously at multiple displays, for preoperative planning and intraoperative navigation. Part of the improvements in display ergonomics may come from using compact or readily wearable devices, arenas in which the consumer electronics and gaming sectors are making significant progress. The use of smartphone or tablet cameras and displays, coupled with internal accelerometers for positional tracking, has already shown promise for inexpensive video see-through AR. ⁵⁰ Given their relative affordability, tablet-based AR windows are particularly useful in telesurgical applications for developing nations. ⁶⁷ Consumer AR HMDs, such as Microsoft’s HoloLens, are optimized for consumption of video and gaming content, but certainly may be applicable to surgical environments with the addition of tracking capabilities for head pose and position.

Finally, the interaction between surgeons, augmented displays, and robotically actuated instruments shows tremendous promise for faster and safer surgery in multiple disciplines. Here again, the consumer and gaming sectors have made significant strides in how consumers interaction with virtual content, particularly in gaming environments, where gesture controls must exceed the comfort and accuracy provided by classic handheld controllers. Novel techniques to improve interaction with virtual content, including the ability to freeze and manipulate virtual objects over a live real-world scene, are in development. ⁶⁸ Haptic devices, including styli and gloves, continue to evolve in the consumer arena in an effort to improve tactile feedback; although these may not be practical in a sterile intraoperative setting, they show significant promise for preoperative planning and for surgical education. ⁶⁹

Conclusions

In an era when image guidance is used increasingly across multiple surgical disciplines, AR represents the next frontier where guidance systems are integrated seamlessly into the surgical workflow. We review here the current state of AR using neurosurgery as an example, as one of the surgical disciplines most heavily reliant on advanced imaging. This work represents one of the most comprehensive recent overviews of neurosurgery-specific applications of AR. Challenges to the routine adoption of augmented displays remain, from technical aspects, such as depth misperception and temporal asynchrony, to human factors, such as visual fatigue and inattentive blindness. Nonetheless, rapid advances in display technology and interaction techniques, driven in part by the consumer gaming industry, promise a burgeoning role for AR in the modern neurosurgical operating room.