Augmented reality-based nasal endoscope video reconstruction and registration

Neuroendoscopic surgery is a minimally invasive surgical technique commonly used to remove a tumor through the patient's mouth or nose, which requires the surgeon to avoid important neural and vascular structures. Augmented reality-based surgical navigation can provide surgeons with more information and visual aids, clarifying the structure and details of the surgical area. It requires real-time registration of intraoperative endoscopic video with pre-operative CT models. However, 3D reconstruction and camera tracking from nasal endoscopic video are challenging due to the narrow nasal cavity and the lack of texture. Besides, the nasal endoscopic datasets that can be used for deep learning-based depth estimation are scarce and hard to elaborate. To this end, a style-augmented module (SAM) is proposed in this study to minimize discrepancies between different endoscopic datasets. An unsupervised approach was trained to generate depth maps and camera motion paths, which were used to reconstruct the nasal scene's 3D model. The obtained reconstruction was aligned with a pre-operative CT model for augmented reality-based surgical navigation. The results showed that the proposed SAM improved the generalization of the depth estimation model on nasal endoscopy data. The proposed approach to combining augmented reality technology with surgical navigation is considered instrumental in furnishing surgeons with richer information on the nasal endoscopic surgical scene.