A deep learning framework for 2D-3D image registration and volumetric imaging in the presence of biologically-driven motion

Abstract Biomedical imaging often relies on 2D images to capture information about moving 3D objects. However, without sufficient prior knowledge, the problem of mapping from 2D images to 3D motion is computationally intractable. In this paper we introduce Voxelmap, a deep learning framework that achieves 2D-3D image registration and volumetric imaging in real-time by imposing implicit constraints on biologically-driven motion. Here we demonstrate the use of this framework in image-guided radiotherapy with data from two lung cancer patients. By efficiently estimating biologically-driven 3D motion from 2D images, this framework could also find application in contexts such as fetal imaging and functional neuroimaging.