Cycle Ynet - Semi-supervised Tracking of 3D Anatomical Landmarks.

Real-time tracking of anatomical landmarks in 3D medical images is of great importance, ranging from live quantification to optimal visualization.Existing deep network models have shown promising performance but typically require a large amount of annotated data for training. However, obtaining accurate and consistent annotations on sequences of 3D medical images can be very challenging even for skilled clinicians. In this paper, we propose a semi-supervised spatial-temporal modeling framework for real-time anatomical landmark tracking in 3D transesophageal echocardiography (TEE) images, which requires annotations on only a small fraction of frames in a sequence. Specifically, a spatial discriminative feature encoder is first trained via deep Q-learning on static images across all patients. Then we introduce a Cycle Ynet framework that integrates the encoded spatial features and learns temporal landmark correspondence over a sequence using a generative model by enforcing both cycle-consistency and accurate prediction on a couple of annotated frames. We validate the proposed model using 738 TEE sequences with around 15,000 frames and demonstrate that by combining a discriminative feature extractor with a generative tracking model, we could achieve superior performance using a small number of annotated data compared to state-of-the-art methods.