Automation of population-based recurrence map for PSMA-PET prostate cancer patients after prostatectomy

Purpose Investigate and evaluate the accuracy of deep learning (DL)-based segmentation and deformable image registration (DIR) for the automatization of recurrence risk map atlas definition. Materials and methods Twelve patients with visible recurrence on 18F-DCFPyL PET/CT after prostatectomy were retrospectively analyzed. The bladder, rectum, iliac arteries and veins, and recurrence sites were manually delineated. A previously trained DL model for female pelvic anatomy was re-optimized for male to automatically segment the anatomical regions of interest (ROI). Inter-patient registration was investigated using 4 registration methods: rigid, B-Spline Plastimatch, intensity DIR, and a hybrid intensity-based DIR with varying number of controlling ROI. Performance of the methods were reported using contour-based metrics, determinant of the Jacobian, contour variability in term of volume and position, and probability of overlap with the template organs. Results Transfer learning of the DL model provided greater accuracy for the bladder and rectum than for new structures such as iliac arteries and veins with average Dice similarity coefficient ranges of 0.82-0.96 and 0.63-0.77, respectively. Compared to intensity only DIR, hybrid intensity-based DIR with controlling ROI provided better contour-based metrics, determinant of Jacobian, and less incidence of overlap between recurrence sites and template organs. Centroid position variability between the registration approaches were reported with average range of 1.6-11.3 mm and up to 5.7-30 mm. Conclusion DL and hybrid DIR models can be used to automatize inter-patient registration in the definition of population-based recurrence risk map. DIR uncertainties in the propagation of the recurrence between patients need to be carefully verified before being used in population-based model.