MSTS-Net: malignancy evolution prediction of pulmonary nodules from longitudinal CT images via multi-task spatial-temporal self-attention network

Purpose Longitudinal CT images contain the law of lesion growth and evolution over time. Therefore, our purpose is to explore the growth and evolution law of pulmonary lesions in the time dimension to improve the performance of predicting the malignant evolution of pulmonary nodules. Methods In this paper, we propose a Multi-task Spatial-Temporal Self-attention network (MSTS-Net) to predict the malignancy growth trend of pulmonary nodules from different periods. More specifically, the model achieves lesion segmentation task and lesion prediction task by sharing the same encoder. Segmentation task boosts the performance of the prediction task. In addition, a Static Context Spatial Self-attention Module and a Dynamic Adaptive Temporal Self-Attention Module are introduced to capture both static spatial coherence patterns between consecutive slices of lesions in the same period and temporal dynamics across different time points. Results We repeatedly evaluated the proposed method on the National Lung Screening Trial dataset and the Shanxi Cancer Hospital dataset. The final experimental results show that our MSTS-Net has an area under the ROC curve score of 0.919. Conclusion In the computer-aided prediction of the malignant evolution of pulmonary nodules, combining the characteristics of the temporal dimension of pulmonary nodules with CT data can effectively improve the accuracy of prediction. The MSTS-Net we developed has high predictive value and broad prospects for clinical application.