Unsupervised disease tags for automatic radiology report generation

Recently, the automatic radiology report generation system that aims to accurately describe medical observations for given X-ray images has gained increasing attention. Compared to image captioning, radiology report generation poses greater challenges for data-driven neural-based methods due to the significant data biases. Faced with these challenges, conventional approaches typically utilize a supervised multi-label classification process to accurately recognize disease tags, thereby facilitating the detection of abnormal findings. However, these methods depend heavily on paired image-tag datasets that are costly to acquire. To tackle this challenge, we propose the unsupervised disease tags (UDT) model for automatic radiology report generation. In detail, our UDT model introduces three major modules: the General Disease Tags Cluster (GDTC) module for the clustering of general disease tags, which is a more generalizable way to learn and store medical knowledge; the Parallel Dual Attention (PDA) module for generating the attended visual and disease tag features in a parallel manner; and the Parallel Visual-Tag Hybrid (PVTH) module for enhancing the final feature representation by the joint utilization of attended visual and disease tag features. Experiments on two benchmark datasets show that our proposal achieves state-of-the-art (SOTA) performance on various natural language generation evaluation (NLG) metrics. Further clinical efficacy (CE) metrics also validate that our proposal can provide more accurate abnormal findings. The source code will be available at https://github.com/SKD-HPC/UDT.