FDT − Dr2T: a unified Dense Radiology Report Generation Transformer framework for X-ray images

Medical Image Captioning (MIC), is a developing area of artificial intelligence that combines two main research areas, computer vision and natural language processing. In order to support clinical workflows and decision-making, MIC is used in a variety of applications pertaining to diagnosis, therapy, report production, and computer-aided diagnosis. The generation of long and coherent reports highlighting correct abnormalities is a challenging task. Therefore, in this direction, this paper presents an efficient FDT-Dr^2T framework for the generation of coherent radiology reports with efficient exploitation of medical content. The proposed framework leverages the fusion of texture features and deep features in the first stage by incorporating ISCM-LBP + PCA-HOG feature extraction algorithm and Convolutional Triple Attention-based Efficient XceptionNet ( C-TaXNet ). Further, fused features from the FDT module are utilized by the Dense Radiology Report Generation Transformer ( Dr^2T ) model with modified multi-head attention generating dense radiology reports by highlighting specific crucial abnormalities. To evaluate the performance of the proposed FDT-Dr^2T extensive experiments are conducted on publicly available IU Chest X-ray dataset and the best performance of the work is observed as 0.531 BLEU@1, 0.398 BLEU@2, 0.322 BLEU@3, 0.251 BLEU@4, 0.384 CIDEr, 0.506 ROUGE-L, 0.277 METEOR. An ablation study is carried out to support the experiments. Overall, the results obtained demonstrate the efficiency and efficacy of the proposed framework.