Multi-modal Broad Learning System for Medical Image and Text-based Classification

Automatic classification of medical images plays an essential role in computer-aided diagnosis. However, the medical images arise from the small number of available data and the improvement of existing data-enhancement methods are limited. In order to fulfil this demand, a Multi-Modal Broad Learning System (M-2-BLS) is proposed, which has two subnetworks for simultaneous learning of both medical images and the corresponding radiology reports. M-2-BLS provides two advantages: i) our M-2-BLS has closed-form solution and avoids iterative training, once the image feature is available; ii) benefit from the simultaneous learning of both image and text data, our M-2-BLS achieves high accuracy for medical classification. Experimental results on the publicly available datasets IU XRAY and PEIR GROSS 895 show that our M-2-BLS highly improves the classification performance, compared to SOTA deep models that learn single-type of data information only.