Prompt-Based Grouping Transformer for Nucleus Detection and Classification

Automatic nuclei detection and classification can produce effective information for disease diagnosis. Most existing methods classify nuclei independently or do not make full use of the semantic similarity betweennucleiandtheir grouping features. Inthispaper, we propose anovel end-to-end nuclei detection and classification framework based on a grouping transformer-based classifier. Thenuclei classifier learns and updates the representations of nuclei groups and categories via hierarchically grouping the nucleus embeddings. Then the cell types are predicted with the pairwise correlations between categoricalembeddings and nucleus features. For the efficiency of the fully transformer-based framework, we take the nucleus group embeddings as the input prompts of backbone, which helps harvest grouping guided features by tuning only the prompts instead of the whole backbone. Experimental results show that the proposed method significantly outperforms the existingmodels on three datasets.