Sparse summary generation

The state-of-the-art summary generators build on powerful language models, such as BERT, which achieves impressive performance. However, most models employ softmax transformation in their output layer, leading to dense alignments and strictly positive output probabilities. This density is wasteful since it assigns probability mass to many implausible outputs. In this paper, we propose a sparse summary generation model with a new gp-entmax transformation, which includes 1.5-entmax and gradient penalty. The 1.5-entmax has the great effect of filtering noise, retaining important information and improving model performance. Experimental results show that the generated summary has improved in both ROUGE and BLEU metrics, and when tested on the CSL summarization dataset, our method outperforms the softmax model by more than 3 ROUGE-L points. For the purpose of measuring the level of important information in model-generated summaries, we propose a new metric called M2I. Simulation tests on human evaluation showed that the summary generated by the sparse model is more fluent and closer to the text’s main idea.