Double Attention Mechanism for Sentence Embedding.

This paper proposes a new model for sentence embedding, a very important topic in natural language processing, using a double attention mechanism to combine of a recurrent neural network (RNN) and a convolutional neural network (CNN). First, the proposed model uses a bidirectional Long Short Term Memory Recurrent Neural Network (RNN-LSTM) with a self-attention mechanism to compute a first representation of the sentence called primitive representation. Then the primitive representation of the sentence is used along with a convolutional neural network with a pooling based attention mechanism to compute a set of attention weights used during the pooling step. The final sentence representation is obtained after concatenation of the output of the CNN neural network with the primitive sentence representation. The double attention mechanism helps the proposed model to retain more information contained in the sentence and then to be able to generate a more representative feature vector for the sentence. The model can be trained end-to-end with limited hyper-parameters. We evaluate our model on three different benchmarks dataset for the sentence classification task and compare that with the state-of-art method. Experimental results show that the proposed model yields a significant performance gain compared to other sentence embedding methods in all the three dataset.