Learning to refine source representations for neural machine translation

Machine translation is one of the most classic application technologies in artificial intelligence and natural language processing. Neural machine translation models generally adopt an encoder–decoder architecture for modeling the entire translation process. However, without considering target context ( e.g., decoding state) to guide the encoding, encoded source representations struggle to put great emphasis on important information for predicting some target word, yielding the weakness in generating more discriminative attentive representations across different decoding steps. Towards tackling this issue, we propose a novel encoder–refiner–decoder framework, which dynamically refines the source representations based on the generated target-side information at each decoding step. Since the refining operations are time-consuming, we propose a policy network to decide when to refine at specific decoding steps. We solve such a problem using the Gumbel-Softmax reparameterization, which makes our network differentiable and trainable through standard stochastic gradient methods. Experimental results on both Chinese–English and English–German translation tasks show that the proposed approach significantly and consistently improves translation performance over the standard encoder–decoder framework. Furthermore, when refining strategy is applied, experimental results still show a reasonable improvement over the baseline with much decrease in decoding speed.