Enhance understanding and reasoning ability for image captioning

Image captioning aims to generate a grammatically correct and semantically accurate natural language description of a given image. To better capture the complex information contained in an image and expand the relevant external knowledge outside the image to generate a better image caption, this paper proposes an end-to-end image captioning framework named Enhance Understanding and Reasoning Ability for Image Captioning (EURAIC) based on the Transformer model. EURAIC provides an enhanced visual understanding ability and caption reasoning ability to improve image captioning performance. To achieve this goal, we use the semantic features of the core objects detected from the image to guide the visual features, which incorporate information on the spatial position relationships between the objects. Then, we introduce an external knowledge network to obtain information other than the inherent content of the image. In this way, a high-quality image caption sentence can be generated for the given image. Experiments on the MSCOCO dataset prove that our method is superior to the baseline model and comparable to other state-of-the-art methods.