Context-aware image captioning using cosine transformer attention and optimal vision transformer with improved walrus optimization algorithm

Abstract

Abstract Automatic image captioning aims to acquire the learning of the correlation between the visual and the linguistic cognitive domain through the generation of descriptive captions with respect to input images. But it is affected by the semantic distance between image features and text, and problems in modeling fine-grained object interactions and contexts. We present a new encoder-decoder architecture, that utilizes optimal vision transformer + attention to the transformer by using cosine distance in order to get more context by providing better mapping. It relies on Inception-V3 to acquire fine-grained visual features and skip-gram features of the textual semantics. Another algorithm named improved walrus optimization dynamically updates the key hyperparameters to enhance convergence and performance. Experiments on the MS-COCO and Flickr8k datasets show that it is better than prior methods with BLEU-4 scores of 0.8864 and 0.8920, and higher performance gains in ROUGE, METEOR and CIDEr scores. In conclusion, the method outputs with better contextual details and flow of natural language, semantic fidelity. It is optimal in practice due to its cosine attention and adaptive tuning.