Explainable image caption generator using attention and bayesian inference

Abstract

In this thesis, we propose an explainable image caption generation model by applying an attention mechanism and Bayesian inference. Image captioning task is the process of generating textual descriptions of a given image. In traditional studies, such tasks were addressed by directly combining computer vision techniques and natural language processing. Because deep learning has shown great performance in various applications, some recent studies have applied deep learning to image captioning leading to performance improvements compared to traditional approaches. However, these image captioning models cannot reflect the important objects in the given image when generating captions, because these models simply learn the direct correlation between the given image and a corresponding true caption sentence. Moreover, these models cannot explain why specific words are selected in the generated caption because of the lack of deep learning interpretability. To overcome these limitations, we propose a novel image captioning model, Explainable Image Caption Generator, which generates a caption for a given image by reflecting specific objects in the image, and by providing evidence explaining why specific words are generated. Our model is composed of two parts: the generation part, which generates the caption for a given image

and the explanation part, which generates the image--sentence relevance loss, which guides the generation part to capture the important objects in the image and to reflect them during training. Furthermore, the generation part provides a correlation matrix between extracted regions and generated words that can be used for visualizing the evidence of the words in the generated caption. We evaluate our model on three benchmark datasets: MSCOCO, Flickr8K, and Flickr30K. Qualitative results are provided for presenting the effectiveness of the explanation. Quantitative results for generated caption show that the proposed model outperforms traditional approaches both for quantitative and qualitative results.