Toward Backdoor Attacks for Image Captioning Model in Deep Neural Networks

Abstract

Deep neural networks perform well in image recognition, speech recognition, and text recognition fields. The image caption model provides captions for images by generating text after image recognition. After extracting features from the original image, this model generates a representation vector and provides captions for the image by generating text through a recursive neural network. However, this image caption model has weaknesses in the backdoor sample. In this paper, we propose a method for generating backdoor samples for image caption models. By adding a specific trigger to the original sample, this proposed method creates a backdoor sample that is misrecognized as a target class by the target model. The MS-COCO dataset was used as the experimental dataset, and Tensorflow was used as the machine learning library. When the trigger size of the backdoor sample is 4%, experimental results show that the average attack success rate of the backdoor sample is 96.67%, and the average error rate of the original sample is 9.65%.