Besra: Self-correction for hallucination mitigation in large vision-language models

Abstract

Large Vision-Language Models (LVLMs) have revolutionized the field of computer vision by unifying various computer vision tasks through their ability to comprehend visual information. However, they often suffer from hallucination, generating inconsistent descriptions not aligned with input images. This paper introduces Besra, a Large Vision-Language Model designed to address hallucination by incorporating a self-correction task. Besra leverages its iterative refinement capability to enhance generated sentences' consistency with provided images. The model iteratively refines descriptions by refeeding them alongside corresponding images, facilitating a detailed examination of specific image regions. Besra-Self-Correction-30K, a proposed dataset, trains Besra's self-correction ability by inducing corrections based on predictions from a baseline LVLM. The approach aims to mitigate hallucination, enabling Besra to generate more accurate and contextually relevant descriptions through active image scrutiny. We evaluate Besra on POPE and MME benchmark and prove that a self-correction task is helpful for hallucination mitigation.