LLM-based framework for generating barrier-free audio descriptions

Abstract

We address a framework for generating barrier-free audio descriptions (AD), specifically designed for visually impaired people. Barrier-free audio description is a narration that delivers key visual information to visually impaired people about the contents of video media such as movies and dramas. Given the increasing importance of visual media, it holds significant value in preventing social exclusion of visually impaired people. AD generation is more challenging than general video captioning as it requires reflecting the overall context of the movie and the names of characters for each video description. We leverage the rich contextual information from movie scripts and the capabilities of multi-modality LLM to generate audio descriptions. It involves identifying parts of the movie script relevant to the videos to be described, acquiring information about the contextual narrative of the movie and names of characters. Then, we incorporate this information into video descriptions using the multi-modality LLM. Our framework produces higher quality audio descriptions than previous works, without the need for additional training.