Semantic scene change detection for intelligent visual surveillance system

Abstract

Driven by continued advancements in deep learning and robotics, the demand for intelligent visual surveillance and investigation systems is increasing. An intelligent visual surveillance system is a system that performs surveillance and investigation tasks in a desired environment based on various computer vision technologies. An intelligent surveillance system reduces human errors, such as missing or misjudging important events related to surveillance work, by automating surveillance tasks. It reduces labor costs by fully automating relatively simple tasks during surveillance work. It allows you to reduce the cost of work. Additionally, by linking an intelligent surveillance system with robotics technology, it is possible to prevent industrial casualties by using robots to perform investigation tasks in hazardous industrial environments. The core technology in this intelligent visual surveillance system is Scene Change Detection (SCD), a technology that detects changes between images. SCD is a process of comparing two photos taken in the same place at different times to segment the changed area, and there is a random time interval between the photos taken. The longer the time interval, the greater the illuminance or environmental changes in the same place, such as night and day or summer and winter, so it is important to develop an SCD technique that is robust to such environmental changes. In addition, when performing SCD using a robot, the difference in shooting angle between photos can be large, so it is important to develop technology to compensate for this. To this end, studies based on data augmentation, introduction of correlation layers, and supervised learning using optical flow models have been conducted, showing good results in outdoor and indoor environments. However, existing models require datasets containing expensive pixel-level labels for learning, have limitations in that only binary classification of changes is possible, and do not consider the image matching step required before SCD, making them impractical. There is. To improve the practicality of SCD, this paper uses an unsupervised learning method based on the feature-metric alignment of two input images, a semantic change detection technique using a multimodal model such as CLIP, and images extracted through the SCD model. We propose a technique to perform image matching based on features. The proposed techniques reduce the cost of SCD model learning by utilizing the features of unsupervised learning and pre-trained large models and increase the practicality of SCD technology by integrating image matching and SCD. Comparative experiments with existing techniques were performed on various indoor and outdoor data to verify the performance of the proposed technique, and the feasibility of the proposed technique was verified through ablation studies.