(A) study on adaptive scheduling scheme for accelerating visual explanation generation with hierarchical clustered cam method and its application for change detection

Abstract

The large-scale data accumulation and the rapid growth of computing power have energized intelligent systems based on deep neural networks (DNNs). However, DNNs do not transparently show the reason for the decision, leaving the question of whether to trust the recommendations of deep neural networks and justify their use for humans who utilize them unanswered. Furthermore, DNNs have become essential components for intelligent systems in power budget- and computing capability-constrained environments, such as embedded systems. However, DNNs require significant amounts of computing resources and power, hindering advances in the intelligent system for embedded computing environments.

Visual explanation methods generate explanation by highlighting the input characteristics that strongly influence the output of DNNs, in the form of a saliency map or attention map. Generated visual explanation builds reliability by providing the basis for the DNN’s decision and judgment to practitioners. In this dissertation, we classify existing visual description methods into two folds: activation map-based methods with shorter computation time using activation maps and gradients, and region-based methods with mechanisms to quantify the impact of individual masks on the prediction of DNNs. They have problems such as gradient noise, false confidence, and heavy computing load. Moreover, conventional visual explanation methods consider only the final convolutional layer as the target layer, so there are limitations on achievable explainability due to the low resolution of the final convolutional layer output and the nature of missing object details. Besides, it is important to effectively analyze video frames to achieve maximum potential in onboard systems with the potential to process video inputs in real-time to provide a wider range of new applications and services. Existing deep learning-based analysis methods use video analysis frameworks such as YOLO, but their application to onboard environments with power and computing resource constraints limits achievable detection performance (e.g., mAP).

In chapter 3 of this dissertation, we present Collection-CAM, which leverages multi-level feature maps to improve the explainability of DNNs while minimizing additional computation overhead. The process of creating a visual explanation of the Collection-CAM is summarized as follows. First, the Collection-CAM searches for the most suitable form of partitions through bottom-up clustering and clustering verification processes. The Collection-CAM then overcomes false positivity when used without distinction by applying different preprocessing procedures for shallow and final feature maps. Finally, the Collection-CAM completes the attention map generation process by combining collection-specific masks with the use of contributions to confidence scores. The proposed visual explanation technique shows a dramatically lower computational overhead and better explainability compared to conventional region-based visual explanation methods.

In addition, chapter 4 presents scheduling techniques for effective object change detection in onboard environments with power budget constraints. To this end, we first used an object change detector using Vision GNN as a backbone. Vision GNN is a state-of-the-art graph convolutional network designed for vision tasks. Furthermore, we derive optimization problems for maximum performance within the power budget when running Vision GNN-based object change detectors in heterogeneous accelerator onboard environments. Furthermore, we derive optimization problems for maximum performance within the power budget when running Vision GNN-based object change detectors in heterogeneous accelerator onboard environments. We propose an adaptive scheduling heuristic based on the bottleneck-free condition to resolve the optimization problem. We validate the performance of the proposed method with a custom onboard environment equipped with NVIDIA Jetson Nano GPU and Xilinx XCZU7EV FPGA. Experimental results show that the proposed scheduling method achieves a detection performance improvement of more than 5\% at a much lower computational cost compared to YOLO-v5l.

In conclusion, in this dissertation, we propose adaptive scheduling schemes for accelerating visual explanation generation using hierarchical clusters of feature maps and detecting object changes in onboard environments. We also theoretically validate the proposed scheduling scheme and demonstrate superiority through experiments in general computing and onboard computing environments.