(An) intelligent vision algorithm and a real-time processor for advanced driver assistance system

Abstract

The advanced driver assistance system (ADAS) has been actively researched to enable adaptive cruise control and collision avoidance, and it is strongly dependent upon the robust image recognition technology such as lane detection, vehicle detection, pedestrian detection, traffic sign recognition, etc. However, the conventional ADAS cannot realize more advanced functions, such as collision evasion in real environments, due to the absence of intelligent decision making algorithms such as vehicle/pedestrian behavior analysis. Moreover, accurate distance estimation is essential in ADAS applications and semi-global matching (SGM) is most widely adopted for its high accuracy, but its System-on-Chip (SoC) implementation is difficult due to the massive external memory bandwidth. Most algorithms in automotive applications are accelerated by GPUs where its power consumption exceeds the power requirement for practical usage. In this paper, an energy-efficient real-time ADAS SoC with deep risk prediction algorithm, which predicts risky objects prior to collision by behavior prediction and analysis, and hardware implementation of SGM is proposed to provide practical application to the intelligent ADAS. The proposed SoC is capable of SGM, object detection, object tracking, ego-motion compensation, and deep risk prediction with only 330mW of power consumption in average. Moreover, it has dual-mode operations of high-performance operation for intelligent ADAS with real-time SGM in driving mode (d-mode) and ultra-low-power operation for black box system in parked mode (p-mode). The SoC features: 1) task-level pipelined SGM processor to reduce external memory bandwidth by 85.8%

2) region-of-interest generation processor to reduce 86.2% of computation

3) mixed-mode deep risk prediction engine for dual-mode intelligence

4) dynamic voltage and frequency scaling control to save 36.2% of power in d-mode. For validation, the risky urban scene stereo (RUSS) database including 50 stereo video sequences captured under various risky road situations is created and tested. The proposed system is tested under various databases including the RUSS, KITTI, and Daimler-Stereo databases. The proposed ADAS processor achieves 862 GOPS/W energy efficiency and 31.4 GOPS/mm2 area efficiency, which are 1.53x and 1.75x improvements than the state-of-the-art. Also, the entire system can maximally achieve 30 frames/s throughput with 720p stereo images and achieves 98.1% of prediction accuracy.