A Pipelined Point Cloud Based Neural Network Processor for 3-D Vision With Large-Scale Max Pooling Layer Prediction

Abstract

The point cloud data provides useful geometric information to 3-D intelligent systems such as autonomous driving, 3-D reconstruction, and hand pose estimation (HPE). Many mobile devices have implemented the 3-D intelligent system with their limited hardware resources. However, previous processors were not designed for accelerating the point cloud based neural network (PNN) which consists of sampling-grouping layers (SGLs) and convolution layers (CLs). In this article, a pipelined PNN processor is proposed for low latency PNN-based 3-D intelligent systems in mobile devices. The processor adopts the pipelined heterogeneous architecture to accelerate both SGLs and CLs in PNNs. The window-based sampling-grouping (WSG) algorithm boosts up the throughput x2.34 higher in SGLs by directly sampling and grouping the 3-D point cloud data from the depth image. Furthermore, the max pooling (MP) prediction core (MPPC) predicts the large-scale (64- and 128-to-1) MP layers, which increases the throughput by x1.31 higher. Besides, the MP prediction on the tiled data can hide the latency of the MPPC and solve the bank conflict problem on the in-out memories in the convolution core (CC). As a result, the processor successfully demonstrates the PNN-based HPE system resulting in 4.45 ms of the processing time with 8.24 mm of HPE error and 266 mW of power consumption.