An energy-efficient memory-centric convolutional neural network (CNN) processor architecture is proposed for smart devices such as wearable devices or the internet of things (IoT) devices. To achieve energy-efficient processing, it has two key features: First, 1-D shift convolution PEs with fully distributed memory architecture achieve 1.5TOPS/W energy efficiency, and it can be boosted up equivalent 3.1TOPS/W energy efficiency with separable filter approximation and transpose-read SRAM. Compared with conventional architecture, even though it has massively parallel 1024 MAC units, it achieves high energy efficiency by scaling down the voltage to 0.46V due to its fully local routed design. Second, fully configurable 2-D mesh core-to-core interconnection support the various size of input features to maximize utilization. The proposed architecture is evaluated 16mm(2) chip, which is fabricated with a 65nm CMOS process. As a result, it performs real-time face recognition with the only 68.9mW at 40MHz and 0.6V.