In this paper, we present an ultra-low-power analog-digital mixed-mode face recognition processor for user authentication in mobile devices. Unlike conventional face recognition processor architecture composed of an image sensor, high-resolution analog-to-digital converters (ADCs), and a digital signal processor, the proposed mixed-mode architecture removes the power-hungry ADCs and introduces an analog signal processor for processing the first layer of convolutional neural networks used by both face detection and face recognition as well as for performing ternary quantization. At the circuit level, we propose the reconfigurable readout circuit and exposure time-division scheme to integrate image sensor and analog signal processor without losing input data under various illumination conditions. We also propose the error-tolerant weighted-sum unit for analog convolution processing with only 15.09 uW power consumption. As a result, post-layout simulation results in 65 nm process demonstrate that the proposed mixed-mode face recognition processor has the total 0.205 mW power consumption while dissipating only 64 uW for always-on operation, which are 66.9% and 33.9% less than the state-of-the-art design.