We propose a drone classification method based on convolutional neural network (CNN) and micro-Doppler signature (MDS). The MDS only presents Doppler information in time domain. The frequency domain representation of MDS is called as cadence-velocity diagram (CVD). To analyze the Doppler information of drone in time and frequency domain, we propose a new image by merging MDS and CVD, as merged Doppler image. GoogLeNet, a CNN structure, is utilized for the proposed image data set because of its high performance and optimized computing resources. The image data set is generated by the returned Ku-band frequency modulation continuous wave radar signal. Proposed approach is tested and verified in two different environments, anechoic chamber and outdoor. First, we tested our approach with different numbers of operating motor and aspect angle of a drone. The proposed method improved the accuracy from 89.3% to 94.7%. Second, two types of drone at the 50 and 100 m height are classified and showed 100% accuracy due to distinct difference in the result images.