Deep learning (DL) classifiers have significantly outperformed traditional likelihood-based or feature-based classifiers for signal modulation recognition in non-cooperative environments. However, despite these recent improvements, the conventional DL classifiers still have an unintended problem in handling the received signal in which the in-phase and quadrature components are separated. Even though the two components seem to be individually uncorrelated to each other, they are definitely the theoretical real and imaginary parts of a signal sample in the complex domain. Thus, it may be helpful for a classifier to regard and treat the modulated signal as a complex data array representing beneficial mutual information between the two real data arrays. In this paper, we propose two types of fully complex convolutional neural network (CNN) and residual neural network (ResNet) classifiers that deal with the complex data instead of the two separated data. First, we organize and define the core complex operations for implementing the complex DL classifiers. Next, the architectures of the proposed classifiers are realized by applying the structural optimizations and the regularization technique. Then, the various aspects of the classifiers' performance are analyzed and explained for providing comprehensive and deep understandings: (a) the effectiveness of the complex signal handling, (b) the in-depth evaluation of classification accuracy, (c) the impact of the data size on performance, and (d) the computational complexity. The experimental results show that the proposed classifiers can provide the faster learning speed, the higher optimization, and the better generalization with sacrificing acceptable learning and classification costs. Especially, our approaches remarkably improve the performance on the mutually correlated modulation types (i.e., the phase-related modulations: PSK, APSK, and QAM) even in the less-scale datasets.