An apparatus and method for self-calibrating mismatching orthogonal signals and non-linearity of a circuit occurring in a mobile transceiver. For this, in a mobile terminal, a transmitter is used as a signal generator and a receiver is used as a response characteristic measurer. A base band processor calibrates mismatching signals and non-linearity in a reception side and a transmission side, based on a test signal received via the transmitter and a test signal received via the receiver.