Multiple-input multiple-output (MIMO) systems have been shown to achieve significant increases in efficiency using arrays of transmit and receive antennas with spatial processing. Among various MIMO systems, we focus on low complexity tree searching algorithm in a vertical Bell Labs layered space-time (V-BLAST) architecture for single-user MIMO and vector precoding for multi-user MIMO systems. In both systems, signal processing technique is key role for alleviating interference among users or antennas, respectively. Many efficient algorithms such as sphere decoding, QRD-M, and QRD-Stack are proposed for reducing complexity than that of maximum-likelihood (ML) decoding. In this thesis, we propose a reduced candidate set election algorithm by channel condition which is represented by condition number and the smallest singular value for tree searching algorithms. The proposed algorithm offers substantial computational savings over conventional ones, while maintaining performance arbitrarily close to ML. Firstly, We elaborate the impact of the condition number and the smallest singular value in the case for performance of tree searching algorithms. From this result, we recognize that the channel condition is closely related to the complexity and performance. Also, the determination of the number of candidates is crucial to achieve a balance between bit error rate (BER) performance and computational complexity. Motivated by these results, we propose small candidate set for well-conditioned channels and large candidate set for ill-conditioned channels. In V-BLAST systems, moreover, approximation of the condition number is suggested to reduce additional complexity required for finding the condition number. In vector precoding systems, we suggest sphere constraint to restrict the candidate set to reduce complexity in stack-based iterative precoding(SBIP) and QRD-M based iterative precoding (QRMIP) algorithms. Simulation results demonstrate the proposed approaches prov...