It is well known that set of unstable periodic orbits (UPOs) can be thought of as the skeleton for the dynamics. However, detecting UPOs of nonlinear map is one of the most challenging problems of nonlinear science in both numerical computations and experimental measures. In this paper, a new method is proposed to detect the UPOs in a non-Lyapunov way. Firstly three special techniques are added to quantum-behaved particle swarm optimization (QPSO), a novel mbest particle, contracting the searching space self-adaptively and boundaries restriction (NCB), then the new method NCB-QPSO is proposed. It can maintain an effective search mechanism with fine equilibrium between exploitation and exploration. Secondly, the problems of detecting the UPOs are converted into a non-negative functions' minimization through a proper translation in a non-Lyapunov way. Thirdly the simulations to 6 benchmark optimization problems and different high order UPOs of 5 classic nonlinear maps are done by the proposed method. And the results show that NCB-QPSO is a successful method in detecting the UPOs, and it has the advantages of fast convergence, high precision and robustness. (c) 2009 Elsevier Ltd. All rights reserved.