This paper presents a nonlinear sliding mode control (SMC) based autopilot system design for precise missile operation, saturation function is introduced to avoid chattering problem. The main contribution of the study is in order to enhance the control performance of the SMC autopilot system, the effect of boundary layer thickness on the missile system responses is studied, the different response results reveal that the smaller the boundary layer thickness is, the better performance results will be obtained. Based on the conclusion, a more reasonable SMC missile autopilot system is obtained, which also can be used to deal with the non-minimum phase that exists in the tail-controlled missile system, and the simulation result of different responses also has comparative performance with slightly error existence.