The laser interferometer system such as HP5529A is one of the most powerful equipment fur measurement of the straightness error in precision stages. The straightness measurement system, HP5529A is composed of a Wollaston prism and a reflector. In this system, the straightness error is defined as relative lateral motion change between the prism and the reflector and computed from optical path difference of two polarized laser beams between these optics. However, rotating motion of the prism or the reflector used as a moving optic causes unwanted straightness error. In this paper, a compensation method is proposed for removing the unwanted straightness error generated by rotating the moving optic and an experiment is carried out for theoretical verification. The result shows that the unwanted straightness error becomes very large when the reflector is used as the moving optic and the distance between the reflector and the prism is far. Therefore, the prism must be generally used as the moving optic instead of the reflector so as to reduce the measurement error. Nevertheless, the measurement error must be compensated because its not a negligible error if a rotating angle of the prism is large. In case the reflector must be used as the moving optic, which is unavoidable when the squareness error is measured between two axes, this compensation method can be applied and produces a better result.