In the previous study, a paired structured light system which incorporates lasers, cameras, and screens was proposed, and experimental tests validated the potential of a displacement measurement system for large structures. However, the estimation of relative translational and rotational displacements between two sides was based on an assumption that there is zero initial displacement and that three laser beams are always on the screens. In this paper, a calibration method is proposed to offset the initial displacement using the first captured image. The calibration matrix derived from the initial offset is used for subsequent displacement estimation. A newly designed 2-DOF manipulator for each side is visually controlled to prevent the laser beams from leaving the screen. As the manipulator actively controls the laser beams to target the center of the screen, it contains all three laser points within the bounds of the screen. To verify the feasibility of the proposed system, various simulations and experimental tests were performed. The results show that the proposed visually servoed paired structured light system solves the main problem with the former system and that it can be utilized to enlarge the estimation range of the displacement.