For creating a precise visual map by autonomous ship-hull inspection using an unmanned underwater vehicle, it is a crucial capability for the vehicle (or camera) to maintain a pose relative to the hull surface. In this study, a relative pose estimation algorithm is introduced using a stereo vision system. The proposed approach utilizes 3D point cloud data that can be generated by a sparse feature matching technique between a pair of stereo images. The relative pose information can be obtained by applying a surface normal estimation algorithm for the 3D points. Experimental results using underwater images is shown to verify the practical feasibility of the proposed approach.