In this paper, we propose a method of AUV localization and mapping using visual measurement of underwater structures. Since the inertial navigation system (INS) of AUV suffers from drift, observing fixed objects can enhance the localization performance. Depth map estimation of underwater structures and tracking of an AUV are complementarily handled in a framework of pose graph optimization. Estimation of absolute scale is resolved by exploiting incremental measurements of inertial sensors. The proposed method is validated by experiments performed in a structured basin environment.