Accurate localization and mapping in a large-scale environment is an essential system of an autonomous vehicle. The difficulty of the previous LiDAR or LiDAR-inertial simultaneous localization and mapping (SLAM) methods is correcting long-term drift error in a large-scale environment. This paper proposes a novel approach of a large-scale, graph-based SLAM with traffic sign data involved in a high-definition (HD) map. The graph of the system is structured with the inertial measurement unit (IMU) factor, LiDAR-inertial odometry factor, map-matching factor, and loop closure factor. The local sliding window-based optimization method is employed for real-time processing. As a result, the proposed method improves the accuracy of the localization and mapping compared with the state-of-the-art LiDAR or LiDAR-inertial SLAM methods. In addition, the proposed method can localize accurately without revisit, required for conventional graph-based SLAM for graph optimization, unlike previous studies. The proposed method is intensively validated with a data set collected in a city where the Global Navigation Satellite System (GNSS) signal is unreliable and on a university campus.