This paper proposes a novel complete navigation system for autonomous flight of small unmanned aerial vehicles (UAVs) in GPS-denied environments. The hardware platform used to test the proposed algorithm is a small, custom-built UAV platform equipped with an onboard computer, RGB-D camera, 2D light detection and ranging (LiDAR), and altimeter. The error-state Kalman filter (ESKF) based on the dynamic model for low-cost IMU-driven systems is proposed, and visual odometry from the RGB-D camera and height measurement from the altimeter are fed into the measurement update process of the ESKF. The pose output of the ESKF is then integrated into the open-source simultaneous location and mapping (SLAM) algorithm for pose-graph optimization and loop closing. In addition, the computationally efficient collision-free path planning algorithm is proposed and verified through simulations. The software modules run onboard in real time with limited onboard computational capability. The indoor flight experiment demonstrates that the proposed system for small UAVs with low-cost devices can navigate without collision in fully autonomous missions while establishing accurate surrounding maps.