Real-time bundle adjustment and OPA camera depth refinement system for 3D reconstruction

Abstract

Joint or interleaved optimizations of the camera parameters and 3D structure in structure-from-motion (SfM) systems are computationally expensive due to the nature of non-linear solvers, which limits their use in resource-constrained environments. To solve this problem, we propose to model the cost function and incrementally refine the 3D structure using a direct refinement method. The proposed incremental direct structure refinement (IDSR) is non-iterative and does not require re-triangulation and non-linear solver, which significantly reduces the computational cost of the system. We also implement a resection-intersection (RI) approach where the traditional non-linear solver of the intersection phase is replaced by the proposed IDSR. Experimental results on publicly available datasets show that, compared with conventional refinement methods, the proposed IDSR-based RI approach provides comparable reconstruction accuracy and is 3 to 20 times faster on single-core CPU environment depending on the IDSR configuration. In addition, we propose a depth refinement system for offset pixel aperture (OPA) camera which has narrow depth range due to a small baseline. Multi-view depth measurements of the OPA camera are fused using the proposed model.