This paper proposes a method of generating 3D building models with precise geospatial information and a photograph-based fa double dagger ade appearance from panoramic image sequences and digital maps. 3D building modeling research is actively being conducted in areas such as geographic information systems, virtual reality, and augmented reality. However, the generation of realistic 3D models from a ground-level viewpoint is still extremely costly in terms of labor of modeling experts, and collection of data. We have developed a method for 3D building modeling with high-resolution photograph-based appearance information using panoramic images captured at ground level with a mobile mapping system, and geospatial information obtained from a digital map. The proposed method includes 1) pre-processing for tilt correction and base 3D model generation, 2) geo-registration of panoramic images with minimal user input, and 3) building height and shape estimation. This paper presents the proposed method and the quantitative performance measure obtained from a developed test modeling system. In addition, modeling results from an experimental dataset are also presented.