The generation of a depth map is one of the most important processes in common stereoscopic conversion. Since generating a depth map by manual painting is time consuming, many automatic depth estimation techniques have been proposed. One approach to depth map generation is to utilize vanishing point detection to calculate relative depths in the image. However, existing techniques typically produce gradually changing depth variations toward a vanishing point, and are not capable of capturing detailed depth that is caused by complex scene geometry. This paper presents a new method to generate a depth map of a single input image of a building. Our method builds upon the assumption that an image of a building contains detailed line segments coupled with vanishing points. We show how to detect vanishing points, extract line segments, and find relative positional relations between the line segments. The relationship helps generate the relative depth of each line segment, which can be propagated to cover the entire building. As the depth map is generated from the collection of line segments, the details of the depth are faithfully reproduced.