Many studies have found that the ambience of a place has a significant effect on the satisfaction or behavioral intention of its visitors. To utilize the atmospheric characteristics of places, in this paper, we present a novel method to recognize the ambience of a place from images that are taken at a place based on a deep convolutional neural network (ConvNet). We trained our model such that it can infer place ambience without any help from other feature extractors. By transferring generic visual features, we improve the performance as well. Experiments were done on the public dataset shared on the Yelp Dataset Challenge. The results show that the proposed method can recognize the place ambience better than existing methods.