Image deblurring is a classic computer vision task which aims to recover the latent sharp image from blurry input. A number of deep-learning-based methods have been introduced to solve this problem, using multi-scale, multi-patch, or multi-temporal approaches. These approaches achieved good performances in image deblurring. However, when we focus on inference speed, these approaches have some limitations. In this paper, we propose a fast and accurate single-stage network named FDANet. FDANet can handle dynamic blurs in computationally efficient operations by exploiting frequency-domain information with frequency-domain attention. The proposed frequency-domain attention block separately attends regions with different degrees of blurs. The shallowest version of our network has the shortest inference time among existing deblurring methods, and the deeper counterpart surpasses other methods with comparable processing time.