One of the fundamental problems in MR image is its slow acquisition time compared to other imaging modalities. Several approaches have been proposed to accelerate MRI, and nowadays deep learning has been showing great promises in this area. However, since k-space line contains different information, which line to acquire (or sampling pattern) plays an important role in MRI reconstruction. Previous works focused on jointly optimizing the sampling pattern and reconstruction network or active sampling. In this work, we propose a novel strategy for determining sampling patterns, named Progressive Sampling Pattern Network (PSP-Net), which progressively optimize subject-common and subject-specific sampling patterns to improve the reconstruction performance with time efficiency.