Image retargeting aims at adapting a given image to fit the size of arbitrary displays without severe visual distortions. To achieve this task successfully, it is essential to define a reliable image importance map (IIM) since it guides subsequent retargeting procedures. In this letter, we introduce a novel IIM for effective image retargeting. Specifically, we define our IIM by exploiting the higher order statistics (HOS) of the diffusion space for image retargeting. We call it texture-aware salient edge (TASE) map. Based on the proposed TASE map, we obtain visually acceptable retargeting results, even in the cluttered background and in the presence of noise as well. The proposed method has been extensively tested, and experimental results show that the proposed scheme is effective for image retargeting compared to other various state-of-the-art methods.