Recent studies have shown that the use of fashion landmark information has achieved great success in the task of clothes recognition. However, the landmark annotation is very labor intensive and time consuming. It also suffers from inter-and intra-individual variability. To overcome these problems, we propose a `landmark-free' fashion recognition method. We introduce a two-branch feature selective network exploiting class-activated regions for category classification and attribute prediction. Note that we prove that the proposed network has an excellent ability to effectively learn a discriminative feature representation of a `clothing image' without any additional supervisions. Experimental results on the benchmark dataset show that the proposed network yields comparable performance to the state-of-the-art methods, which strongly depend on the fashion landmark.