Dataset condensation with atypical samples

Abstract

The advancement of deep learning relies on huge datasets, which involves extensive training cost. To reduce the size of dataset, dataset distillation emerges, which aims to synthesize a small dataset encoding the information of the entire training set. However, we argue that existing dataset condensation method excessively focus on synthesizing typical samples rather than atypical samples, thereby diminish the network’s generalization capacity. To address the limitation, we propose a new strategy, Select-and-Match (SelMatch), that combines dataset condensation and selection. Our method selects important samples first, and perform condensation with regard to selected samples, which enriches the overall information within the synthetic dataset. We evaluate our method on the CIFAR-100 benchmark and demonstrate its superiority over state-of-the-art selection-only and condensation-only methods.