Ensemble distillation in federated learning with dataset condensation

Abstract

Federated learning is a collaboratively training process of several local client models. It aims to train a global server model from client models without directly accessing to private client data. However, federated learning suffers from data heterogeneity of client data. Ensemble distillation that distills knowledge of clients to server by matching prediction is a possible solution for data heterogeneity but it requires a public dataset. In this research we propose a method to make use of condensed client data in federated learning that doesn’t require public dataset. To mitigate data heterogeneity issue, we condensed client data and use it as an input of ensemble distillation and empirically show that it is beneficial to improve overall performance in heterogeneous situations.