Loosely-constrained federated learning for heterogeneous local participants

Abstract

The conventional deep learning methods, which involve centralizing data for training, have given rise to concerns regarding privacy. As a response, Federated Learning has emerged as a viable alternative, wherein individual models are trained on local devices without directly sharing data, and the acquired knowledge is then integrated into a global model. However, existing Federated Learning techniques encounter certain limitations when confronted with the growing heterogeneity among participants. In this study, we identify six key challenges: (1) the lack or absence of labels, (2) disparities in domains and labels among participants, (3) challenges related to continual learning and forgetting, (4) issues pertaining to learning from relational data, (5) discrepancies in computational capabilities among participants, and (6) the retrieval of optimal neural network models. To tackle these challenges, we propose a Loosely-Constrained Federated Learning framework, which enables participants with diverse heterogeneity to share mutually beneficial knowledge for the purpose of learning. We demonstrate that our proposed methods surpass existing approaches in terms of performance and communication efficiency for each of the aforementioned problems.