A Comparative Study of Federated Learning Aggregation Algorithms under Non-IID Conditions

Abstract

Privacy can be maintained in collaborative model training by enabling clients to contribute without exposing their local datasets, as supported by Federated Learning (FL). Performance-wise, however, in scenarios involving non-IID data across clients, training becomes significantly more complex and less reliable, as is often the case with real-world datasets. In this work, we provide a comprehensive comparative review of three popular aggregation algorithms in FL: Federated Averaging (FedAvg), Federated Proximal (FedProx) and Stochastic Controlled Averaging (SCAFFOLD). Among these baselines, FedAvg is the fundamental approach, FedProx adds the proximal term to suppress the drift of clients and SCAFFOLD utilizes control variates to adjust the updating directions of local clients. According to the related work, FedAvg, which is communication-efficient, is unstable and slow on non-IID data. FedProx stabilizes convergence, and SCAFFOLD has the best convergence performance, but it has to exchange control variates to solve them, thus consuming more communication. While no algorithm appears to dominate for all metrics, it underscores the significance of choosing FL strategy which can meet certain trade-off between accuracy, speed of convergence and communication expense.