| Speaker : | Fabio Pianese |
| Nokia Bell Labs | |
| Date: | 13/12/2023 |
| Time: | 10:30 am - 11:30 am |
| Location: | Room 4B01 |
Abstract
Federated Learning (FL) attempts to protect the privacy of the participants in the scheme by uploading locally-trained models from edge devices to a server, where they get combined into a global model and redistributed to the edge devices. In the established FL literature, this combination usually happens by averaging, which appears to limit the accuracy of the global model and introduces challenges in applying the scheme to heterogeneous devices with different model architectures.
The paper we will present in this session, “Ensemble distillation for robust model fusion in federated learning”, introduces model fusion techniques based on knowledge distillation that enable the simultaneous use of various local model sizes / architectures. Distillation also drastically improves the global accuracy compared to average-based methods such as FedAVG – still without compromising users’ data privacy. The proposed scheme, FedDF, is faster to train and requires fewer communication rounds than previously published FL techniques. We liked this paper because it introduces a couple of nifty ideas that might also be profitably applied in other FL scenarios.
Reference
Tao Lin, Lingjing Kong, Sebastian U. Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, pages 2351–2363, Red Hook, NY, USA, December 2020.