Federated Learning : Theory and Practice

Federated Learning: Theory and Practice provides a holistic treatment to federated learning, starting with a broad overview on federated learning as a distributed learning system with various forms of decentralized data and features. A detailed exposition then follows of core challenges and practical modeling techniques and solutions, spanning a variety of aspects in communication efficiency, theoretical convergence and security, viewed from different perspectives. Part II features emerging challenges stemming from many socially driven concerns of federated learning as a future public machine learning service, and Part III and IV present a wide array of industrial applications of federated learning, including potential venues and visions for federated learning in the near future. This book provides a comprehensive and accessible introduction to federated learning which is suitable for researchers and students in academia and industrial practitioners who seek to leverage the latest advances in machine learning for their entrepreneurial endeavors Presents the fundamentals and a survey of key developments in the field of federated learning Provides emerging, state-of-the art topics that build on fundamentals Contains industry applications Gives an overview of visions of the future Cover image Title page Table of Contents Copyright Contributors Preface Part 1: Optimization fundamentals for secure federated learning Chapter 1: Gradient descent-type methods Abstract Acknowledgements 1.1. Introduction 1.2. Basic components of GD-type methods 1.3. Stochastic gradient descent methods 1.4. Concluding remarks References Chapter 2: Considerations on the theory of training models with differential privacy Abstract 2.1. Introduction 2.2. Differential private SGD (DP-SGD) 2.3. Differential privacy 2.4. Gaussian differential privacy 2.5. Future work References Chapter 3: Privacy-preserving federated learning: algorithms and guarantees Abstract 3.1. Introduction 3.2. Background and preliminaries 3.3. DP guaranteed algorithms 3.4. Performance of clip-enabled DP-FedAvg 3.5. Conclusion and future work References Chapter 4: Assessing vulnerabilities and securing federated learning Abstract 4.1. Introduction 4.2. Background and vulnerability analysis 4.3. Attacks on federated learning 4.4. Defenses 4.5. Takeaways and future work References Chapter 5: Adversarial robustness in federated learning Abstract 5.1. Introduction 5.2. Attack in federated learning 5.3. Defense in federated learning 5.4. Conclusion References Chapter 6: Evaluating gradient inversion attacks and defenses Abstract 6.1. Introduction 6.2. Gradient inversion attacks 6.3. Strong assumptions made by SOTA attacks 6.4. Defenses against the gradient inversion attack 6.5. Evaluation 6.6. Conclusion 6.7. Future directions References Part 2: Emerging topics Chapter 7: Personalized federated learning: theory and open problems Abstract 7.1. Introduction 7.2. Problem formulation of pFL 7.3. Review of personalized FL approaches 7.4. Personalized FL algorithms 7.5. Experiments 7.6. Open problems 7.7. Conclusion References Chapter 8: Fairness in federated learning Abstract Acknowledgements 8.1. Introduction 8.2. Notions of fairness 8.3. Algorithms to achieve fairness in FL 8.4. Open problems and conclusion References Chapter 9: Meta-federated learning Abstract 9.1. Introduction 9.2. Background 9.3. Problem definition and threat model 9.4. Meta-federated learning 9.5. Experimental evaluation and discussion 9.6. Conclusion References Chapter 10: Graph-aware federated learning Abstract 10.1. Introduction 10.2. Decentralized federated learning 10.3. Multi-center federated learning 10.4. Graph-knowledge based federated learning 10.5. Numerical evaluation of GFL models 10.6. Summary References Chapter 11: Vertical asynchronous federated learning: algorithms and theoretic guarantees Abstract Acknowledgements 11.1. Introduction 11.2. Vertical federated learning 11.3. Convergence analysis 11.4. Perturbed local embedding for smoothness 11.5. Numerical tests References Chapter 12: Hyperparameter tuning for federated learning – systems and practices Abstract 12.1. Introduction 12.2. Systems resources 12.3. Cross-device FL hyperparameters 12.4. System challenges in FL HPO 12.5. State-of-the-art 12.6. Conclusion References Chapter 13: Hyper-parameter optimization in federated learning Abstract 13.1. Introduction 13.2. State-of-the-art FL-HPO approaches 13.3. FLoRA: a single-shot FL-HPO approach 13.4. Empirical evaluation 13.5. Conclusion References Chapter 14: Federated sequential decision making: Bayesian optimization, reinforcement learning, and beyond Abstract Acknowledgements 14.1. Introduction 14.2. Federated Bayesian optimization 14.3. Federated reinforcement learning 14.4. Related work 14.5. Open problems and future directions References Chapter 15: Data valuation in federated learning Abstract Acknowledgements 15.1. Introduction 15.2. Data valuation: motivations and incentives 15.3. Simple valuation methods 15.4. Related work: conventional data valuation 15.5. Extending to the federated setting: does it work? 15.6. Vertical data valuation: feature valuation 15.7. Horizontal data valuation: gradient valuation 15.8. Learning-based valuation 15.9. Conclusion and future work References Part 3: Applications & ethical considerations Chapter 16: Incentives in federated learning Abstract Acknowledgements 16.1. Overview and motivation 16.2. Problem setting 16.3. Incentives 16.4. Contribution evaluation 16.5. Client selection 16.6. Reward allocation 16.7. Other incentives 16.8. Monetary rewards 16.9. Non-monetary rewards 16.10. Conclusion and future work References Chapter 17: Introduction to quantum federated machine learning Abstract 17.1. Introduction 17.2. Quantum federated learning 17.3. Variational quantum circuits 17.4. Demonstration 17.5. Advanced settings 17.6. Discussion 17.7. Conclusion References Chapter 18: Federated quantum natural gradient descent for quantum federated learning Abstract 18.1. Introduction 18.2. Variational quantum circuit 18.3. Quantum natural gradient descent 18.4. Quantum natural gradient descent for VQC 18.5. Federated quantum natural gradient descent 18.6. Experimental results 18.7. Conclusion and discussion References Chapter 19: Mobile computing framework for federated learning Abstract 19.1. Federated learning on mobile platforms 19.2. Challenge in mobile-based federated learning 19.3. Helios: a self-coordinated federated learning framework for mobile platform 19.4. Performance evaluation 19.5. Conclusion and future directions References Chapter 20: Federated learning for privacy-preserving speech recognition Abstract 20.1. From voice protection to federated assistant 20.2. Federated speech recognition with synthetic data 20.3. Conclusion References Chapter 21: Ethical considerations and legal issues relating to federated learning Abstract Acknowledgements 21.1. Introduction 21.2. Global trends and ethical guidelines for trustworthy AI and the universal fundamental principles 21.3. Privacy and personal data rights and the international data protection regime 21.4. Intellectual property rights relating to federated learning systems 21.5. Governance structure 21.6. Conclusion References Index Federated Learning: Theory and Practi ce provides a holisti c treatment to federated learning as a distributed learning system with various forms of decentralized data and features. Part I of the book begins with a broad overview of opti mizati on fundamentals and modeling challenges, covering various aspects of communicati on effi ciency, theoretical convergence, and security. Part II features emerging challenges stemming from many socially driven concerns of federated learning as a future public machine learning service. Part III concludes the book with a wide array of industrial applicati ons of federated learning, as well as ethical considerations, showcasing its immense potential for driving innovation while safeguarding sensitive data. Federated Learning: Theory and Practi ce provides a comprehensive and accessible introducti on to federated learning which is suitable for researchers and students in academia, and industrial practitioners who seek to leverage the latest advance in machine learning for their entrepreneurial endeavors. Presents the fundamentals and a survey of key developments in the field of federated learning Provides emerging, state-of-the art topics that build on fundamentals Contains industry applications Gives an overview of visions of the future