Federated Learning

FRS - Fedscope
Recommendation
System

Mid Semester Evaluation

By - Ujjwal Kumar, Aditya Pawar, Shruti Tyagi and Sargam Mahajan

Table of Contents

01 Motivation 06 Other Diagrams

02 Problem Definition 07 Implementation Details

03 Scope 08 Hardware/Software Requirements

04 Literature Survey 09 Conclusion

05 System Architecture 10 References

Motivation
In the era of big data, data security and user privacy have become priority for both businesses
and individuals. Machine learning is critical to our future and has massively grown in past 2
decades, revolutionizing industries such as manufacturing, banking, healthcare, and more

With merits and development of Machine Learning came various drawbacks, some of them are :
Data Privacy
Identical Data Distribution
Aggregation of Data Sets

Federated learning is new approach to machine learning which has been devised to overcome
these 3 drawbacks. In federated learning, only learning parameters are shared between central
server and participants.

It addresses crucial issues such as data confidentiality, data access rights, and data protection,
by allowing multiple users to create centralized and precise model without sharing private user
data. It achieves personalization, reduces communication costs, and is considered the future.
 Jose Corbacho (Apr 30, 2018). Federated Learning (Bringing Machine Learning to the edge with Kotlin and Android)
https://proandroiddev.com/federated-learning-e79e054c33ef

Working of Federated Learning explained diagramatically

Motivation
However, federated learning has it's own set of challenges

Systems Heterogeneity : Federated networks have large number of heterogeneous devices

with varying levels of computing power, storage, and network connectivity. This results in
challenges such as stragglers and fault tolerance

Statistical Heterogeneity : Data in federated networks is generated and collected in a non-

identically distributed manner across network. This can increase the likelihood of stragglers
and add complexity in terms of modeling, analysis, and evaluation.

Privacy Concerns : Major concern in federated learning, as communicating model updates

throughout training process can reveal sensitive information. Recent methods aim to
enhance privacy, but come at the cost of reduced model performance or system efficiency.

Expensive Communication : Communication can be more expensive than classical data

centers, making it necessary to develop communication-efficient methods
Problem Definition

Use Federated Learning to create a Client-Server cluster using Hardware or

Software, Implement FedScope Architecture which is a highly customisable
platform and leverage it's modular and personalised aspects to form a
Recommendation system which has improved privacy preservation and quality
Scope
Our project was initiated to address the critical issues regarding data privacy and protection,
which are of paramount importance with the ongoing proliferation of data.
The scope of our project aims to solve the following challenges :

1. Statistical Heterogeneity
2. Systems Heterogeneity
3. Privacy Concerns

Solving the "Expensive Communication" challenge is not a part of our project's scope

We aim to implement the following during the course of our project :

1. Create a Client-Server Cluster using Hardware or Software

2. Implement FedScope Architecture on generalised dataset
3. Implement FedScope Architecture on graph data
4. Develop a Recommendation System based on FedScope Architecture
Literature Survey
Our group read and studied more than 15 research papers as part of Literature Survey.
Some of the most relevant and notable papers are as follow :

Federated Social Recommendation with Graph Neural Network : This paper proposes a novel
framework for social recommendation with Graph neural network (FeSoG) to address 3
critical challenges: heterogeneity, personalisation, and privacy protection.

FederatedScope: A Flexible Federated Learning Platform for Heterogeneity : FederatedScope

is a novel FL platform that uses an event-driven architecture to express the behaviors of
servers and clients independently from a global perspective, making it easier to implement
FL algorithms and more modularized.

Layer-Wised Model Aggregation for Personalized Federated Learning : This paper solves the
problem of feature extraction of different layers by deploying a dedicated hypernetwork for
client on the server side to identify mutual contributions and provide layer wise aggregation.
Literature Survey

Personalized Federated Learning Using HyperNetworks : This paper deals with personalized
federated learning using hypernetworks, which is a small neuralnetwork applied to a target
network. It provides better results than conventional algorithms, and has been compared to
multiple standard datasets and implemented using graphs and architecture diagrams.

FEDBN: Federated Learning on NON-IID features via local batch normalization : This paper
uses an algorithm to deal with statistical heterogenity in Federated learning. It is better than
FedAvg and FedProx algorithms and is a properly tested and accurate algorithm.

FedGNN: Federated Graph Neural Network for Privacy-Preserving Recommendation: The

paper proposes a federated framework for privacy-preserving GNN-based recommendation
that can locally train GNN models for each user client, while protecting user privacy through
differential privacy techniques and incorporating randomly sampled items. The approach
achieves competitive results with existing centralized GNN-based recommendation methods
while effectively protecting user privacy.
System Architecture
Other Diagrams
Class Diagram
Other Diagrams

Sequence Diagram

The sequence diagram explains

the entire process of model
sharing, individual training and
aggregation in a message passing
manner. It shows the process for
an entire round of training.
Other Diagrams
Collaborative filtering

ends ITEM 3
mm
Reco
Likes
ITEM 1 Likes
Similar ITEM 1
Likes

User 1 User 2
ITEM 2 Likes Likes ITEM 2
Other Diagrams

UI Design made on Figma

For Research Paper For Keyword

Recommendation Recommendation
Implementation Details
Implementation of our Project consists of four different stages as follow :

1. Create a Client-Server Cluster using Hardware or Software

In this step our focus will be on creating a cluster of machines consisting of clients and servers to
demonstrate a simulation of federated learning approach. Cluster creation will be carried out in
either of these methodologies

Hardware Approach : Establish connection between different machines by treating one as server
and others as clients. We have tried the thunderbolt approach to connect MAC devices and are
currently working on an approach utilizing Hubs available at college premises

Software Approach : We are also looking at creating a virtual cluster on tools such as Proxmox or
by creating multiple machines over the cloud
Implementation Details

2. Implement FedScope Architecture on generalised dataset

Results of the Implementation of the "Femnist" dataset on FedScope Architecture are as below :

Implemented 300 rounds

consisting of 10 clients
Implementation Details

3. Implement FedScope Architecture on graph data

Results of Implementation of "Word Net 18" graph data on FedScope Architecture are as below :

Implemented 400 rounds

consisting of 5 clients
Implementation Details

4. Develop a Recommendation System based on FedScope Architecture

User Item Graph

Vertical Federated Learning : In VFL, the set of users is the same across
different databases, and each participators only has partial items. In this
setting, the user embedding is shared across all participators and each
client maintains its own item embedding.

Horizontal Federated Learning : In HFL, the set of items is the same

across different participators, and they only share the item embedding
with the coordination server.

Collaborative Filtering : Technique that can filter out items that a user
might like on the basis of reactions by similar users.
Hardware/Software Requirements
Hardware requirements (for creation of clusters) :
10 GPU computers for distributed mode

Software requirements (for creation of clusters) :

Proxmox VE (Open-source software)

Software requirements (for development) (Tech Stack)

Language: Python with ML Libraries such as Tensorflow,
NLTK, rdkit, numpy, PyTorch and more
Federated Framework: Flower/TFF

Hardware and software requirements (for deployment)

AWS for standalone mode
10 GPU computers for distributed mode
Conclusion

Our group members have gained hands-on experience in Federated Learning because of this
project. We have gained insights on different Federated learning architectures, algorithms, and
learned about challenges of federated learning. Our team has developed a good understanding
about the research process.

Picking up an extremely new, research oriented and challenging topic has provided all of us with
an enriching as well as challenging experience.Our focus will lie on developing the system as
planned, implementing the cluster and improvising it in the future
References
[1] Liu, Z., Yang, L., Fan, Z., Peng, H., & Yu, P. S. (2022, February 4). Federated Social Recommendation with Graph Neural
Network. ACM Transactions on Intelligent Systems and Technology; Association for Computing Machinery.
(https://doi.org/10.1145/3501815)

[2] Xie, Y., Wang, Z., Gao, D., Chen, D., Yao, L., Kuang, W., Li, Y., Ding, B., & Zhou, J. (2023). FederatedScope: A Flexible
Federated Learning Platform for Heterogeneity. Proceedings of the VLDB Endowment, 16(5), 1059–1072.
(https://doi.org/10.14778/3579075.3579081)

[3] Shamsian, A. (2021b, March 8). Personalized Federated Learning using Hypernetworks. arXiv.org.
(https://arxiv.org/abs/2103.04628)

[4] Li, X. (2021, February 15). FedBN: Federated Learning on Non-IID Features via Local Batch Normalization. (arXiv.org.
https://arxiv.org/abs/2102.07623)

[5] Bonawitz, K. (2019, February 4). Towards Federated Learning at Scale: System Design. arXiv.org.
(https://arxiv.org/abs/1902.01046)

[6] Wang, Z. (2022, April 12). FederatedScope-GNN: Towards a Unified, Comprehensive and Efficient Package for Federated
Graph Learning. arXiv.org. (https://arxiv.org/abs/2204.05562)

https://www.linkedin.com/pulse/4-key-differences-between-federated-learning-classical-naveen-joshi/?trk=pulse-
article_more-articles_related-content-card