FedClassAvg Local Representation Learning for Personalized Federated Learning on Heterogeneous Neural Networks

2025-05-06 0 0 2.15MB 10 页 10玖币

侵权投诉

FedClassAvg: Local Representation Learning for Personalized

Federated Learning on Heterogeneous Neural Networks

Jaehee Jang

hukla@snu.ac.kr

Department of Electrical and Computer Engineering

Seoul National University

Seoul, South Korea

Heonseok Ha

heonseok.ha@snu.ac.kr

Department of Electrical and Computer Engineering

Seoul National University

Seoul, South Korea

Dahuin Jung

annajung0625@snu.ac.kr

Department of Electrical and Computer Engineering

Seoul National University

Seoul, South Korea

Sungroh Yoon∗

sryoon@snu.ac.kr

Department of Electrical and Computer Engineering

Interdisciplinary Program in Articial Intelligence

Seoul National University

Seoul, South Korea

ABSTRACT

Personalized federated learning is aimed at allowing numerous

clients to train personalized models while participating in collabora-

tive training in a communication-ecient manner without exchang-

ing private data. However, many personalized federated learning

algorithms assume that clients have the same neural network archi-

tecture, and those for heterogeneous models remain understudied.

In this study, we propose a novel personalized federated learning

method called federated classier averaging (FedClassAvg). Deep

neural networks for supervised learning tasks consist of feature

extractor and classier layers. FedClassAvg aggregates classier

weights as an agreement on decision boundaries on feature spaces

so that clients with not independently and identically distributed

(non-iid) data can learn about scarce labels. In addition, local fea-

ture representation learning is applied to stabilize the decision

boundaries and improve the local feature extraction capabilities

for clients. While the existing methods require the collection of

auxiliary data or model weights to generate a counterpart, FedClas-

sAvg only requires clients to communicate with a couple of fully

connected layers, which is highly communication-ecient. More-

over, FedClassAvg does not require extra optimization problems

such as knowledge transfer, which requires intensive computation

overhead. We evaluated FedClassAvg through extensive experi-

ments and demonstrated it outperforms the current state-of-the-art

algorithms on heterogeneous personalized federated learning tasks.

∗Corresponding author

Permission to make digital or hard copies of all or part of this work for personal or

classroom use is granted without fee provided that copies are not made or distributed

for prot or commercial advantage and that copies bear this notice and the full citation

on the rst page. Copyrights for components of this work owned by others than ACM

must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,

to post on servers or to redistribute to lists, requires prior specic permission and/or a

fee. Request permissions from permissions@acm.org.

ICPP ’22, August 29-September 1, 2022, Bordeaux, France

ACM ISBN 978-1-4503-9733-9/22/08. . . $15.00

https://doi.org/10.1145/3545008.3545073

CCS CONCEPTS

•Computing methodologies →Distributed articial intelli-

gence

;Computer vision tasks;Computer vision representations;Su-

pervised learning by classication.

KEYWORDS

Neural Networks, Federated Learning, Model Heterogeneity, Re-

source Constraint, Communication Ecient, Representation Learn-

ing

ACM Reference Format:

Jaehee Jang, Heonseok Ha, Dahuin Jung, and Sungroh Yoon. 2022. FedClas-

sAvg: Local Representation Learning for Personalized Federated Learning on

Heterogeneous Neural Networks. In 51st International Conference on Parallel

Processing (ICPP ’22), August 29-September 1, 2022, Bordeaux, France. ACM,

New York, NY, USA, 10 pages. https://doi.org/10.1145/3545008.3545073

1 INTRODUCTION

Federated learning is a privacy-preserving collaborative machine-

learning technique. It enables multiple clients and a global server

to train by exchanging knowledge from local training and the data

itself. Because the data distributions of clients are not independent

and identically distributed (non-iid), and conventional parallel ma-

chine learning algorithms assume iid data distributions of clients,

new algorithms are needed. Beginning with FedAvg [

], many

studies [

] have been proposed to improve the generalization

performance of federated learning algorithms. However, because

federated learning concentrates on improving the global model, the

client model performance for local data distribution deteriorates.

Therefore, the concept of personalized federated learning has

been proposed. It aims for the client to train personalized models

collaboratively while maintaining model performance on local data

distributions. Many personalized federated learning techniques [

] have signicantly contributed to addressing data heterogeneity

among clients. Most personalized federated learning algorithms

constrain all clients to use the same model architecture. However,

it is necessary for personalized federated learning to allow clients

to choose the dierent model architectures that are eective for

various data distributions of clients.

arXiv:2210.14226v2 [cs.LG] 27 Oct 2022

ICPP ’22, August 29-September 1, 2022, Bordeaux, France Jang et al.

B)Client update:ℱ−1→ ℱ ,−1→

A)Global classiﬁer update: −1→

( , )

′

′′

ℱ

ℱ(′)

ℱ(′′ )

ℒ(,)

ℒ(ℱ(′),ℱ(′′ ))

ℱ11

Client 1

ℱ22

Client 2

ℱ

Client K

…

Server

←+ + ⋯+

③① ① ①

②

③ ③ ℒ(,)

|1|

| |

|2|

| |

Figure 1: Illustration of FedClassAvg. F∗are feature extractors and C∗are classiers. FedClassAvg aggregates client classiers

C𝑘and build a global classier Cas described in A), by the following workow: 1. clients transmit local classiers to the

server, 2. the transmitted local classiers are linearly combined as a global classier, and 3. the global classier is broadcast

to clients. The client models are updated with local feature representation learning (L𝐶 𝐿 ), supervised learning (L𝐶 𝐸 ), and

proximal regularization (L𝑅) as described in B).

Several studies [

] have resolved model heterogene-

ity through knowledge transfer. They have successfully delivered

learned knowledge from one client to another by using soft predic-

tions on common public data. However, it is a burden for the global

server to collect auxiliary data, when it is inaccessible to actual

training data distributions that clients possess. Moreover, it might

be infeasible for some tasks in which data privacy is crucial, such as

medical or nancial data, to require even minimal information on

client data distributions. Furthermore, additional optimization prob-

lems for knowledge transfer occur in addition to model training and

aggregation, resulting in extra computation overhead. There has

also been a study of heterogeneous personalized federated learning

using prototype learning [

], but requires models to have the same

output shape which highly limts the model choices for clients.

Therefore, we introduce a novel personalized federated learning

framework for heterogeneous models called federated classier

averaging (FedClassAvg). An overview of the proposed method is

presented in Figure 1. In general, a deep neural network model for

a supervised learning task can be divided into a feature extractor

and a classier. The feature extractors maps input data onto fea-

ture spaces, and the classier determines the decision boundaries

between feature space representations of dierent class labels. Fed-

ClassAvg learns heterogeneous models through classier weight

aggregation. By unifying the classier, client models learn the same

decision boundary, and dierent feature extractors learn how the

feature space representation should be positioned to t in the de-

cision boundaries. Therefore, FedClassAvg enables heterogeneous

personalized federated learning without the need for additional

data collection and transmission. Moreover, FedClassAvg does not

require computations other than model training or classier aggre-

gation. It is communication-ecient because only a couple of fully

connected layers are transferred instead of the parameters of the

entire model. In our implementation, the clients in FedClassAvg

transfer only 2KB of classier weights for every communication

round.

In addition to classier aggregation, we applied proximal regu-

larization to reduce the L2 distance between the global and client

classiers. This reinforces the unied objective of the client models

and improves the overall training accuracy. Moreover, we apply

local feature representation learning using a supervised contrastive

loss [

]. Feature representation learning through a contrastive

loss helps the feature representations of semantically the same data

to be closer while dierent data are farther away. However, classier

aggregation alone cannot prevent decision boundary drifts caused

by client models and data heterogeneity. Therefore, we use the

supervised contrastive loss to distance the feature-space represen-

tation of dierent labels, so that a slight migration of the decision

boundary does not ip the labels.

The contributions of this paper and the proposed FedClassAvg

are as follows:

•

We introduced FedClassAvg, a novel framework for per-

sonalized federated learning on heterogeneous models, by

combining classier aggregation with local representation

learning. It does not require any auxiliary data, or intensive

computations other than model training and aggregation.

•

We evaluated the proposed method using various deep neu-

ral network models and datasets. The experimental results

suggest that FedClassAvg outperforms state-of-the-art algo-

rithms.

•

Through several analyses, we demonstrated that FedClas-

sAvg can convey collaborative knowledge using only classi-

er aggregations.

2 RELATED WORK

2.1 Personalized federated learning for

heterogeneous models

After several studies have discovered the possibility of federated

learning methods with heterogeneous models [

], personal-

ized federated learning methods for heterogeneous models using

knowledge transfer also have been proposed in the literature [

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

FedClassAvg:LocalRepresentationLearningforPersonalizedFederatedLearningonHeterogeneousNeuralNetworksJaeheeJanghukla@snu.ac.krDepartmentofElectricalandComputerEngineeringSeoulNationalUniversitySeoul,SouthKoreaHeonseokHaheonseok.ha@snu.ac.krDepartmentofElectricalandComputerEngineeringSeoulNationalUniv...

展开>> 收起<<

FedClassAvg Local Representation Learning for Personalized Federated Learning on Heterogeneous Neural Networks.pdf

共10页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

FedClassAvg Local Representation Learning for Personalized Federated Learning on Heterogeneous Neural Networks

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: