1 Federated Fuzzy Neural Network with Evolutionary Rule Learning

2025-04-30 0 0 1.69MB 12 页 10玖币

侵权投诉

Federated Fuzzy Neural Network with Evolutionary

Rule Learning

Leijie Zhang, Ye Shi∗,Member, IEEE, Yu-Cheng Chang and Chin-Teng Lin∗,Fellow, IEEE

Abstract—Distributed fuzzy neural networks (DFNNs) have

attracted increasing attention recently due to their learning

abilities in handling data uncertainties in distributed scenarios.

However, it is challenging for DFNNs to handle cases in which

the local data are non-independent and identically distributed

(non-IID). In this paper, we propose a federated fuzzy neural

network (FedFNN) with evolutionary rule learning (ERL) to cope

with non-IID issues as well as data uncertainties. The FedFNN

maintains a global set of rules in a server and a personalized

subset of these rules for each local client. ERL is inspired by

the theory of biological evolution; it encourages rule variations

while activating superior rules and deactivating inferior rules

for local clients with non-IID data. Speciﬁcally, ERL consists of

two stages in an iterative procedure: a rule cooperation stage

that updates global rules by aggregating local rules based on

their activation statuses and a rule evolution stage that evolves

the global rules and updates the activation statuses of the local

rules. This procedure improves both the generalization and

personalization of the FedFNN for dealing with non-IID issues

and data uncertainties. Extensive experiments conducted on a

range of datasets demonstrate the superiority of the FedFNN

over state-of-the-art methods. Our code is available online1

Index Terms—Federated fuzzy neural network, federated

learning, non-IID dataset, data uncertainty, evolutionary rule

learning.

I. INTRODUCTION

Combining neural networks with fuzzy logic [1], [2], fuzzy

neural networks (FNNs) [3]–[7] have been proposed with pow-

erful learning capabilities and uncertainty handling abilities in

centralized scenarios. However, due to increasing data privacy

concerns, the samples collected from distributed parties must

be processed locally. Distributed FNNs (DFNNs) [8]–[12]

address this issue by learning a global FNN via the integration

of local models. However, the existing DFNNs are fragile to

non-independent and identically distributed (non-IID) data. In

addition, as DFNNs tend to learn a shared group of global rules

for all clients, their personalization ability is limited and their

learned global rules are less adaptive. Furthermore, they regard

the local model integration process as a convex optimization

problem and solve it with the alternating direction method of

multipliers [13], which overlooks the powerful learning ability

of feedforward FNNs.

Ye Shi and Chin-Teng Lin are the corresponding authors.

Leijie Zhang, Yu-Cheng Chang and Chin-Teng Lin are with CIBCI

Lab, the Australian Artiﬁcial Intelligence Institute, the School of Com-

puter Science, University of Technology, Sydney, NSW 2007, Australia.

Email: Leijie.Zhang@student.uts.edu.au, Yu-Cheng.Chang@uts.edu.au, Chin-

Teng.Lin@uts.edu.au. Ye Shi is with the School of Information Science

and Technology, ShanghaiTech University, Shanghai, 201210, China. Email:

shiye@shanghaitech.edu.cn.

1https://github.com/leijiezhang/FedFNN

Activate Inactivate

Creatures

Diverse Living

Environment

Gene 1

Gene 3

Gene

𝐾

Gene 1

Gene

𝐾

Adaptive

Evolve

Gene Status:

Features

𝐾

Features 2

Gene 1

Gene 3

Gene

𝐾

Features 2

Features 3

Gene 3

Gene 2

Adaptive

Evolve

White

Fur

Features 1

Fin-shaped

Feet

Features

𝐾

Smooth

Skin

Thick

Fat

Fin-shaped

Feet

Features

𝐾

Smooth

Skin

Features 3

Thick

Fat

White

Fur

Features 1

Chromosome Selective Gene Activation

Variation Within

A Species

(a) The evolvement of variants within a species via the selective

activation of genes.

FedFNN

Heterogeneous

Local Dataset

Rule 1

Rule 2

Rule 3

Rule𝐾

Features 1

Features 2

Features 3

Features𝐾

Rule 1

Rule 2

Rule 3

Rule 𝐾

Features 1

Features 2

Features 3

Features 𝐾

Personalized

local FNN

Evaluation

Evolutionary

Rule Learning

Features 1

Features 2

Features

Rule 1

Rule 2

Rule 3

Rule

𝐾

Features 3

𝐾

Then

CCC

Rule Set

Then

Selective Rule Activation

Evaluation

Evolutionary

Rule Learning

Activate Inactivate

Gene Status:

(b) The rule selective activation for FedFNN to generate person-

alized local FNNs.

Fig. 1. (a) A brief demonstration of how a species evolve vari-

ants to survive in diverse living environments based on genes

selective activation and expressions. (b) A brief demonstration

of how FedFNN selectively activate a personalized subset of

contributive rules for clients to effectively deal with their local

non-IID data.

Fortunately, federated learning (FL) models [14]–[16] offer

decentralized learning architectures that allow local models

to optimize their parameters using gradient descent. The FL

approach learns a shared model by aggregating the updates ob-

tained from local clients without accessing their data. Notably,

data distribution heterogeneity is also one of the key challenges

for FL. Yet, many efforts [17]–[21] have been made to handle

this issue in FL, in which many approaches consider solutions

that allow clients to have personalized models. However, few

of the existing FL methods are able to simultaneously cope

with data uncertainties and non-IID issues in distributed learn-

arXiv:2210.14393v1 [cs.LG] 26 Oct 2022

ing scenarios. Although several studies have adopted Bayesian

treatments [22] and Gaussian processes [20], [21] to enable

FL methods to handle data uncertainties, their performance

heavily relies on the learning of good posterior inferences and

kernel functions, which is very time-consuming.

To solve the aforementioned issues, in this paper, we

propose a federated fuzzy neural network (FedFNN) with

evolutionary rule learning (ERL) to handle data uncertainties

and non-IID issues. As shown in Fig. 1 (a), the theory of bio-

logical evolution [23] states that variants of the same species

can evolve to adapt to their different living environments by

selectively activating and expressing their genes. Inspired by

this, we use FNNs as our local models and consider them as

compositions of fuzzy rules, which capture valuable local data

information from multiple views, such as distributions. Similar

to the genes of a species, each rule of the FedFNN is a basic

functional component that can be activated or deactivated for

clients according to their performance on local data. Thus,

our FedFNN aims to obtain a group of global fuzzy rules that

can be selectively activated for local clients to enable them

to outperform competing approaches on non-IID data. It is

worth noting that our ERL is a novel algorithm different from

genetic algorithms [24]. These two algorithms are inspired by

the same theory but designed in different ways. The genetic

algorithm treats the whole population as the evolution subject.

It keeps selecting the ﬁttest individuals in each generation

to form more competitive populations until the performance

gets stable. However, it needs to be re-adjusted whenever

the environment is modiﬁed. Instead, the ERL considers the

internal diversity of species, which means that a species

normally includes several types of populations (subspecies)

living in diverse environments. It regards subspecies as its

evolution subjects and selectively optimises and shares gene

groups among subspecies until they perform well in different

environments. In our FedFNN, each agent that handles non-IID

data corresponds to a sub-species. The ERL enables agents to

cooperate with each other during their evolutions but preserve

their personalities to adapt to diverse environments. In general,

our FedFNN aims at learning 1) a group of global rules that

capture valuable information among local clients and 2) a

rule activation strategy for each local client to ensure the

personalization and superior performance of the FedFNN.

The ERL is an iterative learning approach with two stages:

a rule cooperation stage, where the global rules are updated

cooperatively by clients based on their rule activation statuses,

and a rule evolution stage, where the activation statuses of

local rules are adjusted according to their contributions to

the performance of the FedFNN on non-IID local data. The

former stage enhances the cooperation among clients for

learning more representative global rules, which increases the

generalizability of the FedFNN. In contrast, the latter stage

fulﬁls the selective activation of rules that enable the local

FNNs to be more adaptive and perform better on non-IID

data, which improves the personalization of the FedFNN. For

an explanation of the FedFNN model, refer to the diagram

shown in Fig. 1 (b).

The contributions of this paper are as follows,

•We are the ﬁrst to propose FedFNN that integrates fuzzy

neural networks into a federated learning framework to

handle data non-IID issues as well as data uncertainties

in distributed scenarios. FedFNN is able to learn person-

alized fuzzy if-then rules for local clients according to

their non-IID data.

•Inspired by the theory of biological evolution, we design

an ERL algorithm for the learning of FedFNN. ERL

encourages the global rules to evolve while selectively

activating superior rules and eliminating inferior ones

during the training procedure. This procedure ensures the

ability of generalization and personalization of FedFNN.

II. RELATED WORK

A. Distributed fuzzy neural networks

DFNNs [8]–[12] have been proposed to handle the uncer-

tainties encountered in distributed applications. The authors in

[8] proposed a DFNN model that randomly sets the parameters

in antecedents and only updates the parameters in consequent

layers. Later, they extended this work to an online DFNN

model [9]. Their models assume that all clients share the

information in antecedent layers, making this technically not

a seriously distributed method. To avoid this problem, a

fully DFNN [10] model was proposed by adopting consensus

learning in both the antecedent and consequent layers. As

its subsequence variant, a semisupervised DFNN model [12]

was presented to enable the DFNN to leverage unlabeled

samples by using the fuzzy C-means method and distributed

interpolation-based consistency regularization. However, the

existing DFNNs cannot handle situations in which the data dis-

tribution varies across clients. The authors in [11] proposed a

DFNN with hierarchical structures to process the heterogeneity

existing in the variables of training samples. However, instead

of processing the data heterogeneity across distributed clients,

they focused on variable composition heterogeneity, which

meant that data variables were collected from different sources.

Generally, by employing the well-known Takagi-Sugeno (T-

S) fuzzy if-then rules [2], the existing DFNN models build

the antecedent layers of their local models in traditional ways

(e.g., K-means) and calculate the corresponding consequent

layers with closed-form solutions. Then, the original DFNNs

are transformed into convex optimization problems. While

efﬁcient and effective, they are not able to learn local models

with personalized rule sets. Worse, they fail to utilize the

strong learning abilities of neural networks that enable local

FNNs to investigate more adaptive rules.

B. Federated Learning

FL [14] is an emerging distributed paradigm in which

multiple clients cooperatively train a neural network without

revealing their local data. Recently, many solutions [14], [25],

[26] have been presented to solve FL problems, among which

the most known and basic solution is federated averaging

(FedAvg) [14], which aggregates local models by calculating

the weighted average of their updated parameters.

However, FL has encountered various challenges [27],

among which the non-IID issue is the core problem that

makes the local model aggregation process harder and leads

to performance degradation. Numerous FL algorithms have

been presented to solve the non-IID problem, e.g., stochastic

controlled averaging for FL (SCAFFOLD) [17]; FedProx

[18]; model-contrasted FL (MOON) [28], which attempts to

increase the effect of local training on heterogeneous data by

minimizing the dissimilarity between the global model and

local models; FedMA [22] and FedNova [29], which improve

the aggregation stage by utilizing Bayesian nonparametric

methods and local update normalization, respectively; CCVR

[30], which calibrates the constitutive classiﬁers using virtual

representations to eliminate the global model bias caused by

local distribution variance; and FedEM [31], which introduces

expectation maximization to make the learned model robust to

data heterogeneity. Though these methods have been proposed

based on FedAvg by trying to learn a more robust global

model, they focus on learning a shared global model, which

degrades their performance when the data distributions heavily

vary across clients.

Recently, personalized FL (PFL) [19], [20], [32], [33] has

been proposed; this approach aims to process heterogeneous

local data with personalized models. Many of the existing PFL

methods were proposed to solve the distributed meta-learning

problem [33]–[36]. Among the methods that target normal PL

tasks, multitask learning [37] is applied to learn personalized

local models by treating each client as a learning task; model

mixing [38], [39] achieves the same goal by allowing clients to

learn a mixture of the global model and local models. Notably,

the authors in [40] presented a new local model structure that

comprises a global feature encoder and a personalized output

layer. By contrast, LG-FedAvg provides clients with a local

feature encoder and a global output layer.

However, very few of the mentioned FL methods are able to

handle data uncertainties, except for that of [22], [29], which

adopts Bayesian treatment, and that of [20], which adopts a

Gaussian process. In addition, building Bayesian posteriors

and Gaussian kernels is time-consuming. In contrast, our study

uses FNNs as local models, which are viewed as assemblies

of fuzzy rules. Thus, taking rules as basic functional units, we

break down the task of learning a global model into learning

global fuzzy rules, each of which can independently investigate

its local sample space and contribute to the local training

process.

III. FEDERATED FUZZY NEURAL NETWORK

In this section, we describe the general structure of our

FedFNN. As depicted in Fig. 2, the FedFNN includes one

server and several local clients. The server is responsible for

communication with local clients and maintaining a group of

global rules by aggregating the uploaded local rules. Local

clients download the global rules as local rules for constructing

their FNNs, which are then updated via training on their own

data. Due to the concerns of data privacy, each local agent

learns from its own data without accessing the data of other

agents and communicates, while the server communicates

with all local agents and aggregates the locally learned rules

according to their activation status. An overview of a local

FNN is shown in Fig. 3.

To mimic the selective activation of genes, the rules in the

local clients are activated selectively to make the FedFNN

personalized and properly adapted to local non-IID data. Thus,

we introduce an activation vector containing the rule status sq

of each client. Accordingly, the global server can help local

clients activate useful rules and deactivate useless or harmful

rules based on their own local data. For example, if sq

k= 0,

the server will deactivate the k-th rule for the FNN on the q-th

client; otherwise, the corresponding rule will be activated and

involved in the operations of the q-th client.

In the FedFNN, we adopt the fuzzy logic presented

in the ﬁrst-order T-S fuzzy system [2]. Suppose that our

FedFNN holds Qclients; then, the dataset owned by the

q-th client can be denoted as Dq:= {xq

i, yq

i}Nq

i=1, where

i= [xq

i1, xq

i2,· · · , xq

iD]Tand yq

i∈RCare the i-th sample

and its one-hot vector label, respectively, and Cand Nqdenote

the category number and the local dataset size. Then, the k-th

fuzzy rule of the local FNN on the q-th client can be described

Rule k: If xq

i1is Aq

k1and · · · and xq

iD is Aq

Then, yq

i=gq(xi;θk)

where Aq

kj is the j-th fuzzy set of rule kon Pqand gq(xi;θk)

is the corresponding consequent rule built by a fully connected

layer parameterized with θk. Many types of membership

functions can be employed to describe the fuzzy set Aq

kj , such

as singleton, triangular, trapezoidal, and Gaussian ones [41].

Here we choose the Gaussian membership function for three

reasons: 1) Gaussian membership function is differentiable,

which is more suitable for end-to-end learning models; 2)

Gaussian membership function is proved to be effective in

approximating nonlinear functions on a compact set [42]; 3)

Gaussian membership functions are able to represent data

features using different Gaussian distributions, which can

capture data heterogeneity and handle data uncertainty using

several distributions.

Generally, the Gaussian membership function

ϕq(xq

ij ;mq

kj , σq

kj )of a fuzzy set Aq

kj has the form

ϕq

k(xq

ij ;mq

kj , σq

kj ) = exp

− xq

ij −mq

σq

kj !2

,(1)

where mkj and σkj are the mean and standard devia-

tion of the Gaussian membership function, respectively. Let

ϕq

k(xq

i;mq

k, σq

k)collect the membership value of the k-th rule

on client Pqin vector form. The output of the antecedent

layer (also known as the ﬁring strength) considering the rule

activation status sq

kcan be written as:

k(xq

i;mq

k, σq

k, sq

k) = sq

kexp[||ϕq

k(xq

i;mq

k, σq

k)||2]

k=1 sq

kexp[||ϕq

k(xq

i;mq

k, σq

k)||2],(2)

The consequent output of the k-th rule is denoted as gq(xq

i;θq

and can be calculated by:

k(xq

i;θk) = [1; xq

i]Tθq

k,(3)

where θq

k∈R(D+1)×Cdenotes the consequent parameters.

Thus, by considering rule statuses, the FedFNN is able to

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

1FederatedFuzzyNeuralNetworkwithEvolutionaryRuleLearningLeijieZhang,YeShi,Member,IEEE,Yu-ChengChangandChin-TengLin,Fellow,IEEEAbstractDistributedfuzzyneuralnetworks(DFNNs)haveattractedincreasingattentionrecentlyduetotheirlearningabilitiesinhandlingdatauncertaintiesindistributedscenarios.However,i...

展开>> 收起<<

1 Federated Fuzzy Neural Network with Evolutionary Rule Learning.pdf

共12页,预览3页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

1 Federated Fuzzy Neural Network with Evolutionary Rule Learning

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: