FedGRec Federated Graph Recommender System with Lazy Update of Latent Embeddings Junyi Li Heng Huang

2025-05-06 0 0 686.53KB 20 页 10玖币

侵权投诉

FedGRec: Federated Graph Recommender System with Lazy

Update of Latent Embeddings

Junyi Li∗

, Heng Huang †

Abstract

Recommender systems are widely used in industry to improve user experience. Despite great

success, they have recently been criticized for collecting private user data. Federated Learning

(FL) is a new paradigm for learning on distributed data without direct data sharing. Therefore,

Federated Recommender (FedRec) systems are proposed to mitigate privacy concerns to non-

distributed recommender systems. However, FedRec systems have a performance gap to its

non-distributed counterpart. The main reason is that local clients have an incomplete user-item

interaction graph, thus FedRec systems cannot utilize indirect user-item interactions well. In

this paper, we propose the Federated Graph Recommender System (FedGRec) to mitigate this

gap. Our FedGRec system can eﬀectively exploit the indirect user-item interactions. More

precisely, in our system, users and the server explicitly store latent embeddings for users and

items, where the latent embeddings summarize diﬀerent orders of indirect user-item interactions

and are used as a proxy of missing interaction graph during local training. We perform extensive

empirical evaluations to verify the eﬃcacy of using latent embeddings as a proxy of missing

interaction graph; the experimental results show superior performance of our system compared

to various baselines. A short version of the paper is presented in the FL-NeurIPS’22 workshop.

1 Introduction

Recommender systems play an essential role in reducing information overload in the current era of

information explosion. A recommender system predicts a small set of candidates in which a user

may be interested from a large number of items. Collaborative Filtering (CF) [

] is one of the

most successful approaches to making recommendations. CF is based on the idea that users with a

similar interaction history tend to share interests in items. Naturally, CF is highly dependent on

collecting user behavior data. Gathering such information undermines the privacy of the user. To

alleviate this challenge, researchers exploited the idea of Federated Learning (FL) and developed

Federated Recommender Systems (FedRec). In a FedRec system, users keep its data locally and

only share the model with the server. FedRec systems mitigate privacy concerns, but still have a

performance gap with non-distributed recommender systems [

]. In a FedRec system, a user

performs local training with its own interaction data and cannot access the data of other users. With

this incomplete interaction graph, the learned model generally cannot capture indirect user-item

∗

Department of Electrical and Computer Engineering, University of Pittsburgh, Pittsburgh, USA. Email: juny-

ili.ai@gmail.com

†

Department of Electrical and Computer Engineering, University of Pittsburgh, Pittsburgh, USA. Email:

henghuanghh@gmail.com

arXiv:2210.13686v1 [cs.LG] 25 Oct 2022

interactions well. In contrast, non-distributed recommender systems [

] have access to the

whole interaction graph and can capture such indirect interaction with various techniques such as

graph embedding. Therefore, it is essential to develop a technique to mitigate the bias caused by the

incomplete local interaction graph so that FedRec systems can better capture indirect interaction.

In this paper, we take one step forward and propose the Federated Graph Recommender System

(FedGRec), which can take advantage of indirect interaction eﬃciently.

Recently proposed federated recommender systems either abandon taking advantage of indirect

interactions [

], or rely on complicated cryptography techniques [

] to access data from other

users. In particular, [

] proposed FedGNN, which adapted graph neural network (GNN)-based

recommender systems to the FL setting. In FedGNN, a user can request embeddings of its neighbors

using encryption techniques. However, this is achieved at high cost. First, it assumes the existence

of a trusted third party; second, this request needs a large amount of computing power for expensive

Homomorphic Encryption [

] operations. Furthermore, even at this high cost, FedGNN only

exploits ﬁrst-order user-item interactions, i.e. the direct neighbors of a user, while in non-distributed

GNN-based models, second-order interactions (users that interact with same items) and even higher-

order indirect interactions are exploited. To alleviate the limitations of FedGNN and fully exploit

indirect user-item interactions, we propose our FedGRec system, and the key feature of our system

is to explicitly store latent embeddings of users and items. The concept of latent embedding of

a user/item stems from the non-distributed GNN-based recommender system. Non-distributed

GNN-based recommender systems [

] usually have an embedding layer and multiple embedding

propagation layers. The embedding layer encodes users and items to obtain a vector representation

of them. Embedding propagation layers reﬁne user/item embeddings sequentially. Each embedding

propagation layer linearly combines neighbor embeddings of the last layer. Finally, embedding and

output of embedding propagation layers are combined (such as average) as the ﬁnal representation

of a user/item. In fact, the output of the embedding propagation layers encodes diﬀerent orders of

user-item interactions. For ease of discussion, we denote them as latent embeddings. Our system is

built on using latent embeddings as a proxy of the miss interaction graph during local training.

In our FedGRec system, users store their embeddings, and the server stores embeddings for all

items as normal federated recommender systems. In addition, users also keep their latent embeddings,

and the server has latent item embeddings. The training process includes two parts: the optimization

of user/item embeddings and the optimization of latent user/item embeddings. First, assume that

latent embeddings encode indirect user-item interactions; then users update user/item embeddings

by treating latent embeddings as constants for multiple training steps locally. Next, the latent

user/item embeddings are updated only in the server synchronization step based on the current

user/item embeddings. Note that latent user/item embeddings are ﬁxed during users’ local training,

which diﬀers from that in the non-distributed setting. In the non-distributed setting, embedding

propagation performed in real time, i.e. the latent embeddings encode up-to-date indirect user-item

interaction information. In contrast, we use ﬁxed latent user/item embeddings during local training

due to communication constraints. This makes the information encoded in these latent embeddings

stale. However, we empirically show that the stale latent embeddings are still useful in capturing

indirect interaction. We verify their eﬃcacy through extensive empirical studies. Finally, our system

preserves the privacy of users. During the whole training process, only the item embeddings are

transferred between users and the server, in addition, we take advantage of the secure aggregation

technique [

]. Secure aggregation is a privacy-preserving technique that allows the server to get the

sum of user updates without knowing individual details. Finally, we summarize the contributions of

our paper as follows:

We propose a novel Federated Graph Recommender System (FedGRec) that eﬀectively uses

the indirect user-item interactions;

We design and store latent embeddings to encode the indirect user-item interaction. Latent

embeddings are a proxy for absent neighbors during local training. We also propose a lazy

way to update these latent embeddings;

Our new system is evaluated via extensive experimental studies, and results show the superior

performance of our system compared to various baselines.

Organization:

The remainder of this paper is organized as follows: In Section 2, we discuss some

related works; In Section 3, we introduce the preliminaries; In Section 4, we formally introduce our

new Federated Graph Recommender System (FedGRec); In Section 5, we perform experiments to

verify the eﬀectiveness of our system; In Section 6, we conclude and summarize the paper.

2 Related Work

The study of Recommender Systems dates back to the 1990s [

]. Most recommender systems aim

to develop representations for users and items. A classic approach is based on matrix factorization:

we associate the embedding vectors with the one hot user(item) ID [

], and then take inner

products between the embeddings of the user and the item to match the ratings. Later, researchers

pooled interacted item embeddings to augment user representation, such as FISM [

], SVD++ [

and approaches based on attention mechanisms [

]. Furthermore, the graph neural network is also

used to generate representations [

]. Unlike learning representations, another line of work

focuses on modeling interactions. There are two limitations of the inner product approach [

]. First,

it does not satisfy the triangle inequality, so the propagation of similarity is not good; second, the

linear nature of the inner product constrains its ability to model complicated interactions. Therefore,

the researchers propose to use other metrics such as

distance [

], and deep neural networks [

Although most of the recommender systems only use user-item interaction data, some additional

data can boost model performance if available. These models can be divided into two categories:

Content-based models [

] and context-based models [

]. Content-based models

use additional user features (items). In contrast, context-based models use auxiliary information

from the interaction. [56] and [58] provide a detailed review of the recommender systems.

Federated learning [

] is a promising distributed data mining paradigm in which a server

coordinates a set of clients to learn a model. A widely used algorithm for FL is the FedAvg [

]

algorithm, where clients receive the up-to-date model from the server at the start of each epoch

and then train the model locally for several iterations and upload the new model back to the server.

There are three main challenges in FL: data heterogeneity, high communication cost, and user privacy.

Some variants of FedAvg are proposed to address heterogeneity [

]. To reduce

the cost of communication, various compression techniques are applied, such as quantization [

sparsiﬁcation [

] and sketching [

]. Regarding user privacy, although the server cannot

see the data directly, it is possible to recover the data based on model updates with a model

inversion attack [

]. Therefore, some cryptography techniques are applied, such as homomorphic

encryption [

], diﬀerential privacy [

] and multiparty secure computation [

]etc.. A simple

but eﬀective technique to defend a malicious server is the secure aggregation [

] technique.

With this technique, the server aggregates updates from clients without knowing the input of each

client. There are works focus on other aspects such as the fairness [9, 27] and data corruption [29]

etc.

More recently, Recommender systems have been considered in the federated learning setting

(FedRec) [

]. In particular, [

] applied the matrix factorization

approach to FL. It used the Alternating Least Squares (ALS) algorithm. At each epoch, each

client computes the optimal user embedding, then calculates the item embedding gradients, and

uploads them to the server. Finally, the server aggregates the gradients from all clients to update the

embeddings of the items. The above approach directly transfers the gradients of the item embeddings,

which has the risk of leaking private ratings; Some privacy preservation techniques [

] are

exploited to mitigate this risk [

]. In addition to classic matrix factorization-based approaches,

deep neural collaborative ﬁltering techniques are also adapted to the FL setting [

]. In [

], the

authors proposed a two-stage training framework. In the ﬁrst stage, item embeddings are learned

with self-supervised learning. Then, in the second stage, a federated neural recommender system

is learned with the help of diﬀerential privacy. Graph-based recommender systems have gained

state-of-the-art performance in the non-distributed setting. However, it is not trivial to adapt them

to the FL setting. In FL, each client only has a subgraph. Recent work [

] proposed to obtain

the embeddings of neighboring users using the homomorphic encryption technique. Our paper

also considers graph-based FedRec. However, we do not require the time-consuming homomorphic

encryption technique, but we use the fact that only aggregated representations are needed in the

training. The survey paper [

] provides a good overview of the problem of federated recommender

systems.

3 Preliminaries

Graph Recommender Systems.

By exploiting indirect user-item interactions, graph-based rec-

ommender systems have gained state-of-the-art recommendation performance in the non-distributed

setting. LightGCN [

] is a recently proposed graph recommendation system. It simpliﬁes the

classical graph neural network by removing the transformation matrix and the nonlinear activation

function. The system includes two types of layer: the input embedding layer and the embed-

ding propagation layer. More precisely, there is one embedding layer which initializes (item) user

embeddings, and several embedding propagation layers which reﬁne embeddings with high-order

user-item connectivity relations. Suppose that there are

embedding propagation layers; then the

kth (k∈[0 ...,K −1]) embedding propagation layer performs the following rule:

ek+1

u=X

t∈Nu

p|Nt|p|Nu|;ek+1

t=X

u∈Nt

p|Nt|p|Nu|(1)

is the set of connected items of the user

and

is the set of connected users of the item

The ﬁnal representation (embedding) of a user/item is a weighted average of the output of these

embedding layers, i.e.:

eu=

k=0

αkek

u;et=

k=0

αkek

t(2)

where

αk

are weights. Then we calculate the inner product between the user and the item

representation as a measure of their aﬃnity:

ˆyut

heu, eti

. During training, we optimize the

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

FedGRec:FederatedGraphRecommenderSystemwithLazyUpdateofLatentEmbeddingsJunyiLi*,HengHuangAbstractRecommendersystemsarewidelyusedinindustrytoimproveuserexperience.Despitegreatsuccess,theyhaverecentlybeencriticizedforcollectingprivateuserdata.FederatedLearning(FL)isanewparadigmforlearningondistribute...

展开>> 收起<<

FedGRec Federated Graph Recommender System with Lazy Update of Latent Embeddings Junyi Li Heng Huang.pdf

共20页,预览4页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

FedGRec Federated Graph Recommender System with Lazy Update of Latent Embeddings Junyi Li Heng Huang

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: