PREVIEW 1 Towards Consistency and Complementarity A Multiview Graph Information Bottleneck Approach

2025-04-24 0 0 1.27MB 11 页 10玖币

侵权投诉

PREVIEW 1

Towards Consistency and Complementarity: A

Multiview Graph Information Bottleneck Approach

Xiaolong Fan, Maoguo Gong, Senior Member, IEEE, Yue Wu, Member, IEEE,

Mingyang Zhang Member, IEEE, Hao Li, and Xiangming Jiang

Abstract—The empirical studies of Graph Neural Networks

(GNNs) broadly take the original node feature and adjacency

relationship as singleview input, ignoring the rich information

of multiple graph views. To circumvent this issue, the mul-

tiview graph analysis framework has been developed to fuse

graph information across views. How to model and integrate

shared (i.e. consistency) and view-speciﬁc (i.e. complementarity)

information is a key issue in multiview graph analysis. In

this paper, we propose a novel Multiview Variational Graph

Information Bottleneck (MVGIB) principle to maximize the

agreement for common representations and the disagreement for

view-speciﬁc representations. Under this principle, we formulate

the common and view-speciﬁc information bottleneck objectives

across multiviews by using constraints from mutual information.

However, these objectives are hard to directly optimize since

the mutual information is computationally intractable. To tackle

this challenge, we derive variational lower and upper bounds of

mutual information terms, and then instead optimize variational

bounds to ﬁnd the approximate solutions for the information

objectives. Extensive experiments on graph benchmark datasets

demonstrate the superior effectiveness of the proposed method.

Index Terms—Graph data mining, multiview graph represen-

tation learning, graph neural networks, deep neural networks.

I. INTRODUCTION

AS a ubiquitous data structure, graph is capable of mod-

eling real-world systems in numerous domains, ranging

from social network analysis [1, 2], natural language process-

ing [3, 4], computer vision [5, 6], and other domains. In recent

years, as a powerful tool for learning and analyzing graph

data, Graph Neural Networks (GNNs) [7, 8, 9, 10, 11] have

received tremendous research attention and have been widely

employed for graph analysis tasks. The common graph neural

networks broadly follow the message passing framework [9]

which involves a message passing phase and a readout phase.

The message passing ﬁrst iteratively aggregates the neighbor

representations of each node to generate new node represen-

tations, and then the readout phase capture the global graph

information from node representation space to generate the

graph representation. The success of GNNs can be attributed

to their ability to simultaneously exploit the rich information

X. Fan, M. Gong (corresponding author), M. Zhang, H. Li, and

X. Jiang are with the School of Electronic Engineering, Key Labo-

ratory of Intelligent Perception and Image Understanding, Ministry of

Education, Xidian University, Xi’an, Shaanxi province, China, 710071.

(e-mail: xiaolongfan@outlook.com; gong@ieee.org; haoli@xidian.edu.cn;

myzhang@xidian.edu.cn; xmjiang@xidian.edu.cn)

Y. Wu is with the School of Computer Science and Technol-

ogy, Xidian University, Xi’an, Shaanxi province, China, 710071. (e-mail:

ywu@xidian.edu.cn)

inherent in the singleview graph topology structure and input

node attributes.

However, for real-world applications, graph data are often

manifested as multiple types of sources or different topology

subsets, which can be naturally organized as multiview graph

data. For example, a multiplex network [12, 13] contains mul-

tiple systems of the same set of nodes, and there exists various

types of relationships among nodes, which can be seen as

different topology subsets. Furthermore, existing approaches

may confront challenges, such as difﬁculty in obtaining labeled

data and generalization bias under the supervised learning

setting [14]. One way to alleviate these issues is unsupervised

representation learning on graph. Therefore, how to integrate

different graph views into low-dimensional representations

for downstream tasks in an unsupervised manner becomes a

fundamental problem for graph representation learning.

Recently, several multiview graph representation learning

approaches have been proposed to effectively explore the

multiview graph data. For instance, Adaptive Multi-channel

Graph Convolutional Networks (AM-GCN) [15] show that

performing graph convolutional operation over both topology

and feature views can improve the performance of node repre-

sentation learning, where the feature view can be generated by

distance based K-Nearest Neighbor algorithm. Note that AM-

GCN works in the supervised learning manner and therefore

may suffer from the challenge of annotating graphs. Con-

trastive Multiview Graph Representation Learning (MVGRL)

[16] presents an unsupervised mutual information maximiza-

tion approach for learning node and graph representations by

contrasting graph views, where the additional graph view is

generated by structural augmentations such as Personalized

PageRank diffusion and heat diffusion strategies. Speciﬁcally,

MVGRL maximizes the mutual information between two

views by contrasting node representations from one view with

graph representation from the other view, but neglects to ex-

plicitly distinguish the common and view-speciﬁc information

across multiviews. Recent work [17] has shown that explicitly

modeling common and view-speciﬁc information can improve

the performance of multiview representation model.

In this paper, we focus on the in-depth analysis of multiview

graph representation learning and aim to answer the question

that how to extract common and view-speciﬁc information

in an unsupervised manner for graph representation learning.

Inspired by the idea of mutual information constraints in

information bottlenecks [18, 19, 20, 21], we ﬁrst formulate

the information bottleneck objective to encourage that the

latent representation contains as much information as possible

arXiv:2210.05676v1 [cs.LG] 11 Oct 2022

PREVIEW 2

about the corresponding input view and as less information as

possible about the other view to explore the complementary

information across multiple graph views. Then, to further

explore the complex associations among different graph views,

we also formulate an information bottleneck objective to retain

as much information as possible about the corresponding input

view and the other view simultaneously. Note that, different

from previous information bottleneck methods [19, 20, 22]

that discard superﬂuous information for the given task, the

proposed method focuses on using mutual information to

constrain the latent graph representations to model consistency

and complementarity across graph multiviews.

However, the mutual information terms are challenging

to calculate in practice, since they require computing the

intractable posterior distribution. To tackle this challenge, we

follow the variational inference methods [19, 23] and derive

the variational lower bounds of mutual information terms that

need to be maximized. For the mutual information terms that

need to be minimized, we adopt Contrastive Log-ratio Upper

Bound (CLUB) [24] as the upper bound approximation. To

enable the computation of CLUB, a key challenge is to model

the intractable conditional distribution. Here, we employ the

classical graph neural network encoder to variationally approx-

imate conditional distribution. By combining the derived lower

and upper bounds, we can easily optimize the overall objective

by using the stochastic gradient descent algorithm.

Extensive validation experiments on seven graph benchmark

datasets demonstrate the superior effectiveness of the proposed

method. Speciﬁcally, we evaluate the proposed method on

the graph classiﬁcation and graph clustering tasks, and also

perform the ablation study to investigate the performance of

certain components in the proposed method to understand the

contribution to the overall system.

To summarize, we outline the main contributions in this

paper as below:

1. We propose a novel Multiview Variational Graph Infor-

mation Bottleneck (MVGIB) principle to maximize the

agreement for common representations and the disagree-

ment for view-speciﬁc representations.

2. We present the variational lower and upper bounds of

the mutual information terms in MVGIB by exploiting

the variational approximation method.

3. Extensive validation experiments and ablation studies on

seven graph benchmark datasets demonstrate the superior

effectiveness of the proposed method.

The rest of this paper is organized as follows. In section II,

we make a brief review of the related works of graph neural

networks and information bottleneck. Section III describes

the preliminaries. Section IV gives the proposed method by

introducing representation model towards complementarity

and consistency, and then discusses the optimize and inference.

Section V evaluates the proposed method with extensive ex-

periments on graph representation benchmark datasets. Finally,

section VI concludes this paper and discusses the future work.

II. RELATED WORK

A. Graph Neural Network

Graph Neural Networks (GNNs) aim to model the non-

Euclidean graph data structure and have been demonstrated to

achieve state-of-the-art performance on graph analysis tasks.

As a uniﬁed framework for graph neural networks, graph

message passing neural network [9, 25] generalizes several

existing representative graph neural networks, such as Graph

Convolutional Networks (GCN) [10], Graph Isomorphism

Network (GIN) [11], Deep Hierarchical Layer Aggregation

(DHLA) [26], and others. In recent years, several label-

free contrastive self-supervised graph representation learning

methods have been developed, such as Deep Graph Info-

max (DGI) [27], Contrastive Multiview Graph Representation

Learning (MVGRL) [16], InfoGraph [28], and have achieved

competitive performance, even exceeding supervised graph

representation learning. In this paper, we use the graph iso-

morphism network as graph encoder to generate the graph

representation.

B. Information Bottleneck

The information bottleneck methods for deep neural net-

works [19] are developed to generate a compact but informa-

tive representation. Under this setting, if latent representation

discards information from the input which is not useful for a

given task, it can increase robustness for the downstream tasks.

In recent years, several information bottleneck based graph

representation learning methods have been proposed, such as

Graph Information Bottleneck (GIB) [20], Subgraph Informa-

tion Bottleneck (SIB) [22], and Variational Information Bot-

tleneck for Graph Structure Learning (VIB-GSL) [21]. In this

paper, we follow the idea of mutual information constraints

in information bottlenecks and develop the multiview graph

information bottleneck for graph representation learning.

III. PRELIMINARIES

This section ﬁrst introduces notations and then discusses the

recent variational information bottleneck method.

A. Notations

We ﬁrst introduce the notations throughout this paper. Let

G= (V,E)be a graph with Vand Edenoting the node

set and edge set, respectively. The nodes are described by

the adjacency matrix A∈Rn×nwith the number of nodes

nwhere Aij = 1 if the edge between i-th node and j-th

node exists and else Aij = 0, and the node feature matrix

X∈Rn×dwith the number of feature dper node. Under the

multiview graph input setting, suppose the number of graph

views is r, the graph contains a set of adjacency matrices,

i.e., A={A1, A2, ..., Ar}, and a set of node features, i.e.,

X={X1, X2, ..., Xr}. Following the frequent notations in

information theory, we use H(x) = −Ep(x)log p(x)denoting

Shannon entropy, H(p, q) = −Eq(x)log p(x)denoting cross

entropy, I(x;y) = Ep(x,y)log p(x,y)

p(x)p(y)denoting mutual infor-

mation, and DKL(pkq) = Ep(x)log p(x)

q(x)denoting Kullback-

Leiber divergence.

PREVIEW 3

Encoder

InputInput

1 1 2 1

max ( ; ) ( ; )I h I h



−

()|q h

()|q c

()|q h

()|qh

()|ph

()|pc

()|ph

View

()|qh

2 2 1 2

max ( ; ) ( ; )I h I h



−

1 1 2 1

max ( ; ) ( ; )I c I c



2 2 1 2

max ( ; ) ( ; )I c I c



Fig. 1. The overall framework of the proposed approach. Here, G1with (A1, X)and G2with (A2, X)to be two input graph views. To extract consistent

relationships between multiview graphs, the proposed approach maximizes the mutual information between input graph and the common representation, while

maximizing the mutual information between G2and the common representation c1. To extract complementarity relationships between multiview graphs, the

proposed approach maximizes the mutual information between G1and h1, while minimizing the mutual information between G2and h1.

B. Information Bottleneck

The Information Bottleneck (IB) method [18] is introduced

as an information theoretic principle for extracting compact

but informative representations. Given the input data x, rep-

resentation z, and predict label y, the IB method maximizes

the mutual information between representation zand predicted

label y, while constraining the mutual information between

representation zand input data x, i.e.,

max I(z;y)s.t. I(x;z)≤Ic(1)

where Icdenotes the information constraint. By introducing

Lagrange multiplier α, Equation 1 can be transformed as

max I(z;y)−αI(x;z).(2)

Intuitively, the ﬁrst term of Equation 2 encourages zto be

predictive of yand the second term encourages zto forget

x, resulting in the minimal but sufﬁcient encoding zfor

predicting y. However, due to the computational challenge

of mutual information, the IB objective is hard to directly

optimized. To compensate for this, recent work [19] proposes

a feasible way to approximate IB objective by incorporating

variational inference in form of

Ep(x,y)Ep(z|x)log q(y|z)−αEp(x)Ep(z|x)log p(z|x)

r(z)(3)

where q(y|z)and r(z)are the variational approximations to

p(y|z)and marginal distribution p(z), respectively.

IV. PROPOSED APPROACH

As a motivating example, suppose G1with (A1, X1)and G2

with (A2, X2)to be two input graph views. Without loss of

generality, we assume that the different graph views differ only

in their adjacency relationship, i.e., X1=X2. To extract con-

sistent relationship and complementary information between

multiview graph data, we propose Multiview Variational Graph

Information Bottleneck (MVGIB) by extending the informa-

tion bottleneck principle to the unsupervised multiview graph

representation setting. The overall framework of MVGIB is

shown in Figure 1.

A. Towards Complementarity

Given two input graph views G1and G2of graph G,

complementarity principle aims to generate the view-speciﬁc

representation h1and h2for the given graph views G1and

G2, respectively. Taking fenc :G1→h1as an example, we

expect to maximize the mutual information between G1and

h1, while minimizing the mutual information between G2and

h1, i.e.,

max I(G1;h1)−δI(G2;h1)(4)

where δis the trade-off parameter to balance I(G1;h1)and

I(G2;h1). Analogously, we can formulate the information

bottleneck objective for fenc :G2→h2, i.e.,

max I(G2;h2)−ηI(G1;h2)(5)

where ηis the trade-off parameter. Intuitively, maximizing

Equation 4 and Equation 5 encourage representation h1and

h2to maximally express the input graph views generated

from, while maximally compressing the other graph views, i.e.,

towards complementarity. However, the mutual information

is computationally intractable since it requires computing the

intractable posterior distribution, resulting in these objectives

being hard to optimize. To tackle this challenge, we introduce

the variational inference approach [19] to derive the variational

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

PREVIEW1TowardsConsistencyandComplementarity:AMultiviewGraphInformationBottleneckApproachXiaolongFan,MaoguoGong,SeniorMember,IEEE,YueWu,Member,IEEE,MingyangZhangMember,IEEE,HaoLi,andXiangmingJiangAbstractTheempiricalstudiesofGraphNeuralNetworks(GNNs)broadlytaketheoriginalnodefeatureandadjacencyrela...

展开>> 收起<<

PREVIEW 1 Towards Consistency and Complementarity A Multiview Graph Information Bottleneck Approach.pdf

共11页,预览3页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

PREVIEW 1 Towards Consistency and Complementarity A Multiview Graph Information Bottleneck Approach

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: