GLIF A Uniﬁed Gated Leaky Integrate-and-Fire Neuron for Spiking Neural Networks Xingting Yao12 Fanrong Li12 Zitao Mo1 Jian Cheng13

2025-05-06 0 0 2.2MB 16 页 10玖币

侵权投诉

GLIF: A Uniﬁed Gated Leaky Integrate-and-Fire

Neuron for Spiking Neural Networks

Xingting Yao1,2, Fanrong Li1,2, Zitao Mo1, Jian Cheng1,3∗

1Institute of Automation, Chinese Academy of Sciences

2School of Future Technology, University of Chinese Academy of Sciences

3CAS Center for Excellence in Brain Science and Intelligence Technology

{yaoxingting2020, lifanrong2017, mozitao2017, jian.cheng}@ia.ac.cn,

Abstract

Spiking Neural Networks (SNNs) have been studied over decades to incorpo-

rate their biological plausibility and leverage their promising energy efﬁciency.

Throughout existing SNNs, the leaky integrate-and-ﬁre (LIF) model is commonly

adopted to formulate the spiking neuron and evolves into numerous variants with

different biological features. However, most LIF-based neurons support only single

biological feature in different neuronal behaviors, limiting their expressiveness

and neuronal dynamic diversity. In this paper, we propose GLIF, a uniﬁed spiking

neuron, to fuse different bio-features in different neuronal behaviors, enlarging

the representation space of spiking neurons. In GLIF, gating factors, which are

exploited to determine the proportion of the fused bio-features, are learnable during

training. Combining all learnable membrane-related parameters, our method can

make spiking neurons different and constantly changing, thus increasing the hetero-

geneity and adaptivity of spiking neurons. Extensive experiments on a variety of

datasets demonstrate that our method obtains superior performance compared with

other SNNs by simply changing their neuronal formulations to GLIF. In particular,

we train a spiking ResNet-19 with GLIF and achieve

77.35%

top-1 accuracy with

six time steps on CIFAR-100, which has advanced the state-of-the-art. Codes are

available at https://github.com/Ikarosy/Gated-LIF.

1 Introduction

Spiking neural network (SNN) is considered the third generation of the neural network, and the

bionic modeling of its activation unit is called the spiking neuron, which brings about the spike-based

communication between layers [

]. Due to the high sparsity and multiplication-free operation

in information processing, such spike-based communication can improve energy efﬁciency [

Moreover, spiking neuron can effectively capture temporal information and has good performance in

processing neuromorphic data [5] and anti-noise [6]. Together with artiﬁcial neural network (ANN)

colleagues, SNN further shows great potential for general intelligence [

]. Thus, SNN has aroused

heated research interests recently.

The commonly used spiking neuron in different SNNs [

] is the leaky integrate-

and-ﬁre (LIF) model, named vanilla LIF. As the name suggests, it can implement three different

neuronal behaviors: membrane potential leakage, integration accumulation, and spike initiation. To

better model neurons, prior works [

] have improved the vanilla LIF from the above

three aspects, and proposed many LIF variants with different biological features. For all these vanilla

and variant LIF models (referred to as simplex LIF), any of the three neuronal behaviors supports

∗Corresponding author.

36th Conference on Neural Information Processing Systems (NeurIPS 2022).

arXiv:2210.13768v4 [cs.NE] 13 Feb 2023

Rethinking

τexp

St−1

Ut−1

τlin

Vre

(a)

(b)

(c)

σ(x)

−

𝔾γ

𝔾α𝔾β

Figure 1: Illustration of the GLIF model. (a) Discrete GLIF neuron. The membrane potential

and the output spike

are updated over discrete time steps. Gating units

Gα

Gβ

, and

Gγ

fuse

dual bio-features in membrane potential leakage, integration accumulation, and spike initiation,

respectively. (b) Gating unit. (c) Gating factor.

only single biological feature. For instance, the vanilla LIF model only utilizes exponential decay

in membrane potential leakage, while [

] only supports linear decay. However, different neurons

in the cerebral cortex have different response characteristics [

].This naturally raises an issue:

could the higher neuronal dynamic diversity of spiking neurons help SNNs do better? In this paper,

we investigate the hybrid bio-features in the LIF model to answer this question.

We propose the gated LIF model (GLIF) that fuses different bio-features in the aforementioned

three neuronal behaviors to possess more response characteristics. As illustrated in Fig. 1, GLIF

controls the fusion of different bio-features through gating units

Gα

Gβ

, and

Gγ

for those three

neuronal behaviors that are membrane potential leakage, integration accumulation, and spike initiation,

respectively. In each gating unit, a gating factor is computed from a Sigmoid function

σ(x)

over

a learnable gating parameter

to determine the proportion of each bio-feature, thus guiding the

fusion of different bio-features. Therefore, GLIF can simultaneously contain different bio-features,

possessing more response characteristics. In addition, when the gating factor is

, GLIF can also

support single bio-feature. As a result, GLIF can cover other different LIF models and be viewed as a

super spiking neuron, greatly enlarging the representation space of spiking neurons.

Furthermore, we introduce the channel-wise parametric method to GLIF. This method makes all

membrane-related parameters in GLIF learnable and shares the same GLIF parameters channel-

wisely in SNNs. Combining with learnable gating factors in GLIF, on the one hand, this method

makes different channels in SNNs have completely different spiking neurons, leveraging the larger

representation space of GLIF neurons to increase the neuronal dynamic diversity of spiking neurons.

Meanwhile, the heterogeneity of spiking neurons and the expressive ability of SNNs are also increased.

On the other hand, the spike neurons in SNNs are constantly changing during training, which is

similar to the neuronal maturation during development [

], thus the adaptivity of spiking

neurons being enhanced.

Our contributions are as follows:

•

We propose the GLIF model, a uniﬁed formulation of the spiking neuron that fuses multiple

biological features through gating units, enlarging the representation space of spiking

neurons.

• We exploit the channel-wise parametric method to leverage the larger representation space

of GLIF, thus increasing the neuronal dynamic diversity in SNNs.

•

Our experiments demonstrate the effectiveness of the GLIF model on both static datasets and

the neuromorphic dataset. For example, our model can achieve the state-of-the-art 75.48%

top-1 accuracy on CIFAR-10 with only two time steps.

2 Related Work

Variant LIF.

Variant LIF models with different biological features are proposed in previous arts,

and those models differ in one or more of the three different neuronal behaviors: membrane potential

leakage, integration accumulation, and spike initiation. Firstly, membrane potential leakage represents

the potential decay mechanism. Exponential decay is ﬁrst exploited in the vanilla LIF model [

], and

it brings neuronal dynamics with stable convergence, shrinking the potential to the resting potential

steadily. But, exponential decay is not efﬁcient enough in massive neural simulations due to its

multiplication operation. Therefore, linear decay replaces exponential decay in [

and brings neuronal dynamics with the ﬂexibility, making it possible for conﬁgurable spiking rates

[

]. Although the two types of decay behave differently in neuronal dynamics, both of them

are proven effective in SNNs [

]. Secondly, integration accumulation represents the potential

accumulation induced by presynaptic spikes. Commonly, spike intensities of different time steps

are considered equally high. Consequently, the discrepancies among the spikes at different time

steps are ignored. To address this issue, many prior works propose to assign different weights to the

input spikes at different time steps. These methods are implemented by manual settings [

]

or adaptive settings [

] to improve biological plausibility, SNN performance, or inference

efﬁciency. Lastly, spike initiation functions as the resetting mechanism triggered by spike ﬁring.

There are two common resetting strategies: hard reset [

] and soft reset [

]. When a spike is

triggered, the hard reset directly sets the potential to the resting potential, while the soft reset subtracts

the potential by a conﬁgurable value. Therefore, hard reset is more stable as a strict reset mechanism,

while soft reset is more ﬂexible, making conﬁgurable spiking rates feasible [

]. Compared with the

LIF models above, our GLIF incorporates different neuronal behavioral mechanisms to increase the

representation space of LIF models and improve the performance of SNNs.

Parametric spiking neurons.

Early existing spiking neurons often require manual tuning of the

membrane-related parameters to set their neuronal dynamics, and these parameters are often shared

by all neurons, which limits the diversity of spiking neurons and the expressive ability of SNNs. In

recent years, parametric spiking neurons [

] have been proposed, which set one or more

membrane-related parameters to be learnable, and update these parameters during the training of

SNNs. [

] and [

] set the membrane leak and potential threshold learnable, while [

] and [

]

choose the membrane leak only. FS-neuron [

] is more aggressive, setting all membrane-related

parameters in the spiking neuron learnable to strictly ﬁt the neuronal dynamics to the target activation

function. Compared with the above methods, GLIF adopts the channel-wise parametric method to

fully parameterize the spiking neuron, including learnable exponential decay, linear decay, potential

threshold, resetting voltage, input conductance, and gating factors.

Supervised direct learning of SNNs.

Supervised direct learning of SNNs follows the idea of

backpropagation (BP) in ANN [

]. Bohte et al. [

] ﬁrst adopt the surrogate gradient (SG) to

empirically solve the non-differentiable term of spiking neurons by approximation with the local

derivative of potential. Based on the observation that the proper steepness of the SG curve matters

more than the shape in SNN learning [

], Li et al. [

] conduct a pre-estimation before each epoch

to ﬁnd the SG curve with the optimal smoothness. Besides, many techniques are developed to make

deep SNNs converge faster, such as NeuNorm [

], tdBN [

], SEW block [

], TET [

], MS residual

block [

], etc. In this way, the problem of end-to-end training of deep SNNs can be solved. In this

paper, we choose supervised direct learning of SNNs to estimate the effectiveness of our GLIF.

3 Methodology

3.1 Revisiting LIF models

Vanilla LIF.

The Vanilla LIF model is commonly adopted as a spiking neuron model in SNNs,

which can be discretely formulated as follows:

U(t,l)=τU(t−1,l)(1 −S(t−1,l)) + C(t,l),(1)

C(t,l)=W·S(t,l−1),(2)

S(t,l)=H(U(t,l)−Vth).(3)

Here,



denotes the element-wise multiplication.

U(t,l)

is the membrane potential vector of the

layer

at time-step

and can be updated in a discrete way through Eq.(1), where

C(t,l)

denotes the

input vector and can be obtained by the dot-product between the synaptic weight matrix Wand the

output spike vector of the previous layer

S(t,l−1)

, as described in Eq.(2). The output spike vector

S(t,l)

is given by the Heaviside step function

H(·)

in Eq.(3), indicating that a spike is ﬁred when the

membrane potential exceeds the potential threshold

Vth

. When updating

U(t,l)

in Eq.(1), the three

neuronal behaviors are considered. Speciﬁcally, the time constant

acts as the exponential decay

coefﬁcient on the term

τU(t−1,l)

to implement exponential decay,

C(t,l)

is integrated over time to

U(t,l)

, and the term

(1 −S(t−1,l))

hard resets the membrane potential to

if a spike is ﬁred at the

previous time step.

Bio-features & Primitives.

Bio-features represent the characteristics of the three critical neuronal

behavior. For the instance of vanilla LIF, its bio-features can be described as exponential decay,

uniform-coding scheme, and hard reset. And, the three bio-features qualitatively describe the neuronal

behavior of membrane potential leakage, integration accumulation, and spike initiation, respectively.

Since each behavior is actually implemented by the corresponding term in formulas, for explicitness,

we refer to those neuronal behavior-related parameters as

primitives

in this paper. In this way, bio-

features can be quantitatively described by primitives. For example, exponential decay is realized

by the term

τU(t−1,l)

in Eq.(1), so

is a primitive and determines the amplitude of exponential

decay. Thus, by choosing different primitives for each behavior, we can formulate different LIF

variants. For instance, with the linear decay primitive

τlin

and the soft reset primitive

τre

, we have:

U(t,l)=U(t−1,l)−τlin −VreS(t−1,l)+C(t,l), which formulates the spiking neuron in [15].

3.2 Gated LIF Model

Although variant LIF offers a different choice in modeling the spiking neuron, it still remains simplex,

i.e., the bio-features of the neuronal dynamic are unitary. Once formulated, a hard-reset LIF neuron

cannot perform the soft-reset operation. As a result, these simplex LIF models make the spiking

neurons in SNNs lack dynamic diversity. Thus, we propose the gated LIF (GLIF), a uniﬁed spiking

neuron, to fuse different bio-features in the three neuronal behaviors. GLIF is illustrated in Fig. 1 and

further described as follows:

U(t,l)=L(t,l)+I(t,l)+F(t,l)S(t−1,l),(4)

C(t,l)=W·S(t,l−1),(5)

S(t,l)=H(U(t,l)−Vth),(6)

L(t,l)=Gα(U(t−1,l);τlin, τexp),(7)

I(t,l)=Gβ(C(t,l);gt,1),(8)

F(t,l)=Gγ(L(t,l)

exp ;Vre,0).(9)

In Eq.(4),

L(t,l)

denotes the membrane potential vector after membrane potential leakage,

I(t,l)

denotes the vector of the incremental potential caused by integration accumulation, and

F(t,l)

denotes

the vector of the potential reduction caused by spike initiation. Obviously, the above three vectors

reﬂect the consequences of the three neuronal behaviors. In Eq.(7)(8)(9), the gating units

Gα(·)

Gβ(·)

, and

Gγ(·)

are proposed to formulate the three neuronal behaviors, thus computing the

consequent vectors.

Computing L(t,l).

In computing

L(t,l)

, the two primitives

τlin

and

τexp

are considered.

τlin

is the

linear decay primitive, and

τexp

is the exponential decay primitive. As their names indicate,

τlin

and

τexp

respectively play the roles of the linear decay and exponential decay on

U(t−1,l)

. Through

the gating unit

Gα(·)

, the two primitives are fused together to perform membrane potential leakage.

Following is the formulation of Gα(·):

Gα(U(t−1,l);τlin, τexp) = [1 −α(1 −τexp)]U(t−1,l)−(1 −α)τlin,(10)

where

is the gating factor of

Gα(·)

, and its value is bounded within

(0,1)

. When

approaches

[1 −α(1 −τexp)]

and

(1 −α)

tend to

, which means GLIF performs linear decay. As

increases

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

GLIF:AUniedGatedLeakyIntegrate-and-FireNeuronforSpikingNeuralNetworksXingtingYao1;2,FanrongLi1;2,ZitaoMo1,JianCheng1;31InstituteofAutomation,ChineseAcademyofSciences2SchoolofFutureTechnology,UniversityofChineseAcademyofSciences3CASCenterforExcellenceinBrainScienceandIntelligenceTechnology{yaoxingt...

展开>> 收起<<

GLIF A Uniﬁed Gated Leaky Integrate-and-Fire Neuron for Spiking Neural Networks Xingting Yao12 Fanrong Li12 Zitao Mo1 Jian Cheng13.pdf

共16页,预览4页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

GLIF A Uniﬁed Gated Leaky Integrate-and-Fire Neuron for Spiking Neural Networks Xingting Yao12 Fanrong Li12 Zitao Mo1 Jian Cheng13

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: