STSC-SNN Spatio-Temporal Synaptic Connection with Temporal Convolution and Attention for Spiking Neural Networks Chengting Yu12 Zheming Gu1 Da Li1 Gaoang Wang2 Aili Wang12 Erping Li12

2025-05-02 0 0 9.7MB 9 页 10玖币

侵权投诉

STSC-SNN: Spatio-Temporal Synaptic Connection with Temporal Convolution

and Attention for Spiking Neural Networks

Chengting Yu 1,2, Zheming Gu 1, Da Li 1, Gaoang Wang 2, Aili Wang 1,2,*, Erping Li 1,2

1College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, China

2ZJU-UIUC Institute, Zhejiang University, Haining, China

chengting.21@intl.zju.edu.cn, ailiwang@intl.zju.edu.cn

Abstract

Spiking Neural Networks (SNNs), as one of the algorith-

mic models in neuromorphic computing, have gained a great

deal of research attention owing to temporal information pro-

cessing capability, low power consumption, and high biolog-

ical plausibility. The potential to efﬁciently extract spatio-

temporal features makes it suitable for processing the event

streams. However, existing synaptic structures in SNNs are

almost full-connections or spatial 2D convolution, neither

of which can extract temporal dependencies adequately. In

this work, we take inspiration from biological synapses and

propose a spatio-temporal synaptic connection SNN (STSC-

SNN) model, to enhance the spatio-temporal receptive ﬁelds

of synaptic connections, thereby establishing temporal de-

pendencies across layers. Concretely, we incorporate tem-

poral convolution and attention mechanisms to implement

synaptic ﬁltering and gating functions. We show that endow-

ing synaptic models with temporal dependencies can improve

the performance of SNNs on classiﬁcation tasks. In addition,

we investigate the impact of performance vias varied spatial-

temporal receptive ﬁelds and reevaluate the temporal modules

in SNNs. Our approach is tested on neuromorphic datasets,

including DVS128 Gesture (gesture recognition), N-MNIST,

CIFAR10-DVS (image classiﬁcation), and SHD (speech digit

recognition). The results show that the proposed model out-

performs the state-of-the-art accuracy on nearly all datasets.

Introduction

Spiking neural networks (SNNs) are regarded as the third

generation of neural networks (Maass 1997), with the pur-

pose of addressing the fundamental mysteries of intelligence

and the brain by emulating biological neurons and incor-

porating more biological mechanisms (Roy, Jaiswal, and

Panda 2019). The two fundamental components of SNNs

are spiking neurons and synapses, which create a hierarchi-

cal structure (layers) and subsequently construct a network.

SNNs have attracted a signiﬁcant deal of academic inter-

est in recent years due to their prospective properties, such

as the ability to process temporal information, low power

consumption (Roy, Jaiswal, and Panda 2019), and biologi-

cal interpretability (Gerstner et al. 2014). Currently, SNNs

are capable of processing event stream data with low la-

tency and low power (Pei et al. 2019; Gallego et al. 2020).

However, there is still a performance gap between SNNs and

traditional Artiﬁcial Neural Networks (ANNs). Recent SNN

Figure 1: Illustration of Receptive Fields in Synaptic Con-

nections. (a) The receptive ﬁelds of typical spatial operations

used in SNNs, e.g., fully-connected layers (full) and 2D con-

volutional layers (sparse); (b) The STSC modules proposed

to extend spatial operations with spatio-temporal receptive

ﬁelds.

training techniques based on surrogate gradients and back-

propagation have signiﬁcantly enhanced the performance of

SNNs (Wu et al. 2018; Fang et al. 2021b), while also pro-

moting the further integration of ANNs’ modules into SNNs

(Zheng et al. 2021; Hu, Tang, and Pan 2018; Yao et al. 2021),

greatly accelerating the development of SNNs. However, it

remains challenging to connect these computational tech-

niques with the biological properties of SNNs.

Due to the time-dependent correlation of neuron dynam-

ics, it is believed that SNNs naturally process informa-

tion in both temporal and spatial dimensions (Roy, Jaiswal,

and Panda 2019). Further researches are necessary to har-

ness the spatio-temporal information processing capabilities

of SNNs. Combining ANNs’ modules has signiﬁcantly in-

creased the performance of SNNs in several research stud-

ies. In terms of spatial information processing, CSNN (Xu

et al. 2018) was the ﬁrst to validate the application of con-

volution structure on SNNs, followed by the proposal of

NeuNorm to improve SNNs’ usage of convolution through

auxiliary neurons (Wu et al. 2019). In the time dimension,

(Zheng et al. 2021) implements the time-dependent batch

normalization (tdBN) module to tackle the issue of gradi-

ent vanishing and threshold balancing, and (Yao et al. 2021)

uses the Squeeze-and-Excitation (SE) block (Hu, Shen, and

Sun 2018) to realize the attention distribution of the tem-

poral dimension in order to improve the temporal feature

extraction. Notably, (Zhu et al. 2022) proposes Temporal-

arXiv:2210.05241v1 [cs.NE] 11 Oct 2022

Channel Joint Attention (TCJA) to concurrently process in-

put in both temporal and spatial dimensions, which is a sig-

niﬁcant effort for SNNs’ spatio-temporal feature extraction.

These studies effectively improve the performance of SNNs

by transplanting established ANNs’ modules and method-

ologies. However, applying these computational modules to

SNNs from the standpoint of deep learning dilutes the fun-

damental biological interpretability, bringing SNNs closer

to a mix of existing concepts in machine learning, such as

recurrent neural networks (RNNs), binary neural networks

(BNNs), and quantization networks.

From a biological standpoint, some works focus on the

synapse models, investigating the potential of SNNs in re-

spect of connection modes and information transmission.

(Shrestha and Orchard 2018; Fang et al. 2020a; Yu et al.

2022) integrate impulse response models with synaptic dy-

namics, hence enhancing the temporal information represen-

tation of SNNs; (Cheng et al. 2020) implements intra-layer

lateral inhibitory connections to improve the noise tolerance

of SNNs; from the standpoint of synaptic plasticity, (Bel-

lec et al. 2020; Zhang and Li 2019) introduce bio-plausible

training algorithms as an alternative to back-propagation

(BP), allowing for lower-power training. Experiments re-

vealed that the synaptic models of SNNs have a great deal

of space for modiﬁcation and reﬁnement in order to handle

spatio-temporal data better (Fang et al. 2020a). We propose

a Spatio-Temporal Synaptic Connection (STSC) module for

this reason.

Based on the notion of spatio-temporal receptive ﬁelds,

the structural features of dendritic branches (Letellier et al.

2019) and feedforward lateral inhibition (Luo 2021) moti-

vate this study. By merging the ANNs’ computation mod-

ules (temporal convolutions and attention mechanisms) with

SNNs, we propose the STSC module, consisting of Tem-

poral Response Filter (TRF) module and Feedforward Lat-

eral Inhibition (FLI) module. As shown in Fig. 1, the STSC

can be attached to spatial operations to expand the spatio-

temporal receptive ﬁelds of synaptic connections, hence

facilitating the extraction of spatio-temporal features. The

main contributions of this work are summarized as follows:

• We propose STSC-SNN to implement synaptic connec-

tions with extra temporal dependencies and enhance the

SNNs’ capacity to handle temporal information. To the

best of our knowledge, this study is the ﬁrst to propose

the idea of synaptic connections with spatio-temporal re-

ceptive ﬁelds in SNNs and to investigate the inﬂuence of

synaptic temporal dependencies in SNNs.

• Inspired by biological synapses, we propose two plug-

and-play blocks: Temporal Response Filter (TRF) and

Feedforward Lateral Inhibition (FLI), which perform

temporal convolution and attention operations and can

be simply implemented into deep learning frameworks

for performance improvements.

• On neuromorphic datasets, DVS128 Gesture, SHD, N-

MNIST, CIFAR10-DVS, we have produced positive re-

sults. Speciﬁcally, we acquire 92.36% accuracy on SHD

with a simple fully-connected structure, which is a great

improvement above the 91.08% results obtained with re-

current structure and reaches performance comparable to

ANNs.

Related Work

Learning algorithms for SNNs

In recent years, many works have explored the learning al-

gorithms of SNNs, which can be generally categorized as bi-

ologically inspired approaches (Diehl and Cook 2015; Bel-

lec et al. 2020; Zhang and Li 2019), ANN-to-SNN conver-

sion methods (Orchard et al. 2015; Sengupta et al. 2019;

Han, Srinivasan, and Roy 2020), and surrogate-based di-

rect training methods (Wu et al. 2018; Neftci, Mostafa,

and Zenke 2019; Fang et al. 2021b). Direct training meth-

ods utilize surrogate gradients to tackle the issue of non-

differentiable spike activity (Wu et al. 2018), allowing error

back-propagation (BP) through time to interface the gradi-

ent descent directly on SNNs for training. Those BP-based

methods show strong potential to achieve high accuracy in

a few timesteps by making full use of spatio-temporal in-

formation(Wu et al. 2019; Fang et al. 2021b). However,

more research is required to determine how to better extract

spatio-temporal features for enhanced processing of spatio-

temporal data; this is what we want to contribute.

Attention Modules in SNNs

The attention mechanism distributes attention preferentially

to the most informative input components, which could

be interpreted as the sensitivity of various inputs. The SE

block (Hu, Shen, and Sun 2018) offers an efﬁcient atten-

tion approach to improve representations in ANNs. (Xie

et al. 2016; Kundu et al. 2021) introduced spatial-wise at-

tention in SNNs; then, TA-SNN (Yao et al. 2021) devel-

oped a temporal-wise attention mechanism in SNNs by as-

signing attention factors to each input frame; more subse-

quently, TCJA (Zhu et al. 2022) added a channel-wise atten-

tion module and proposed temporal-channel joint attention.

These studies demonstrate the usefulness of attention mech-

anisms in SNNs by achieving state-of-the-art results on var-

ious datasets. Moreover, based on these investigations, it is

desirable to study other correlations between the attention

mechanism and the biological nature of SNNs, which is the

objective of our research. We employ the attention module

as a feedforward lateral inhibitory connection (Luo 2021),

which develops a gating mechanism for the synapse model,

and enables nonlinear computation by the synapse.

Synaptic Models in SNNs

As one of the fundamental components of SNN, the synap-

tic model has drawn the interest of several researchers.

(Shrestha and Orchard 2018; Fang et al. 2020a; Yu et al.

2022) established temporal relationships between response

post-synaptic currents and input pre-synaptic spikes, there-

fore improving temporal expressiveness. Those temporal re-

lationships are the extension of fully-connected synapses

which are based on the assumption that there is only one

connection between two neurons. Nevertheless, synaptic

connections are often complex, and there are typically many

paths connecting the axons and dendrites of neurons (Luo

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

STSC-SNN:Spatio-TemporalSynapticConnectionwithTemporalConvolutionandAttentionforSpikingNeuralNetworksChengtingYu1,2,ZhemingGu1,DaLi1,GaoangWang2,AiliWang1,2,*,ErpingLi1,21CollegeofInformationScienceandElectronicEngineering,ZhejiangUniversity,Hangzhou,China2ZJU-UIUCInstitute,ZhejiangUniversity,Hainin...

展开>> 收起<<

STSC-SNN Spatio-Temporal Synaptic Connection with Temporal Convolution and Attention for Spiking Neural Networks Chengting Yu12 Zheming Gu1 Da Li1 Gaoang Wang2 Aili Wang12 Erping Li12.pdf

共9页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

STSC-SNN Spatio-Temporal Synaptic Connection with Temporal Convolution and Attention for Spiking Neural Networks Chengting Yu12 Zheming Gu1 Da Li1 Gaoang Wang2 Aili Wang12 Erping Li12

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: