An Analytical Estimation of Spiking Neural Networks Energy Efﬁciency Edgar Lemaire

2025-04-30 0 0 212.07KB 8 页 10玖币

侵权投诉

An Analytical Estimation of Spiking Neural

Networks Energy Efﬁciency

Edgar Lemaire

LEAT, Univ. Cˆ

ote d’Azur, CNRS

edgar.lemaire@univ-cotedazur.fr

Lo¨

ıc Cordone

Renault Software Factory

LEAT, Univ. Cˆ

ote d’Azur, CNRS

loic.cordone@univ-cotedazur.fr

Andrea Castagnetti

LEAT, Univ. Cˆ

ote d’Azur, CNRS

andrea.castagnetti@univ-cotedazur.fr

Pierre-Emmanuel Novac

LEAT, Univ. Cˆ

ote d’Azur, CNRS

pierre-emmanuel.novac@univ-cotedazur.fr

Jonathan Courtois

LEAT, Univ. Cˆ

ote d’Azur, CNRS

jonathan.courtois@univ-cotedazur.fr

Benoˆ

ıt Miramond

LEAT, Univ. Cˆ

ote d’Azur, CNRS

benoit.miramond@univ-cotedazur.fr

Abstract—Spiking Neural Networks are a type of neural

networks where neurons communicate using only spikes. They

are often presented as a low-power alternative to classical neural

networks, but few works have proven these claims to be true.

In this work, we present a metric to estimate the energy

consumption of SNNs independently of a speciﬁc hardware. We

then apply this metric on SNNs processing three different data

types (static, dynamic and event-based) representative of real-

world applications. As a result, all of our SNNs are 6 to 8 times

more efﬁcient than their FNN counterparts.

Index Terms—Spiking neural networks, Energy metrics, Com-

putational metrics, Event-based processing, Low-power artiﬁcial

intelligence

I. INTRODUCTION

Neuromorphic computing has been studied for many years

as a game changer to address low-power embedded AI, assum-

ing that the inspiration from the brain will natively come with

a reduction in energy consumption. Neuromorphic computing

mainly focuses on the encoding and the processing of the

information with spikes. If this property takes an obvious place

in the biological functioning, it is far from obvious that it

is the only one to explain the efﬁciency of the brain. It is

therefore necessary to ask the question whether considering

this characteristic in isolation brings a gain compared to the

classical neural networks used in deep learning. This is the

question that this paper seeks to answer by restricting the study

to standard machine learning tasks on three different types of

data: static, dynamic and event-based data.

There already exist comparisons between Spiking Neural

Networks (SNNs) and Formal Neural Networks (FNNs, i.e.

non-spiking Artiﬁcial Neural Networks) in the literature. How-

ever, such comparisons are hardly generalizable since they

focus on speciﬁc applications or hardware targets [1], [2].

Moreover, the considered applications are often toy examples

not representative of real-world AI tasks. Another approach

consists in producing metrics in order to evaluate the relative

energy consumption between the two coding domains, based

This research is funded by the ANR project DeepSee, Universit´

e Cˆ

ote

d’Azur, CNRS and R´

egion Sud Provence-Alpes-Cˆ

ote d’Azur.

on their respective synaptic operations and activity. We thus

propose a novel metric for energy consumption estimation

taking synaptic operations, memory accesses and element

addressing into account. Moreover, our metric is mostly in-

dependent from low-level implementation or hardware target

to ensure its generality.

The proposed metric is described and applied to three

datasets representative of the aforementioned data types:

CIFAR-10 for the static case, Google Speech Commands

V2 for the dynamic case, and Prophesee NCARS for the

event-based case. Moreover, those datasets are closer to real-

world applications than usual benchmarks of the neuromorphic

community. The metric is used in conjunction with accuracy

measurements to provide an in-depth evaluation of those three

application cases and their relevance for spiking acceleration.

We use the advanced Surrogate Gradient Learning technique

and Direct Encoding spike conversion method, since they offer

the best trade-off between prediction accuracy and synaptic

activity [3].

Our code and trained SNN models are available upon

request.

II. STATE OF THE ART

1) Data Encoding: In order to process data in SNNs, it

must be encoded into spikes. Rate, Time and Direct encoding

are three methods used to convert conventional data towards

spiking domain. Rate coding [4] is the most notorious, since it

provides state-of-the-art accuracy on most AI tasks. However,

it generates a lot of spikes over a large number of timesteps,

drastically impacting computational and energy efﬁciency.

Time coding [5] intends to cope with this issue by encoding

information in latency rather than rate, thus generating much

fewer spikes. Yet, the temporal sparsity of latency-coded

spikes causes long processing times, resulting in an energy

overhead.

To cope with those limitations, we address a novel encoding

scheme: Direct Encoding [6]. It should be noted that this

term is a proposition of ours. In Direct Encoding, the ﬁrst

processing layer is made of hybrid neurons with analog inputs

arXiv:2210.13107v1 [cs.AR] 24 Oct 2022

and spiking behaviour (IF, LIF...). The weights of this layer

are learned during training, thus encoding can be tuned to

reduce spiking activity in the network [7]. In the present

work, we evaluate Direct Encoding on CIFAR-10 and GSC

datasets. Additionally, we evaluate native spike encoding using

event cameras [8]. In this method, each pixel of the sensor

generates a spike whenever it detects a brightness variation,

thus encoding movement into spikes. With such sensor, the

input spiking activity is very low, since only the information

of interest (i.e. moving objects) are returned by the camera.

This property helps improving the computational and energy

efﬁciency of SNNs. We evaluate native encoding using the

Prophesee NCARS dataset.

2) Training of SNN in the literature: Spiking neural net-

works cannot use the classical backpropagation training algo-

rithm to learn their weights because its activations (spikes)

are binary and thus non-differentiable. Encoding static data

such as images using rate coding enables the conversion of

an already trained FNN to a SNN. The most common way is

to replace the ReLU neurons of the FNN by IF neurons [9].

However, the prediction accuracy obtained through conversion

is systematically inferior to their FNN counterpart, while

generating a lot of spikes over a large number of timesteps.

Numerous works have studied how to train SNNs directly

in spiking domain. The best results were obtained using

backpropagation-based learning rules, such as the surrogate

gradient [10]. To circumvent the non-differentiability of spikes

in SNNs, the main idea of surrogate gradient learning is to use

two distinct functions in the forward and backward passes:

an Heaviside step function for the ﬁrst, and a differentiable

approximation of the Heaviside in the latter, such as a sig-

moid function. Using surrogate gradient learning requires a

ﬁxed number of timesteps. As the number of computations

performed by SNNs increases with the number of timesteps,

being able to ﬁx it beforehand and to tune it during the training

is vital to increase the computational efﬁciency of SNNs.

3) Comparisons based on measurements: In the literature,

a few comparisons of SNNs and FNNs have been produced

based on hardware measurements. Some papers show com-

petitive results for SNNs: in [11], the authors highlighted

the inﬂuence of the spike encoding method on the accuracy

and computational efﬁciency of the SNN. They compared the

spiking and formal networks through a Resnet-18 like archi-

tecture on two classiﬁcation datasets. They found that SNNs

reached higher or equivalent accuracy and energy efﬁciency.

In [1], the authors showed that an SNN could reach twice the

power and resource efﬁciency of an FNN, with an MLP on

MNIST dataset targeting ASIC. However, those encouraging

results are still very speciﬁc and thus hardly generalizable. In a

more holistic approach, the authors of [4] performed a design

space exploration (including encoding, training method, level

of parallelism...) and showed that the advantage of the SNN

depended on the considered case, making it difﬁcult to draw

general rules. In [2], researchers showed that SNNs on dedi-

cated hardware (Loihi) demonstrated better energy efﬁciency

than equivalent FNNs on generic hardware (CPU and GPU)

for small topologies, but observed the opposite using larger

CNNs. Once more, the conclusions depended on the studied

case and could not be generalized. Albeit encouraging, those

results are not sufﬁcient to draw general conclusions regarding

the savings offered by event-based processing, since they

depend on the selected application, network hyper-parameters

and hardware targets. Therefore, another approach consists in

comparing both coding domains through estimation metrics,

taking a step back to produce more general conclusions.

4) Comparisons based on metrics: Most energy consump-

tion metrics are based on the number of synaptic opera-

tions: accumulations (ACC) in the SNN and multiplication-

accumulations (MAC) in the FNN. Those models have lim-

itations: energy consumption is assimilated to the energy

consumption of synaptic operations [12], thus other factors

(such as neuron addressing in multiplexed architecture or

memory accesses) are often neglected. Moreover, the models

usually do not take into account some speciﬁc mechanisms,

like membrane potential leakage, reset and biases integration.

In [11], the authors proposed a metric based on synaptic opera-

tions only, and found great energy consumption savings for the

SNN (up to 126×more efﬁcient than the FNN baseline). In [4]

the authors demonstrated that such simplistic metrics were not

always coherent with actual energy consumption of circuits on

FPGA. When taking memory into account, another team [13]

found equivalent energy consumptions for SNNs and FNNs

using various topologies on CIFAR10. Additionally, reference

[14] measured a theoretical maximum spike rate of 1.72 to

guarantee energy savings in the SNN based on a detailed

metric, accounting for synaptic operations, memory accesses

and activation broadcast. Those energy consumption models

are enlightening, but still fail to settle whether event-based

processing is sufﬁcient to increase energy efﬁciency. That is

mostly because those metrics are too hardware speciﬁc, or do

not take all signiﬁcant sources of energy consumption into

account.

In the present work, we propose a metric intended to be

independent from low-level implementation choices, based on

three main operations: neuron addressing, synaptic operations

and memory accesses.

III. METRICS

A. Operational cost

In this section we deﬁne a metric to compute the number

of ACC and MAC due to synaptic operations in SNNs and

FNNs.

1) Convolutional layers: For a convolution layer, the num-

ber of ﬁlters is deﬁned by Cout and their size are noted

Cin ×Hkernel ×Wkernel, where C,Hand Wstands for

channel, height and width. The input and output of the

layer are composed of a set of feature maps, with shapes

(Cin ×Hin ×Win)and (Cout ×Hout ×Wout)respectively. In

the following we consider the padding mode “same” and a

stride S. The number of timesteps is noted T. The equations

describing the number of MAC and ACC operations in FNNs

and SNNs, for convolution layers, are summaried in Eq. 1.

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

AnAnalyticalEstimationofSpikingNeuralNetworksEnergyEfciencyEdgarLemaireLEAT,Univ.Coted'Azur,CNRSedgar.lemaire@univ-cotedazur.frLo¨cCordoneRenaultSoftwareFactoryLEAT,Univ.Coted'Azur,CNRSloic.cordone@univ-cotedazur.frAndreaCastagnettiLEAT,Univ.Coted'Azur,CNRSandrea.castagnetti@univ-cotedazur.frPi...

展开>> 收起<<

An Analytical Estimation of Spiking Neural Networks Energy Efﬁciency Edgar Lemaire.pdf

共8页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

An Analytical Estimation of Spiking Neural Networks Energy Efﬁciency Edgar Lemaire

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: