Boost Event-Driven Tactile Learning with Location Spiking Neurons Peng Kang1 Srutarshi Banerjee2 Henry Chopp2 Aggelos Katsaggelos2

2025-04-30 0 0 2.32MB 24 页 10玖币
侵权投诉
Boost Event-Driven Tactile Learning with
Location Spiking Neurons
Peng Kang1, Srutarshi Banerjee2, Henry Chopp2, Aggelos Katsaggelos2,
and Oliver Cossairt1
1Department of Computer Science, Northwestern, Evanston, IL, USA
2Department of Electrical and Computer Engineering, Northwestern, Evanston, IL, USA
ABSTRACT
Tactile sensing is essential for a variety of daily tasks. Inspired by the event-driven nature and sparse
spiking communication of the biological systems, recent advances in event-driven tactile sensors and
Spiking Neural Networks (SNNs) spur the research in related fields. However, SNN-enabled event-driven
tactile learning is still in its infancy due to the limited representation abilities of existing spiking neurons and
high spatio-temporal complexity in the event-driven tactile data. In this paper, to improve the representation
capability of existing spiking neurons, we propose a novel neuron model called “location spiking neuron”,
which enables us to extract features of event-based data in a novel way. Specifically, based on the classical
T
ime
S
pike
R
esponse
M
odel (TSRM), we develop the
L
ocation
S
pike
R
esponse
M
odel (LSRM). In addition,
based on the most commonly-used
T
ime
L
eaky
I
ntegrate-and-
F
ire (TLIF) model, we develop the
L
ocation
L
eaky
I
ntegrate-and-
F
ire (LLIF) model
*
. Moreover, to demonstrate the representation effectiveness of our
proposed neurons and capture the complex spatio-temporal dependencies in the event-driven tactile data,
we exploit the location spiking neurons to propose two hybrid models for event-driven tactile learning.
Specifically, the first hybrid model combines a fully-connected SNN with TSRM neurons and a fully-
connected SNN with LSRM neurons. And the second hybrid model fuses the spatial spiking graph neural
network with TLIF neurons and the temporal spiking graph neural network with LLIF neurons. Extensive
experiments demonstrate the significant improvements of our models over the state-of-the-art methods
on event-driven tactile learning, including event-driven tactile object recognition and event-driven slip
detection. Moreover, compared to the counterpart artificial neural networks (ANNs), our SNN models are
10×
to
100×
energy-efficient, which shows the superior energy efficiency of our models and may bring
new opportunities to the spike-based learning community and neuromorphic engineering.
Keywords: Spiking Neural Networks, spiking neuron models, location spiking neurons, event-driven
tactile learning, robotic manipulation
1 INTRODUCTION
With the prevalence of artificial intelligence, computers today have demonstrated extraordinary abilities in
visual and auditory perceptions. Although these perceptions are essential sensory modalities, they may fail
to complete tasks in certain situations where tactile perception can help. For example, the visual sensory
modality can fail to distinguish objects with similar visual features in less-favorable environments, such as
dim-lit or in the presence of occlusions. In such cases, tactile sensing can provide meaningful information
like texture, pressure, roughness, or friction and maintain performance. Overall, tactile perception is a
vital sensing modality that enables humans to gain perceptual judgment on the surrounding environment
and conduct stable movements Taunyazov et al. (2020).
With the recent advances in material science and Artificial Neural Networks (ANNs), research on tac-
tile perception has begun to soar, including tactile object recognition Soh and Demiris (2014); Kappassov
et al. (2015); Sanchez et al. (2018), slip detection Calandra et al. (2018), and texture recognition Baishya
and B
¨
auml (2016); Taunyazov et al. (2019). Unfortunately, although ANNs demonstrate promising
performance on the tactile learning tasks, they are usually power-hungry compared to human brains that
require far less energy to perform the tactile perception robustly Li et al. (2016); Strubell et al. (2019).
*
TSRM is the classical
S
pike
R
esponse
M
odel (SRM) in the literature and TLIF is the classical
L
eaky
I
ntegrate-and-
F
ire (LIF)
in the literature. We add the character “T” to highlight their differences from LSRM and LLIF.
arXiv:2210.04277v3 [cs.NE] 19 Dec 2022
Inspired by biological systems, research on event-driven perception has started to gain momentum,
and several asynchronous event-based sensors have been proposed, including event cameras Gallego
et al. (2020) and event-based tactile sensors Taunyazoz et al. (2020). In contrast to standard synchronous
sensors, such event-based sensors can achieve higher energy efficiency, better scalability, and lower
latency. However, due to the high sparsity and complexity of event-driven data, learning with these
sensors is still in its infancy Pfeiffer and Pfeil (2018). Recently, several works Taunyazoz et al. (2020);
Gu et al. (2020); Taunyazov et al. (2020) utilized Spiking Neural Networks (SNNs) Shrestha and Orchard
(2018); Pfeiffer and Pfeil (2018); Xu et al. (2021) to tackle event-driven tactile learning. Unlike ANNs,
which require expensive transformations from asynchronous discrete events to synchronous real-valued
frames, SNNs can process event-based sensor data directly. Moreover, unlike ANNs that employ artificial
neurons Maas et al. (2013); Xu et al. (2015); Clevert et al. (2015) and conduct real-valued computations,
SNNs adopt spiking neurons Gerstner (1995); Abbott (1999); Gerstner and Kistler (2002) and utilize
binary 0-1 spikes to process information. This difference reduces the mathematical dot-product operations
in ANNs to less computationally expensive summation operations in SNNs Roy et al. (2019). Due to
the advantages of SNNs, these works are always energy-efficient and suitable for power-constrained
devices. However, due to the limited representative abilities of existing spiking neuron models and high
spatio-temporal complexity in the event-based tactile data Taunyazoz et al. (2020), these works still cannot
sufficiently capture spatio-temporal dependencies and thus hinder the performance of event-driven tactile
learning.
In this paper, to address the problems mentioned above, we make several contributions that boost event-
driven tactile learning, including event-driven tactile object recognition and event-driven slip detection.
We summarize a list of acronyms and notations in Table 7. Please refer to it during the reading.
First, to enable richer representative abilities of existing spiking neurons, we propose a novel
neuron model called “location spiking neuron”.
Unlike existing spiking neuron models that update
their membrane potentials based on time steps Roy et al. (2019), location spiking neurons update
their membrane potentials based on locations. Specifically, based on the Time Spike Response Model
(TSRM) Gerstner (1995), we develop the “Location Spike Response Model (LSRM)”. Moreover, to make
the location spiking neurons more applicable to a wide range of applications, we develop the “Location
Leaky Integrate-and-Fire (LLIF)” model based on the most commonly-used Time Leaky Integrate-and-
Fire (TLIF) model Abbott (1999). Please note that TSRM and TLIF are the classical Spike Response
Model (SRM) and Leaky Integrate-and-Fire (LIF) in the literature. We add the character “T (Time)” to
highlight their differences from LSRM and LLIF. These location spiking neurons enable the extraction of
feature representations of event-based data in a novel way. Previously, SNNs adopted temporal recurrent
neuronal dynamics to extract features from the event-based data. With location spiking neurons, we can
build SNNs that employ spatial recurrent neuronal dynamics to extract features from the event-based data.
We believe location spiking neuron models can have a broad impact on the SNN community and spur the
research on spike-based learning from event sensors like NeuTouch Taunyazoz et al. (2020), Dynamic
Audio Sensors Anumula et al. (2018), or Dynamic Vision Sensors Gallego et al. (2020).
Next, we investigate the representation effectiveness of location spiking neurons and propose
two models for event-driven tactile learning.
Specifically, to capture the complex spatio-temporal
dependencies in the event-driven tactile data, the first model combines a fully-connected (FC) SNN
with TSRM neurons and a fully-connected (FC) SNN with LSRM neurons, henceforth referred to as the
Hybrid SRM FC
. To capture more spatio-temporal topology knowledge in the event-driven tactile data,
the second model fuses the spatial spiking graph neural network (GNN) with TLIF neurons and temporal
spiking graph neural network (GNN) with LLIF neurons, henceforth referred to as the
Hybrid LIF GNN
.
To be more specific, the Hybrid LIF GNN first constructs tactile spatial graphs and tactile temporal
graphs based on taxel locations and event time sequences, respectively. Then, it utilizes the spatial spiking
graph neural network with TLIF neurons and the temporal spiking graph neural network with LLIF
neurons to extract features of these graphs. Finally, it fuses the spiking tactile features from the two
networks and provides the final tactile learning prediction. Besides the novel model construction, we
also specify the location orders to enable the spatial recurrent neuronal dynamics of location spiking
neurons in event-driven tactile learning. In addition, we explore the robustness of location orders on event-
driven tactile learning. Moreover, we design new loss functions involved with locations and utilize the
backpropagation methods to optimize the proposed models. Furthermore, we develop the timestep-wise
inference algorithms for the two models to show their applicability to the spike-based temporal data.
2/24
Lastly, we conduct experiments on three challenging event-driven tactile learning tasks.
Specif-
ically, the first task requires models to determine the type of objects being handled. The second task
requires models to determine the type of containers being handled and the amount of liquid held within,
which is more challenging than the first task. And the third task asks models to accurately detect the
rotational slip (“stable” or “rotate”) within 0.15s. Extensive experimental results demonstrate the sig-
nificant improvements of our models over the state-of-the-art methods on event-driven tactile learning.
Moreover, the experiments show that existing spiking neurons are better at capturing spatial dependencies,
while location spiking neurons are better at modeling mid-and-long temporal dependencies. Furthermore,
compared to the counterpart ANNs, our models are
10×
to
100×
energy-efficient, which shows the
superior energy efficiency of our models and may bring new opportunities to neuromorphic engineering.
Portions of this work “Event-Driven Tactile Learning with Location Spiking Neurons Kang et al.
(2022)” were accepted by IJCNN 2022 and an oral presentation was given at the IEEE WCCI 2022. We
highlight the additional contributions in this paper.
To make the location spiking neurons user-friendly in various spike-based learning frameworks,
we expand the idea of location spiking neurons to the most commonly-used TLIF neurons and
propose the LLIF neurons. Specifically, the LLIF neurons update their membrane potentials based
on locations and enable the models to extract features with spatial recurrent neuronal dynamics. We
can incorporate the LLIF neurons into popular spike-based learning frameworks like STBP Wu
et al. (2018) and tap their feature representation potential. We believe such neuron models can have
a broad impact on the SNN community and spur the research on spike-based learning.
To demonstrate the advantage of LLIF neurons and further boost the event-based tactile learning
performance, we build the Hybrid LIF GNN, which fuses the spatial spiking graph neural network
with TLIF neurons and the temporal spiking graph neural network with LLIF neurons. The model
extracts features from tactile spatial graphs and tactile temporal graphs concurrently. To the best of
our knowledge, this is the first work to construct tactile temporal graphs based on event sequences
and build a temporal spiking graph neural network for event-driven tactile learning.
We further include more data, experiments, and interpretation to demonstrate the effectiveness
and energy efficiency of the proposed neurons and models. Extensive experiments on real-world
datasets show that the Hybrid LIF GNN significantly outperforms the state-of-the-art methods for
event-driven tactile learning, including the Hybrid SRM FC Kang et al. (2022). Moreover, the
computational cost evaluation demonstrates the high-efficiency benefits of the Hybrid LIF GNN
and LLIF neurons, which may unlock their potential on neuromorphic hardware. The source code
is available at https://github.com/pkang2017/TactileLSN.
We thoroughly discuss the advantages and limitations of existing spiking neurons and location
spiking neurons. Moreover, we provide preliminary results on event-driven audio learning and
discuss the broad applicability and potential impact of this work on other spike-based learning
applications.
The rest of the paper is organized as follows. In Section 2, we provide an overview of related work on
SNNs and event-driven tactile sensing and learning. In Section 3, we start by introducing notations for
existing spiking neurons and extend them to the specific location spiking neurons. We then propose various
models with location spiking neurons for event-driven tactile learning. Last, we provide implementation
details and algorithms related to the proposed models. In Section 4, we demonstrate the effectiveness and
energy efficiency of our models on benchmark datasets. Finally, we discuss and conclude in Section 5.
2 RELATED WORK
In the following, we provide a brief overview of related work on SNNs and event-driven tactile sensing
and learning.
2.1 Spiking Neural Networks (SNNs)
With the prevalence of Artificial Neural Networks (ANNs), computers today have demonstrated ex-
traordinary abilities in many cognition tasks. However, ANNs only imitate brain structures in several
ways, including the vast connectivity and structural and functional organizational hierarchy Roy et al.
3/24
(2019). The brain has more information processing mechanisms like the neuronal and synaptic function-
ality Bullmore and Sporns (2012); Felleman and Van Essen (1991). Moreover, ANNs are much more
energy-consuming than human brains. To integrate more brain-like characteristics and make artificial
intelligence models more energy-efficient, researchers propose Spiking Neural Networks (SNNs), which
can be executed on power-efficient neuromorphic processors like TrueNorth Merolla et al. (2014) and
Loihi Davies et al. (2021). Similar to ANNs, SNNs can adopt general network topologies like convolu-
tional layers and fully-connected layers, but use different neuron models Gerstner and Kistler (2002), such
as the Time Leaky Integrate-and-Fire (TLIF) model Abbott (1999) and the Time Spike Response Model
(TSRM) Gerstner (1995). Due to the non-differentiability of these spiking neuron models, it still remains
challenging to train SNNs. Nevertheless, several solutions have been proposed, such as converting the
trained ANNs to SNNs Cao et al. (2015); Sengupta et al. (2019) and approximating the derivative of
the spike function Wu et al. (2018); Cheng et al. (2020). In this work, we propose location spiking
neurons to enhance the representative abilities of existing spiking neurons. These location spiking neurons
maintain the spiking characteristic but employ the spatial recurrent neuronal dynamics, which enable us
to build energy-efficient SNNs and extract features of event-based data in a novel way. Moreover, based
on the optimization methods for SNNs with existing spiking neurons, we design new loss functions for
SNNs with location spiking neurons and utilize the backpropagation methods with surrogate gradients to
optimize the proposed models.
2.2 Event-Driven Tactile Sensing and Learning
With the prevalence of material science and robotics, several tactile sensors have been developed, in-
cluding non-event-based tactile sensors like the iCub RoboSkin Schmitz et al. (2010) and the SynTouch
BioTacFishel and Loeb (2012) and event-driven tactile sensors like the NeuTouch Taunyazoz et al. (2020)
and the NUSkin Taunyazov et al. (2021). In this paper, we focus on event-driven tactile learning with
SNNs. Since the development of event-driven tactile sensors is still in its infancy Gu et al. (2020), little
prior work exists on learning event-based tactile data with SNNs. The work Taunyazov et al. (2020)
employed a neural coding scheme to convert raw tactile data from non-event-based tactile sensors into
event-based spike trains. It then utilized an SNN to process the spike trains and classify textures. A
recent work Taunyazoz et al. (2020) released the first publicly-available event-driven visual-tactile dataset
collected by NeuTouch and proposed an SNN based on SLAYER Shrestha and Orchard (2018) to solve the
event-driven tactile learning. Moreover, to naturally capture the spatial topological relations and structural
knowledge in the event-based tactile data, a very recent work Gu et al. (2020) utilized the spiking graph
neural network Xu et al. (2021) to process the event-based tactile data and conduct the tactile object
recognition. In this paper, different from previous works building SNNs with spiking neurons that employ
the temporal recurrent neuronal dynamics, we construct SNNs with location spiking neurons to capture
the complex spatio-temporal dependencies in the event-based tactile data and improve event-driven tactile
learning.
3 METHODS
In this section, we first demonstrate the spatial recurrent neuronal dynamics of location spiking neurons by
introducing notations for the existing spiking neurons and extending them to the location spiking neurons.
We then introduce two models with location spiking neurons for event-driven tactile learning. Last, we
provide implementation details and algorithms related to the proposed models.
3.1 Existing Spiking Neuron Models vs. Location Spiking Neuron Models
Spiking neuron models are mathematical descriptions of specific cells in the nervous system. They are
the basic building blocks of SNNs. In this section, we first introduce the mechanisms of existing spiking
neuron models – the TSRM and the TLIF. To enrich their representative abilities, we transform them into
location spiking neuron models – the LSRM and the LLIF.
In the TSRM, the temporal recurrent neuronal dynamics of neuron
i
are described by its membrane
potential
ui(t)
. When
ui(t)
exceeds a predefined threshold
uth
at the firing time
t(f)
i
, the neuron
i
will
generate a spike. The set of all firing times of neuron iis denoted by
Fi={t(f)
i;1 fn}={t|ui(t) = uth},(1)
4/24
(a) (b) (c)
Figure 1.
Recurrent neuronal dynamic mechanisms for the existing spiking neurons of
ν=t
and
location spiking neurons of
ν=l
.
(a)
The refractory dynamics of a TSRM neuron
i
or an LSRM neuron
i
. Immediately after firing an output spike at
ν(f)
i
, the value of
ui(ν)
is lowered or reset by adding a
negative contribution
ηi(·)
. The kernel
ηi(·)
vanishes for
ν<ν(f)
i
and decays to zero for
ν
.
(b)
The
incoming spike dynamics of a TSRM neuron
i
or an LSRM neuron
i
. A presynaptic spike at
ν(f)
j
increases
the value of
ui(ν)
for
νν(f)
j
by an amount of
wi jxj(ν(f)
j)εi j(νν(f)
j)
. The kernel
εi j(·)
vanishes for
ν<ν(f)
j
. “
<
” and “
” indicate the location order when
ν=l
.
(c)
The recurrent neuronal dynamics of a
TLIF neuron
i
or an LLIF neuron
i
. The neuron
i
takes as input binary spikes and outputs binary spikes.
xj
represents the input signal to the neuron
i
from neuron
j
,
ui
is the neuron’s membrane potential, and
oi
is the neuron’s output. An output spike will be emitted from the neuron when its membrane potential
surpasses the firing threshold uth, after which the membrane potential will be reset to ureset .
where
t(n)
i
is the most recent spike time
t(f)
i<t
. The value of
ui(t)
is governed by two different spike
response processes:
ui(t) =
t(f)
iFi
ηi(tt(f)
i) +
jΓi
t(f)
jFj
wi jxj(t(f)
j)εi j(tt(f)
j),(2)
where
Γi
is the set of presynaptic neurons of neuron
i
and
xj(t(f)
j) = 1
is the presynaptic spike at time
t(f)
j
.
ηi(t)
is the refractory kernel, which describes the response of neuron
i
to its own spikes at time
t
.
εi j(t)
is
the incoming spike response kernel, which models the neuron
i
s response to the presynaptic spikes from
neuron
j
at time
t
.
wi j
accounts for the connection strength between neuron
i
and neuron
j
and scales the
incoming spike response. Figure 1a of
ν=t
visualizes the refractory dynamics of the TSRM neuron
i
and Figure 1b of ν=tvisualizes the incoming spike dynamics of the TSRM neuron i.
Without loss of generality, such temporal recurrent neuronal dynamics also apply to other spiking
neuron models, such as the TLIF, which is a special case of the TSRM Maass and Bishop (2001). Since the
TLIF model is computationally tractable and maintains biological fidelity to a certain degree, it becomes
the most commonly-used spiking neuron model and there are many popular SNN frameworks powered
with it Wu et al. (2018). The dynamics of the TLIF neuron iare governed by
τdui(t)
dt =ui(t) + I(t),(3)
where
ui(t)
represents the internal membrane potential of the neuron
i
at time
t
,
τ
is a time constant, and
I(t)
signifies the presynaptic input obtained by the combined action of synaptic weights and pre-neuronal
activities. To better understand the membrane potential update of TLIF neurons, the Euler method is used
to transform the first-order differential equation of Eq. (3) into a recursive expression:
ui(t) = (1dt
τ)ui(t1) + dt
τ
j
wi jxj(t),(4)
where
jwi jxj(t)
is the weighted summation of the inputs from pre-neurons at the current time step.
5/24
摘要:

BoostEvent-DrivenTactileLearningwithLocationSpikingNeuronsPengKang1,SrutarshiBanerjee2,HenryChopp2,AggelosKatsaggelos2,andOliverCossairt11DepartmentofComputerScience,Northwestern,Evanston,IL,USA2DepartmentofElectricalandComputerEngineering,Northwestern,Evanston,IL,USAABSTRACTTactilesensingisessentia...

展开>> 收起<<
Boost Event-Driven Tactile Learning with Location Spiking Neurons Peng Kang1 Srutarshi Banerjee2 Henry Chopp2 Aggelos Katsaggelos2.pdf

共24页,预览5页

还剩页未读, 继续阅读

声明:本站为文档C2C交易模式,即用户上传的文档直接被用户下载,本站只是中间服务平台,本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间,仅对用户上传内容的表现方式做保护处理,对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私,请立即通知玖贝云文库,我们立即给予删除!
分类:图书资源 价格:10玖币 属性:24 页 大小:2.32MB 格式:PDF 时间:2025-04-30

开通VIP享超值会员特权

  • 多端同步记录
  • 高速下载文档
  • 免费文档工具
  • 分享文档赚钱
  • 每日登录抽奖
  • 优质衍生服务
/ 24
客服
关注