Draft BENCHMARKING LEARNT RADIO LOCALISATION UNDER DISTRIBUTION SHIFT

2025-04-27 0 0 3.23MB 26 页 10玖币

侵权投诉

Draft

BENCHMARKING LEARNT RADIO LOCALISATION

UNDER DISTRIBUTION SHIFT

Max Arnold∗

Bell Labs

Mo Alloulah

Bell Labs

ABSTRACT

Deploying radio frequency (RF) localisation systems invariably entails non-trivial

effort, particularly for the latest learning-based breeds. There has been little prior

work on characterising and comparing how learnt localiser networks can be de-

ployed in the ﬁeld under real-world RF distribution shifts. In this paper, we present

RadioBench: a suite of 8 learnt localiser nets from the state-of-the-art to study

and benchmark their real-world deployability, utilising ﬁve novel industry-grade

datasets. We train 10k models to analyse the inner workings of these learnt lo-

caliser nets and uncover their differing behaviours across three performance axes:

(i) learning, (ii) proneness to distribution shift, and (iii) localisation. We use in-

sights gained from this analysis to recommend best practices for the deployability

of learning-based RF localisation under practical constraints.

1 INTRODUCTION

Decades of of radio frequency (RF) localisation research have given us a variety of classic meth-

ods (Patwari et al., 2005; Gezici et al., 2005). Newer machine learning incarnations can enhance

location estimation considerably (Zanjani et al., 2022; Karmanov et al., 2021), albeit at the expense

of proneness to distributional shift in wireless signals. For example, models trained on signals from

a warehouse environment may not work well in another different environment (Arnold et al., 2018).

If learnt localiser networks are to be productised and deployed, it is imperative that we robustify

them. To achieve real-world robustness, we need to understand (i) the performance nuances of

learnt localisation models, (ii) when, how, and why do such models work, and (iii) when do they

fail.

Robustness to distribution shift (i.e., out of distribution (OOD) generalisation) is an established

line of enquiry in mainstream machine learning (Gulrajani & Lopez-Paz, 2020; Hendrycks et al.,

2021; Koh et al., 2021). However, there is little in the way of robustness investigations for learnt RF

localisation. Though lower dimensional than images, wireless signals are prone to acute variabilities

stemming from environment- and/or system-dependent propagation conditions (Tse & Viswanath,

2005), which are hard to control for. Sidestepping this complexity, recent works have incorporated

environment-dependent priors (e.g., ﬂoorplans) in order to achieve robust learnt RF localisation in

that environment (Karmanov et al., 2021; Zanjani et al., 2022; Ghazvinian Zanjani et al., 2021).

In this paper, we seek to understand the practical deployability of learnt RF localiser nets from ﬁrst

principles and without invoking extra robustifying priors. To this end, we build RadioBench: a suite

of RF localiser nets from the state-of-the-art. We conduct a systematic comparative study on these

localiser nets, utilising ﬁve novel industry-grade datasets. We analyse the inner workings of these

localiser nets and uncover their differing behaviours across three performance axes: (i) learning, (ii)

proneness to distribution shift, and (iii) localisation. Our contributions are:

• We introduce RadioBench: a benchmarking suite of RF localiser nets from the state-of-the-

art, as well as a best-in-class classical probabilistic approach.

• We introduce 5 large-scale, industry-grade RF localisation datasets with differing charac-

teristics that pertain to the study of wireless OOD robustness.

∗Correspondence to maximilian.wolfgang.arnold@gmail.com

arXiv:2210.01930v1 [cs.LG] 4 Oct 2022

Draft

• We characterise and contrast the performance of 8 RF localiser methods, training in excess

of 10k models in the process. These model conﬁgurations span: architecture, representation

learning, and domain adaptation methods.

• We ﬁnd that representation learning and pretraining are most important for OOD robustness

in a new RF environment, and that variants based on an autoencoder architecture are the

best all-rounder models.

2 PRIMER ON RF LOCALISATION

We consider a system of Msynchronised locators that listen for user devices, where each mth

locator has known 3D position vector um= [xm, ym, zm], and 3D 3×3 orientation matrix Ωm. Let

Abe the angle of arrival (AoA) matrix, rthe range calculated using time of arrival (ToA) and the

speed of the light, then the position of a user device w.r.t. mth locator

pm=ΩmAr+um(1)

Because user devices and locators are not synchronised, range estimates are biased. This can be

compensated by using one locator as reference using time difference of arrival (TDoA). Typically,

modern RF localisation relies on estimating the aforementioned two wireless propagation properties

ToA and AoA, which together are abbreviated (TAoA).

Challenge in rich scattering. Considering a wireless channel between two radio transceivers, the

baseband model of the channel impulse response is given by (Tse & Viswanath, 2005)

h(k) =

p=1

L−1

`=0

ap,`ej(2πfcτp+φp,`)sinc k−τp,`

Ts, k = 0, . . . , O −1

where ap∈R+,φp,` ∈R,τp∈R+are respectively the attenuation, phase, and propagation delay

of the pth path and `th path cluster. Also sinc(x) = sin(πx)

πx is the normalised sinc function, kis the

discrete sampling time, and O−1is the channel order.

It is generally infeasibly to estimate the above parameters because they are underdetermined in

practical implementations. This is further compounded by environments with rich scattering (i.e.,

large Pand L).

Upper bound. Eq. 1 shows that the best performance can be theoretically achieved using perfect

TAoA labels as input to a deep neural net. TAoA, however, are infeasible to measure as groundtruth

per deployed environment because it would entail extensive and very expensive surveying cam-

paigns. Deployment surveys typically leverage laser measurements and tens of hours of calibra-

tion (Scott & Hazas, 2003). Further, moving from a local coordinate system (i.e., per locator) to a

global coordinate system for the environment requires models of that environment and the locator

hardware. Therefore, we designate a TAoA-based localiser net as an upper bound on performance

that is impractical to implement in the real-world under realistic deployment cost and overhead

constraints.

3 MODEL VARIANTS

RadioBench suite compiles all RF localiser net architectures reported in literature. While all facil-

itate location estimation, these architectures operate on differing input formats, produce differing

output formats, as well as deviate in their training details. We believe RadioBench to be the ﬁrst

effort to comprehensively catalogue and evaluate RF localiser nets in order to concretely establish

and contrast their performances. Appendix A reviews RF localisation fundamentals and treats learnt

localiser net variants in more detail.

3.1 ARCHITECTURES

We evaluate four classes of RF localiser nets: supervised CNN (Chen et al., 2017; Arnold et al.,

2019), supervised residual net (ResNet) akin to vision ResNet (He et al., 2016), unsupervised Au-

toEncoder (AE) (Liu et al., 2018), and unsupervised channel charting (CC) (Studer et al., 2018).

Draft

3.2 INPUT-OUTPUT (IO) FORMATS

The above architectures can ingest various representations of input wireless signals. These are:

(i) Channel state information (CSI) is the raw measurements obtained from transceiver chips, (ii)

periodograms (PER) is CSI’s 2D Fourier projection, (iii) a feature reduced version of (i) or (ii), and

(iv) TAoA are the physical propagation primitives that implicitly encode location (cf., Eq. 1), and

are obtained via surveying the environment as discussed in Sec. 2.

The above architectures can also produce multiple output representations. These representations

either encode location directly, or encapsulate it indirectly. Speciﬁcally, output can be: (i) position

estimate, (ii) TAoA primitives, or (iii) latent space that implicitly contains the location intrinsic

space.

Tab. 1 lists all valid architecture-IO conﬁgurations supported in RadioBench. Speciﬁcally for each

method, Tab. 1 shows the effective mapping and its optimisation objective, which is minimisation for

AE and maximisation for CC. For further details around these methods, consult original literature.

Table 1: RF localiser nets and their valid architecture, input-output conﬁgurations, and training

objective implemented in RadioBench.

Detail Supervised Autoencoder (AE) Channel chart (CC)

Mapping CM→R3CM→RM0CM→R2

Optimisation kpn−g(fn)k2

2kfn−g−1(g(fn))k2

2kd(g(Ta), g(Tp)) −d(g(Ta), g(Tn))k

Type ResNet CNN CNN

Input CSI/PER CSI/PER f(CSI/PER)

Output Position/TAoA M0Features Channel Chart

Conﬁguration CSI2Pos, PER2Pos

CSI2TAoA, PER2TAoA CSI AE, PER AE CSI CC, PER CC

3.3 CLASSICAL BASELINE

A best-in-class probabilistic method is also used for benchmarking (Henninger et al., 2022), which

we designate as Classical. Classical uses super-resolution techniques to estimate TAoA from CSI.

These TAoAs are inputted to a maximum likelihood estimation (MLE) pipeline. Note that Classical

results presented throughout paper are averaged per grid position for best-case analysis.

4 FRAMEWORK FOR EMPIRICAL OOD ROBUSTNESS ANALYSIS

We introduce ﬁve large-scale, industry-grade datasets that enable us to empirically study the OOD

robustness of wireless localiser nets. We review the distribution shift mechanisms at play in these

datasets. We then discuss the distribution shift mitigation strategies we deem applicable to the model

variants of Sec. 3.

4.1 DATASETS

We utilise a set of empirical industry-grade datasets to study the nuances of radio localiser net

variants. We summarise the setup and geometric conﬁgurations under which the radio measurement

campaign was conducted.

Fig. 1 depicts three physically distinct environments. Within each, six locators listen to mobile users.

Each locater is equipped with a 3×3 antenna array. Locators are tightly synchronised using White

Rabbit standard (Eidson et al., 2002). Mobile user devices regularly transmit pilot data known

to the locators. The locators receive user pilots and estimate their CSI. Fig. 1a shows 3 Arena 1

Table 2: Datasets utilised in evaluation and their conﬁgurations.

# Dataset points fc(GHz) Bandwidth (MHz) Subcarriers Antennae Locators Groundtruth∗† Area (m2)

1 Arena 1 52991 3.75 100 1630 8 6 SLAM 134.532

2 Arena 2 46266 3.75 100 1630 8 6 SLAM 134.709

3 Arena 3 2181 3.75 100 1630 8 6 SLAM 57.188

4 Industry 1 7990 3.75 100 1630 8 6 Tachy 387.872

5 Industry 2 5037 3.75 100 1630 8 6 Tachy 103.469

∗SLAM: obtained from simultaneous localization and mapping of mobile robot equipped with Lidar

†Tachy: high-precision laser surveying device

Draft

0 10 20

10 0

1 2 3

x [m]

y [m]

a) Arena

0 5 10

15 0

x [m]

b) Industry 1

0 7.5 15

30 0

x [m]

c) Industry 2

-50MHz 0 50MHz

−60

−50

−40

−30

frequency

Power dBm

d) CSI

-90 -45 0 45 90

Angle °

Range [m]

e) PER

Meas 1 Meas 2 Meas 3 Locators

Figure 1: Measurement environments. a) Arena has three different measurement iterations, b) Typ-

ical Industrial environment, and c) Harsher industrial environment with rich scatterers. Data exam-

ples: d) CSI and e) PER.

measurements in blue, green, and red. These correspond to three data collection iterations. Arena

1 and 2 (blue and green) cover the same area but are different due to hardware effects. Arena 3

(red rectangle) corresponds to high-speed driving to simulate a dynamic environment. Fig. 1b & c

correspond to two other industrial environments, with Industry 2 being particularly rich in scattering

effects. Fig. 1d & e depict two examples of the input formats discussed in Tab. 1.

Tab. 2 summarises the conﬁgurations of our 5 novel datasets.

4.2 DISTRIBUTION SHIFT MECHANISMS

From the 5 datasets listed in Tab. 2, we highlight the following mechanisms that result in distribu-

tional shift in RF signals.

(1) Macro environment-induced. Each of the three environments depicted in Fig. 1 comes with

its signature set of propagation conditions. These propagation conditions are largely a function of

the geometry of the environment as well as the spatial conﬁguration of reﬂective surfaces present

within, e.g., metallic machinery, furniture, partition walls and their material composition, etc. We

designate environmental signatures as macro-level effects that shift the bulk of the distribution of

RF signals.

(2) Micro locator-induced. The 3D position and orientation of locators within the environment

affect how they measure the statistics of user RF signals. That is, a locator will also modulate the

distribution of the RF signals it receives. We designate locator signatures as micro-level effects that

further shift the distribution of RF signals.

(3) Micro scattering-induced. Dynamic activities within the environment induce scattering effects

that modulate the distribution of the RF signals. Example scatterers include moving robots and

people. We designate scattering as micro-level effects that further shift the distribution of RF signals.

(4) Misc. For completeness, there are multiple other factors that affect the distribution of RF signals.

Examples include hardware- and frequency-dependent effects. We, however, are mainly interested

in shift mechanisms 1-3 in this work as captured by our 5 empirical datasets.

4.3 DISTRIBUTION SHIFT MITIGATIONS

There are a wide range of methods from the state-of-the-art that enhances robustness and generalisa-

tion on unseen distribution shifts. However, the relative performance of these methods varies largely

across modalities, datasets, and distribution shifts (Hendrycks et al., 2021; Koh et al., 2021; Wiles

et al., 2021). Further, there is little prior experience in adapting some of these concepts to the RF

localiser net setting we study herein.

Loss landscape. Not all models are created equal. For all models described in Sec. 3, we visualise

a dataset’s loss landscape using (Li et al., 2018). This analysis is motivated by the observation that

the landscape geometry affects generalisation dramatically (Li et al., 2018). All things being equal,

Draft

we would therefore favour model variants that exhibit ﬂatter loss landscape geometry. Our intuition

is that a ﬂatter loss landscape would readily support a weak form of generalisability.

Fine-tuning. Fine-tuning is a consistent indicator of of the quality of zero-shot models (Radford

et al., 2021; Wortsman et al., 2022). It is decoupled from some modality-speciﬁc recipes such as

data augmentation. Therefore, we employ simple universal ﬁne-tuning for zero-shot model variants

from Sec. 3 to gauge their relative robustness and generalisability to unseen distribution shifts.

4.4 LEARNABILITY CONDITIONS

We characterise various aspects around the learnability of the model variants described in Sec. 3.

Active label density. We investigate the required number of labels for validation loss convergence.

We employ active learning strategies to glean comparative insights on the learning behaviour of

localiser model variants.

Latent space. Some model variants utilise a latent space that implicitly encodes location. We inves-

tigate the resultant shift in the latent space as a function of macro and micro RF signal distribution

shifts.

Regression head protocol. For model variants with a latent space, we investigate the feasibility of

a regressor head on top of a frozen backbone that is trained on a different dataset. We intuit that if

quality features have been learnt, their projection would still perform competitively w.r.t. regressing

location estimates notwithstanding distribution shift.

5 EXPERIMENTS

We evaluate 8 RF localiser nets. We conduct a comprehensive analysis to quantify performance

aspects around: (i) learnability, (ii) proneness to distribution shift, and (iii) localisation. We use our

5 industry-grade datasets (cf., Tab. 2) for all experiments. We either use all or a subset of localiser

model variants (cf., Tab. 1) depending on suitability for a given experiment. We begin by distilling

our experimental ﬁndings into a concrete set of key takeaways.

As discussed in Sec. 2, method TAoA2Pos in all analyses represents an upper bound on performance.

This is because in practical deployments, surveying groundtruth TAoAs is prohibitively expensive.

5.1 TAKEAWAYS

1 – Learnability: Under smart sample selection criterion, training samples of the order of the

spatial grid sufﬁce for convergence. We observe that models CSI AE, CSI2TAoA, and PER2TAoA

converge after selecting a number samples (via active learning) comparable to the number of spatial

grid locations. Inspecting Fig. 2, this happens after around 2.7k samples. This is inline with Arena

1 dataset that has around 53k data points of which 2.7k are spatial grid locations (cf., Fig. 1a).

0 1k 2k

10−1

101

# of samples

Loss

Least conﬁdence sampling

0 1k 2k

# of samples

Entropy sampling

0 1k 2k

# of samples

Margin sampling

TAoA2Pos

CSI2Pos

CSI2TAoA

PER2TAoA

CSI AE

Figure 2: Active learning (AL) for model variants on Arena 1. AL criteria help models converge

faster in required training samples. Required number of training samples is of the order of a dataset’s

spatial location sampling grid. Some model variants are qualitatively better than others from a

learning standpoint.

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

DraftBENCHMARKINGLEARNTRADIOLOCALISATIONUNDERDISTRIBUTIONSHIFTMaxArnoldBellLabsMoAlloulahBellLabsABSTRACTDeployingradiofrequency(RF)localisationsystemsinvariablyentailsnon-trivialeffort,particularlyforthelatestlearning-basedbreeds.Therehasbeenlittlepriorworkoncharacterisingandcomparinghowlearntloca...

展开>> 收起<<

Draft BENCHMARKING LEARNT RADIO LOCALISATION UNDER DISTRIBUTION SHIFT.pdf

共26页,预览5页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Draft BENCHMARKING LEARNT RADIO LOCALISATION UNDER DISTRIBUTION SHIFT

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: