1 Edge-based Monocular Thermal-Inertial Odometry in Visually Degraded Environments

2025-04-30 0 0 2.54MB 8 页 10玖币

侵权投诉

Edge-based Monocular Thermal-Inertial Odometry

in Visually Degraded Environments

Yu Wang, Haoyao Chen*, Member, IEEE, Yufeng Liu, Shiwu Zhang, Member, IEEE

Abstract—State estimation in complex illumination environ-

ments based on conventional visual-inertial odometry is a chal-

lenging task due to the severe visual degradation of the visual

camera. The thermal infrared camera is capable of all-day

time and is less affected by illumination variation. However,

most existing visual data association algorithms are incompatible

because the thermal infrared data contains large noise and low

contrast. Motivated by the phenomenon that thermal radiation

varies most signiﬁcantly at the edges of objects, the study

proposes an ETIO, which is the ﬁrst edge-based monocular

thermal-inertial odometry for robust localization in visually

degraded environments. Instead of the raw image, we utilize

the binarized image from edge extraction for pose estimation

to overcome the poor thermal infrared image quality. Then, an

adaptive feature tracking strategy ADT-KLT is developed for

robust data association based on limited edge information and its

distance distribution. Finally, a pose graph optimization performs

real-time estimation over a sliding window of recent states by

combining IMU pre-integration with reprojection error of all

edge feature observations. We evaluated the performance of the

proposed system on public datasets and real-world experiments

and compared it against state-of-the-art methods. The proposed

ETIO was veriﬁed with the ability to enable accurate and robust

localization all-day time.

Index Terms—Thermal-Inertial Odometry, Edge information,

Visual Degradation.

I. INTRODUCTION

ACCURATE and robust state estimation in GNSS-denied

environments is an active research ﬁeld due to its

wide applications in simultaneous localization and mapping

(SLAM), 3D reconstruction, and active exploration. The sen-

sor suit consisting of a monocular camera and IMU, which

provides complementary information, is the minimum solu-

tion for recovering the metric six degrees-of-freedom (DOF)

[1]. Considering that both camera and IMU are light-weight

and low-cost, monocular visual-inertial odometry (VIO) is a

common solution for localization and navigation [2]. Existing

VIO frameworks have been mature in stable environments.

However, the environments in disaster areas are uncertain

and prone to extreme light distribution, dynamic illumination

variation, or visual obscurants such as dust, fog, and smoke

[3]. Such visual degradation always reduces the reliability of

VIO estimation solutions.

This work was supported in part by the National Natural Science Foundation

of China under Grant U1713206 and U21A20119. (Corresponding author:

Haoyao Chen.)

Y. Wang, H. Chen* ,and Y. Liu are with the School of Mechanical

Engineering and Automation, Harbin Institute of Technology Shenzhen, P.R.

China, e-mail: hychen5@hit.edu.cn.

S. Zhang is with Department of Precision Mechinery and Precision Instru-

mentation, University of Science and Technology of China.

Thermal infrared camera, operating in the longwave infrared

spectrum and capturing thermal-radiometric information, has

attracted more attention in recent years. Compared with the

visual camera, thermal infrared cameras exhibit evident advan-

tages when applied to disaster areas for their all-day percep-

tual capability [4]. However, using thermal infrared cameras

directly to existing VIO frameworks is challenging for the

following reasons. First, the captured image data are typically

low contrast [5]. Second, many vision-observable information-

rich textures, such as colors and streaks, are lost in thermal

images due to the indistinguishability of thermal radiation

from surrounding regions. Lastly, nonuniformity correction

(NUC) or ﬂat-ﬁeld correction (FFC) is performed during

thermal infrared camera operation to eliminate accumulated

nonzero-mean noise [6]. Such blackout not only introduces

periods of data interruption but may also signiﬁcantly change

image appearance between consecutive frames.

The current thermal-inertial odometry (TIO) solutions are

mainly improved from normal VIO. Feature-based thermal

odometry that requires special contrast enhancement on in-

frared images for feature extraction was developed [7]–[9].

However, preprocessing will induce additional noise, resulting

in wrong correspondences. 14-bit or 16-bit full radiometric

data from a thermal infrared camera was directly utilized for

motion estimation [3], [10] to avoid a signiﬁcant change in

image appearance resulting from rescaling operation. How-

ever, their approaches require enabling NUC in long-term

applications to address the temperature drift problem and are

not directly compatible with the 8-bit image. By selecting the

most reliable modality through several metrics, ROVTIO [11]

fuses asynchronous thermal, visual and inertial measurements

for estimating the odometry, which leads the system to au-

tonomous switch between the different modalities according

to the environmental conditions.

With the development of deep learning, the neural network

is introduced into pose estimation from thermal infrared im-

ages. TP-TIO [12], which utilizes CNN for feature detection

and IMU-aided full radiometric-based KLT method for feature

tracking, is the ﬁrst tightly coupled deep thermal-inertial

odometry algorithm. Combining the hallucination network

with a selective fusion mechanism, Saputra et al. [13] pro-

posed a deep neural odometry architecture for pose regression

named DeepTIO, which introduced an end-to-end scheme.

Based on DeepTIO, Saputra et al. [14] recently presented

a complete thermal-inertial SLAM system, including neural

abstraction, graph-based optimization, and a global descriptor-

based neural loop closure detection. Combining the advantage

of conventional and learning-based methods, Jiang et al. [15]

arXiv:2210.10033v2 [cs.RO] 22 Oct 2022

proposed a real-time system with an image enhancement

method for feature detection and a light-weight optical ﬂow

network for feature tracking.

The learning-based approach presents promising results.

However, it needs GPU for algorithm acceleration and is

unsuitable for low-cost and low-load applications. In addition,

the transferability of network models hinders the popular-

ization of learning-based methods. Considering the ground

robots or MAVs in rescue applications, this study follows

the conventional framework for real-time state estimation with

CPU-only, and the robustness of data association is the core

problem to be addressed.

Most existing VIO systems generally use the point features

as the visual information. However, point features detection

and tracking in textureless or varying illumination environ-

ments are challenging [16]. Systems integrating additional

geometry structure constraints, such as line and plane in-

formation, were proposed to supplement the point features

and improve the robustness of state estimation [17]–[19].

Edge provides semi-dense information about the environ-

ment’s structure and exhibits a crossover between indirect

and direct methodologies [20]. Literature [21], [22] developed

real-time edge-based visual odometry (REBVO) for indoor

localization, in which edge matching is directly performed

via edge contour alignment by minimizing distance transform

(DT) error. Based on a local sliding window optimization

over several keyframes, Fabian Schenk et al. [23] presented

the ﬁrst edge-based SLAM system. By formulating the ICP-

based motion estimation as maximum a posteriori (MAP)

estimation, Zhou et al. [24] tracked edge features based on

the approximate nearest neighbor ﬁelds.

Inspired by the signiﬁcant change in thermal radiation at

the edge of objects, our previous work developed an edge-

based infrared visual odometry that detects and tracks the

reliable edges in images to address the limitations of thermal

infrared images in data association [20]. As a matter of

course, this study presents an edge-based monocular thermal-

inertial odometry (ETIO) that uses the edge information in

the front-end component to establish reliable correspondences

between images. And in the back-end component, a pose graph

optimization performs estimation over a sliding window of

recent states by combining IMU pre-integration factors with

reprojection error of all edge feature observations. Such an

optimization scheme can effectively suppress the inﬂuence of

data interruption on state estimation. The main contributions

of this letter are three-fold:

1) An edge-based thermal-inertial odometry, named ETIO,

is proposed to provide real-time state estimation in

visually degraded environments. To the best of our

knowledge, it is the ﬁrst edge-based TIO and outper-

forms state-of-the-art TIO methods.

2) An adaptive distance transform-aided KLT (ADT-KLT)

tracker is proposed based on limited edge information

and distance ﬁeld to improve the feature tracking robust-

ness.

3) Experiments in public datasets and the real-world show

our method achieves competitive accuracy and robust-

ness in all-day state estimation.

II. SYSTEM OVERVIEW

Our algorithm is speciﬁcally designed for the thermal in-

frared camera. The overall framework of the proposed ETIO

consists of three major modules shown in Fig.1. The system

starts with an image preprocessing module. Thermal infrared

images even with poor quality are converted to binarized edge

images by the Difference of Gaussians (DoG) and sub-pixel

reﬁnement. As a feature enhancement algorithm, the DoG can

be utilized to increase the visibility of edges and ﬁlter out

the low-contrast areas. Further, sub-pixel reﬁnement is also

utilized for edge thining.

In the data association module, the distance ﬁeld of the edge

image is utilized to introduce spatial distribution constraints

and a stability analysis is performed to assess the consistency

of distance ﬁeld with the help of IMU pre-integration [25].

With an adaptive switching policy, the ADT-KLT tracker is

developed for robust feature tracking on edge images. Finally,

a sliding-window optimization module is developed to tightly

and efﬁciently fuse the measurement information from point

features and IMU pre-integration.

Frame and notation deﬁnitions used throughout this paper

is deﬁned as follows. We consider (·)cas the camera frame,

(·)bas the body frame located at the IMU frame, and (·)was

the world frame. The world frame is consistent with (·)bat

the initial position. (·)w

bireﬂects the coordinate transformation

from ith body frame to the world frame. To formulate the TIO

optimization problem with a commonly-used sliding window

[1], we ﬁrst deﬁne the IMU state at time tias

xi= [pw

bi,qw

bi,vw

bi,bai,bgi],(1)

with the position pw

bi, orientation qw

bi, velocity vw

biand biases

of the accelerometer and gyroscope bai and bgi, respectively.

Together with the point features parameterized by the in-

verse depth λ, the full state vector Xto be estimated is deﬁned

as follows

X= [x1,x2,··· ,xnp,λ1,λ2,··· ,λninv ],(2)

with a sliding window of npstates and ninv edge point features.

III. METHODOLOGY

This section presents the details of ETIO shown in Fig.1.

Firstly, the salient edge points for each new thermal infrared

image frame are extracted to ﬁlter the noisy or low-contrast

areas. Then, the existing features are tracked, and new features

are detected to maintain a minimum number of features. An

ADT-KLT tracker based on Distance Transform is presented

for robust data association on sparse edge images. Finally, a

sliding window-based tightly coupled framework is utilized

for high accuracy and efﬁcient state estimation.

A. Edge Extraction

Robustness and repeatability of edge extraction in con-

secutive frames are essential to implement robust feature

associations. The accuracy of the 2D edges in the thermal

infrared image frame is also an important factor for state

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

1Edge-basedMonocularThermal-InertialOdometryinVisuallyDegradedEnvironmentsYuWang,HaoyaoChen*,Member,IEEE,YufengLiu,ShiwuZhang,Member,IEEEAbstractStateestimationincomplexilluminationenviron-mentsbasedonconventionalvisual-inertialodometryisachal-lengingtaskduetotheseverevisualdegradationofthevisualca...

展开>> 收起<<

1 Edge-based Monocular Thermal-Inertial Odometry in Visually Degraded Environments.pdf

共8页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

1 Edge-based Monocular Thermal-Inertial Odometry in Visually Degraded Environments

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: