The eyes and hearts of UA V pilots observations of physiological responses in real-life scenarios Alexandre Duval1 Anita Paas2 Abdalwhab Abdalwhab1and David St-Onge1

2025-05-06 0 0 5.54MB 7 页 10玖币

侵权投诉

The eyes and hearts of UAV pilots: observations of physiological

responses in real-life scenarios

Alexandre Duval1, Anita Paas2, Abdalwhab Abdalwhab1and David St-Onge1

Abstract— The drone industry is diversifying and the number

of pilots increases rapidly. In this context, ﬂight schools need

adapted tools to train pilots, most importantly with regard to

their own awareness of their physiological and cognitive limits.

In civil and military aviation, pilots can train themselves on

realistic simulators to tune their reaction and reﬂexes, but also

to gather data on their piloting behavior and physiological

states. It helps them to improve their performances. Opposed

to cockpit scenarios, drone teleoperation is conducted outdoor

in the ﬁeld, thus with only limited potential from desktop simu-

lation training. This work aims to provide a solution to gather

pilots behavior out in the ﬁeld and help them increase their

performance. We combined advance object detection from a

frontal camera to gaze and heart-rate variability measurements.

We observed pilots and analyze their behavior over three ﬂight

challenges. We believe this tool can support pilots both in their

training and in their regular ﬂight tasks.

I. INTRODUCTION

The industry of teleoperated drones for service, such as in

infrastructure inspection, crops monitoring and cinematog-

raphy, has expanded at least as fast as the technology that

supports it over the past decade. However, in most countries

the regulation is only slowly adapting. Nevertheless, several

regulating bodies already recognized human factors as a core

contributor to ﬂight hazards. While core technical features

of the aerial systems are evolving, namely autonomy, ﬂight

performances and onboard sensing, the human factors of

UAV piloting stay mostly uncharted territory.

Physiological measures, including eye-based measures

(changes in pupil diameter, gaze-based data, and blink rate),

heart rate variability, and skin conductance, are valuable

indirect measures of cognitive workload. These measures

are increasingly used to measure workload level during a

task and are being integrated into interfaces and training

applications to optimize performance and training programs

[1].

Eye-tracking glasses can be used to monitor training

progress during various levels of task load. In a task requiring

operators to track targets and other vehicles on a map,

Coyne and Sibley [2] found a signiﬁcant decrease in operator

situational awareness when task load was high, which was

related to reduced eye gaze spent on the map. This suggests

that eye gaze may be useful as a predictor of situational

*We thank NSERC USRA and Discovery programs for their ﬁnancial

support. We also acknowledge the support provided by Calcul Qu´

ebec and

Compute Canada.

1Alexandre Duval, Abdalwhab Abdalwhab and David St-Onge are with

the Lab INIT Robots, Department of Mechanical Engineering, Ecole de

technologie sup´

erieure, Canada name.surname@etsmtl.ca

2Anita Paas is with the Department of Psychology, Concordia University,

Canada anita.paas@concordia.ca

awareness. Further, Memar and Esfahani [3] found that gaze-

based data were related to target detection and situational

awareness in a tele-exploration task with a swarm of robots.

Thus, multisensory conﬁgurations can be more robust to

capture cognitive load. While each sensor is susceptible to

some noise, these sources of noise do not overlap between

sensors, such as HRV not inﬂuenced by luminance. This

works aims at extracting gaze behavior and so we also gather

pupil diameter for cognitive load estimation. However, we

added another device extract HRV metrics and enhance our

cognitive load estimation.

Section II opens the path with an overview of the vari-

ous inspirational domains to this work. We then build our

solution on a biophysical capturing software (sec. III) and a

detector trained on a custom dataset (sec. IV). Finally, we

present the results of a small user study in sec. V and discuss

our observations of the pilots behaviors.

II. RELATED WORKS

A. On gaze-based behavioral studies

The beneﬁt of eye tracking is that we can measure gaze

behavior in addition to changes in pupil diameter. Gaze

behavior can provide information about the most efﬁcient

way to scan and monitor multiple sources of input. For

example, in surveillance tasks, operators monitoring several

screens can be supported by systems that track gaze behavior

and automatically notify the operator to adjust their scanning

pattern [4]. In a simulated task, Veerabhadrappa, et al.

[5] found that participants achieved higher performance on

a simulated UAV refuelling task when they maintained a

longer gaze on the relevant region of interest compared to

less relevant regions. Further, in training scenarios, gaze-

based measures can identify operator attention allocation

and quantify progress of novice operators [6]. Gaze-based

measures of novices can also be compared with those of

experts to determine training progress and ensure efﬁcient

use of gaze.

In a review paper focused on pilot gaze behaviour, Ziv

[7] found that expert pilots maintain a more balanced visual

scanning. Expert pilots scan the environment more efﬁciently

and spend less time on each instrument compared to novices.

However, in complex situations, experts spend more time

on the relevant instruments which enables them to make

better decisions than novices. Overall, Ziv concluded that

the differences in gaze behavior between expert and novice

pilots are related to differences in ﬂight performance.

arXiv:2210.14910v1 [cs.HC] 26 Oct 2022

B. On object detection

Object detection identiﬁes various objects in the image

such as cars, planes, dogs and cats and localize them, often

using a bounding box around each object instance.

A plethora of good models have been developed for

object detection [8]. They can mainly be classiﬁed into two

categories, two-stage object detectors, and single-stage object

detectors. With two-stage detectors, such as R-CNN [9],

SPP-net [10] and DetectoRS [11], the ﬁrst step involves

creating region proposals, and the second step further reﬁnes

the proposed bounding boxes and classify them. Whereas

single-stage detectors directly generate bounding boxes and

classes without the need for region proposals. Examples

of those are YOLO [12], SSD [13], and EfﬁcientDet [14].

Generally speaking, two-stage detectors are more accurate

than single-stage detectors but less applicable in real-time

scenarios.

While these solutions are common for robotic perception

[15] and static sensing [16], only a handful of works assess

their potential to understand user behavior.

Geert Brˆ

one, et al [17] combined gaze data with object

detection algorithms to argue the effectiveness of using

detection algorithms for automatic gaze data analysis. They

also designed a proof of concept for their method but did not

report the results. Another research [18] investigated the use

of two detection models (YOLOv2 and OpenPose) to relate

the gaze location speciﬁcally to the interlocutor’s visible head

and hands during humans face-to-face communication.

A more recent work [19] trained a classiﬁcation model on

synthetic data to classify images cropped around the gaze

location as a method to annotate volume of interests. The

issue with this approach is that an image (or image crop) will

be assigned only one class. Even if it contains more than one

object it will be labeled as an image of the most prominent

object, missing the cases where a person could be looking

at multiple objects that are close to each other. Whereas

object detection holds the potential to relay this information

by identifying all objects in the image not just the most

prominent one. Similarly, [20] also used a classiﬁcation

model combined with image cropping, and compared it to

using an object detection model. However, they only relied

on available pretrained models without any ﬁt to speciﬁc

application.

Unlike those previous works, our work presents a model

tailored and tuned to a realist application and use the tools

to extract meaningful behavioral data with UAV pilots.

Closer to our research interest, Miller et al. [21] ﬁne tuned

a pretrained InceptionV2 detection model to annotate objects

of interest in eye-tracking video data, nut also integrate it

with body motion capture data. Their aim was to relate the

gaze location with the participant’s body position and other

objects in the frame. Further, they deployed their method in

a user study to investigate distance from gaze point to target

object in a target interception task. Our work is different

from theirs, in the application scenario, the tools we deploy

(learning model and biophysics measures) and the fact that

we provide a more in depth analysis of the user study.

Finally, object detection demands a lots of processing

power: running state-of-the-art detection algorithms at 30 fps

can hardly be done onboard, a wearable or even a more

capable robot computer. Previous performance test were

conducted between variants of YOLO in live situation for

drone emergency landing [22] in terms of the mean average

precision (mAP) and frames per second (FPS). Two model

stands out : YOLOv3 for is speed and YOLOv5 for its

accuracy.

III. OPERATOR BIOPHYSICS

Fig. 1. Our contribution to HRI4ROS pipeline shown as ROS rqt graph.

The input nodes are on the left. Each node publishes in a set of speciﬁc

topics under the standard namespaces.

Physiological signals capture and analysis are fundamental

to human-centered robotic system design and validation. Sev-

eral works demonstrate the relevance of these measurements,

but each with its own implementation. It can then be time-

consuming to reproduce any of these studies accurately and

to deploy their tools and methods in other contexts.

In robotics, the Robot Operating System (ROS) positions

itself as a standard to share and collaborate on code and

software infrastructure. Its community is slowly integrating

user-based measurements and wearable device drivers to

cope with the reproductibility and sharing challenge of the

Human-Robot Interaction (HRI) community [23]. A stan-

dard to include HRI considerations in ROS is currently in

progress [24].

Our work aims to contribute to the community effort

with the integration of new sensor modality in the stan-

dard. The standard recommends to split the human-related

aspects into ﬁve sub-namespaces. However, to better ﬁt

pupilometry data in the structure, we split the sub-namespace

/human/faces/<face id>in an other subcategory, namely

the eyes. Eyes data are related to gaze, pupil diameter and

blinks. The gaze messages (2D and 3D) are compose of a

geometry msgs/Point and two timestamps : one from ROS,

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

TheeyesandheartsofUAVpilots:observationsofphysiologicalresponsesinreal-lifescenariosAlexandreDuval1,AnitaPaas2,AbdalwhabAbdalwhab1andDavidSt-Onge1AbstractThedroneindustryisdiversifyingandthenumberofpilotsincreasesrapidly.Inthiscontext,ightschoolsneedadaptedtoolstotrainpilots,mostimportantlywithreg...

展开>> 收起<<

The eyes and hearts of UA V pilots observations of physiological responses in real-life scenarios Alexandre Duval1 Anita Paas2 Abdalwhab Abdalwhab1and David St-Onge1.pdf

共7页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

The eyes and hearts of UA V pilots observations of physiological responses in real-life scenarios Alexandre Duval1 Anita Paas2 Abdalwhab Abdalwhab1and David St-Onge1

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: