Opening Access to Visual Exploration of Audiovisual Digital Biomarkers an OpenDBM Analytics Tool Carla Floricel

2025-05-02 0 0 557.56KB 6 页 10玖币

侵权投诉

Opening Access to Visual Exploration of Audiovisual Digital Biomarkers:

an OpenDBM Analytics Tool

Carla Floricel*

University of Illinois Chicago

Jacob Epifano, Stephanie Caamano, Sarah Kark, Rich Christie, Aaron Masino, Andre D Paredes†

AiCure

Figure 1: Individual Panel. A) Head Sketch View which uses custom colored masks to display facial asymmetry, pain expressivity,

overall expressivity (A.3), action unit (AU) intensity (A.1), and head movement (A.3) biomarkers. In A.2, the AUs’ numbers are

displayed and the AUs involved in anger expressivity (bottom selection) are marked with purple highlights. B.1) Timeline used to split

the data into 20 time intervals. Upon change, the timeline will update the mean values displayed in A and highlight the selected

interval in views C,D, and F, while the corresponding frame intervals will be shown in B.2. C, D, F) Facial, Movement, and Acoustics

Views that display temporal data distributions for selected biomarker variables. E) ID Query Subpanel where one video can be

chosen to display its DBM data. G) Correlation View that shows a pairwise Pearson correlation coefﬁcients for a set of variables.

ABSTRACT

Digital biomarkers (DBMs) are a growing ﬁeld and increasingly

tested in the therapeutic areas of psychiatric and neurodegenerative

disorders. Meanwhile, isolated silos of knowledge of audiovisual

DBMs use in industry, academia, and clinics hinder their widespread

adoption in clinical research. How can we help these non-technical

domain experts to explore audiovisual digital biomarkers? The use

of open source software in biomedical research to extract patient be-

havior changes is growing and inspiring a shift toward accessibility

to address this problem. OpenDBM integrates several popular audio

and visual open source behavior extraction toolkits. We present a

visual analysis tool as an extension of the growing open source soft-

ware, OpenDBM, to promote the adoption of audiovisual DBMs in

basic and applied research. Our tool illustrates patterns in behavioral

data while supporting interactive visual analysis of any subset of

derived or raw DBM variables extracted through OpenDBM.

1 INTRODUCTION

The global market value of digital biomarkers (DBMs) is forecasted

to exceed

7 billion by 2026 [1]. DBMs are objective, quantiﬁable,

physiological, and behavioral data collected and measured using

digital devices. Like traditional biomarkers, DBMs have clinical

value, such as predicting and diagnosing disease. However, DBMs

introduce additional beneﬁts that exceed traditional biomarkers’

*e-mail: cﬂori3@uic.edu

†e-mail:andre.paredes@aicure.com

constraints, such as capturing longitudinal and continuous measure-

ments that generate large, rich, and complex datasets [2]. Providing

clinical researchers with practical tools to derive and interpret DBMs

increases their ability to assess changes in health status relevant to

healthcare applications [48]. To support the rising demand for DBM

adoption by clinical researchers, more practical tools are required to

better inform non-technical biomedical researchers on how to use

and identify DBMs [16, 26].

In particular, DBMs’ increasing role in the therapeutic ar-

eas of psychiatric and neurodegenerative disorders provides new-

found interest for clinical researchers to explore measurable au-

diovisual DBM changes to understand better how a patient feels

[8, 12, 30, 31, 38, 40, 41, 43, 49]. Growing open source software

projects, such as OpenDBM, are lowering the barrier for non-

technical clinical researchers to apply quantitative models, includ-

ing machine learning models, to extract audiovisual features in

human speech, voice acoustics, head movement, and facial expres-

sions [21, 22]. However, despite accessible open source tools to

extract audiovisual features, clinical researchers are burdened with

interpreting large and complex quantitative datasets [26].

Given the novelty of DBMs and their still growing taxonomy

and use [14], there is interest among behavioral and biomedical

researchers in ﬁnding practical tools that can facilitate exploratory

analysis for data-informed hypothesis generation. This work aims to

improve researchers’ understanding of the breadth and scope of the

hundreds of audiovisual DBMs available for investigatory adoption.

We propose a visual analytics interface for the OpenDBM software

Our proposed interface reveals patterns and outliers in facial, head

movement, acoustics, and speech DBMs extracted from videos.

To our knowledge, this work presents the ﬁrst audiovisual DBM

1https://aicure.com/opendbm/

arXiv:2210.01618v1 [cs.HC] 4 Oct 2022

interactive visualization tool extracted from and made available

through open source software.

2 RELATED WORK AND BACKGROUND

Open Source Audio and Visual Feature Extraction Tools.

The

system to measure the nature and intensity of vocal and facial ex-

pressions is advancing from manual raters to computerized toolk-

its [23, 42]. These audio and visual toolkits are made possible by

leveraging advancements in machine learning and artiﬁcial intelli-

gence techniques, such as natural language processing and computer

vision [10, 15, 30, 32]. A growing number of open source software

projects are starting to make vocal and facial feature extraction

toolkits freely available online. For vocal feature extractions, Parsel-

mouth [29], Natural Language Toolkit [34], LexicalRichness [44],

and VaderSentiment [56] have been cited for calculating a whole

host of speech and acoustic DBM variables. For facial feature ex-

tractions, OpenFace is a commonly cited behavior analysis toolkit

for detecting and measuring facial landmarks, facial action units,

head pose estimation, and gaze estimation [3, 4, 20]. Collectively,

these software toolkits provide a rich and diverse suite of extracted

features for a more comprehensive analysis of emotional communi-

cation behavior over time. However, none of these projects provide

visualization tools that can aid data interpretation. The project pre-

sented in this paper uses the OpenDBM solution, integrating all of

the previously mentioned vocal and facial toolkits for generating a

visualization of these extracted audiovisual features collectively.

Visualization in Healthcare.

Visualization techniques in health-

care informatics often target cohort data exploration, covering ap-

plications in disease evolution from electronic medical records

[28, 50, 51, 54], heterogeneous longitudinal clinical data [19, 27], or

volumetric patient data [25,52]. However, prior work in healthcare

visual analytics mostly focus on chronic conditions such as can-

cer [9], stroke [33], diabetes [17], or on infectious disease control due

to the COVID-19 pandemic [5, 45], and less on psychiatric and neu-

rodegenerative disorders. Our work incorporates a new approach to

promote individual patient data exploration, while incorporating past

working approaches for cohort data exploration. There is work in

visual analytics for facial activity and head movement and separately,

for voice acoustics and speech measurements [13, 36, 46, 47, 55],

however, some of it doesn’t use video data, and none accounts for all

four measurement categories together. We aim to provide efﬁcient

tools for psychiatric and neurodegenerative health studies using

heterogeneous, audiovisual, behavioral biomarker measurements

extracted during clinical assessments.

3 DESIGN PROCESS AND REQUIREMENTS

The design process followed an Activity-Centered-Design approach

[35]. Our team held remote meetings for nine weeks with ﬁve re-

search groups in DBM therapeutic areas, collectively representing

academia, clinics, and industry. While most collaborators were prin-

cipal investigators with faculty positions, conducting behavioral or

biomedical research, all of them were familiar with the OpenDBM

software. Throughout this process, the team iteratively gained in-

sight into user approaches to explore mappings between DBMs

and conditions and disorders of interest (e.g., major depression and

schizophrenia), gathered functional speciﬁcations for a DBM inter-

face, and prototyped and evaluated the interface. Due to the large

variety in patient behavior for these disorders, we gathered many

speciﬁc requirements. However, we focused on the following subset

of high-level requirements to serve all our collaborators and the open

source community:

R1:

Provide ﬂexibility in showing details about any subset of

DBM variables available through the OpenDBM pipeline. For in-

stance, for early detection of Parkinson’s disease, head movement

measurements are of greater importance than other DBM, such as

voice acoustics. Adaptability to different workﬂows is an essential

factor in open source. Additionally, analyzing hundreds of vari-

ables can be very challenging, and sometimes researchers don’t

know where to start their analyses. Thus, having the means and the

freedom to choose what to explore visually is very important.

R2:

Support interactive visualizations for both raw and derived

data. Visualizing derived, mean variables is important for getting

an effective overview of the cohort data and context for individual

patients, while visualizing raw, temporal variables supports in-depth

analysis for individual patients. This is critical for checking the

quality of the data. For example, researchers might want to exclude

from their analyses videos where the audio or the patient’s face was

not captured.

R3:

Emphasize trends and outliers in DBM data. As an exam-

ple, patients should show negative emotions when talking about

unpleasant or uncomfortable subjects. Domain experts should easily

observe patterns between patients, which is helpful for further stud-

ies. Furthermore, highlighting correlations between biomarkers is

fundamental for better understanding these conditions.

4 VISUALIZATION DESIGN

The visual system is open source and can be operated through the

OpenDBM Github project

from the visualization interface folder.

It is not part of the DBM extraction pipeline, but serves as a comple-

mentary application that visualizes the output of the DBM extraction.

The interface has two interactive panels: the Cohort Panel and the In-

dividual Panel. These panels are composed of multiple coordinated

views that support brushing and linking operations.

4.1 Data Description

Vocal and facial expressions convey emotion and communication

behavior and are one of the most researched topics in psychology

and related disciplines; as a result, audiovisual DBMs extend from

these basic and applied science measurement tools [23]. When a

video is processed through OpenDBM, the several vocal and facial

feature extraction toolkits combine to present hundreds of unique

variable categories relevant to four different audiovisual DBM do-

mains: speech, acoustics, facial expression, and head movement.

Each audiovisual DBM domain provides two sets of quantitative

variables: raw, captured as a frame-by-frame time sequence mea-

surement, and derived, capturing summary statistics on the total

collection of frames. These raw and derived variables provide a

wide range of objective behavioral cues, such as transcription and

lexical richness for speech, jitter and shimmer for acoustics, eye

blink and facial tremor for head movement, and facial action units

and facial asymmetry for facial expressions. The proposed interface

uses these raw and derived variables to display relevant details and

statistics about video cohorts and individual videos using two panels:

the Cohort and the Invididual Panels. The ofﬁcial documentation

provides the full list of DBM variables extracted by OpenDBM.

4.2 Cohort Panel

The Cohort Panel (Fig. 2) has three main views and functions: pro-

vide a cohort overview based on a selected set of variables, observe

variable distributions, and ﬁnd correlations between variables.

Two query subpanels are available for variable and video ID selec-

tion, with the variable query subpanel (Fig. 2.A) having three alterna-

tive components for each of the three main views (Fig. 2.B,D,E). In

the video ID query subpanel (Fig. 2.C), selected IDs are highlighted

in the other views while unselected videos can be hidden from the

rest. All views have accompanying print buttons to generate plot

images that can be used in further studies.

PCA View.

This view (Fig. 2.B.1) uses a scatterplot for a cohort

overview by arranging videos in 2D based on a selected set of

biomarker variables (R1, R2, R3). The axes correspond to the

2https://github.com/AiCure/open_dbm

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

OpeningAccesstoVisualExplorationofAudiovisualDigitalBiomarkers:anOpenDBMAnalyticsToolCarlaFloricel*UniversityofIllinoisChicagoJacobEpifano,StephanieCaamano,SarahKark,RichChristie,AaronMasino,AndreDParedesAiCureFigure1:IndividualPanel.A)HeadSketchViewwhichusescustomcoloredmaskstodisplayfacialasymmet...

收起<<

Opening Access to Visual Exploration of Audiovisual Digital Biomarkers an OpenDBM Analytics Tool Carla Floricel.pdf

共6页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Opening Access to Visual Exploration of Audiovisual Digital Biomarkers an OpenDBM Analytics Tool Carla Floricel

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: