Artiﬁcial Intelligence-Based Methods for Fusion of Electronic Health Records and Imaging Data Farida Mohsen1 Hazrat Ali1 Nady El Hajj12 and Zubair Shah1

2025-04-30 0 0 881.2KB 20 页 10玖币

侵权投诉

Artiﬁcial Intelligence-Based Methods for Fusion of

Electronic Health Records and Imaging Data*

Farida Mohsen1, Hazrat Ali1, Nady El Hajj1,2, and Zubair Shah1,*

1College of Science and Engineering, Hamad Bin Khalifa University, Qatar Foundation, 34110 Doha, Qatar

2College of Health and Life Sciences, Hamad Bin Khalifa University, Qatar Foundation, 34110 Doha, Qatar

*Correspondence to: Zubair Shah

ABSTRACT

Healthcare data are inherently multimodal, including electronic health records (EHR), medical images, and multi-omics data.

Combining these multimodal data sources contributes to a better understanding of human health and provides optimal

personalized healthcare. The most important question when using multimodal data is how to fuse them - a ﬁeld of growing

interest among researchers. Advances in artiﬁcial intelligence (AI) technologies, particularly machine learning (ML), enable

the fusion of these different data modalities to provide multimodal insights. To this end, in this scoping review, we focus

on synthesizing and analyzing the literature that uses AI techniques to fuse multimodal medical data for different clinical

applications. More speciﬁcally, we focus on studies that only fused EHR with medical imaging data to develop various AI

methods for clinical applications. We present a comprehensive analysis of the various fusion strategies, the diseases and

clinical outcomes for which multimodal fusion was used, the ML algorithms used to perform multimodal fusion for each clinical

application, and the available multimodal medical datasets. We followed the PRISMA-ScR (Preferred Reporting Items for

Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines. We searched Embase, PubMed, Scopus,

and Google Scholar to retrieve relevant studies. After pre-processing and screening, we extracted data from 34 studies that

fulﬁlled the inclusion criteria. We found that studies fusing imaging data with EHR are increasing and doubling from 2020

to 2021. In our analysis, a typical workﬂow was observed: feeding raw data, fusing different data modalities by applying

conventional machine learning (ML) or deep learning (DL) algorithms, and ﬁnally, evaluating the multimodal fusion through

clinical outcome predictions. Speciﬁcally, early fusion was the most used technique in most applications for multimodal learning

(22 out of 34 studies). We found that multimodality fusion models outperformed traditional single-modality models for the same

task. Disease diagnosis and prediction were the most common clinical outcomes (reported in 20 and 10 studies, respectively)

from a clinical outcome perspective. Neurological disorders were the dominant category (16 studies). From an AI perspective,

conventional ML models were the most used (19 studies), followed by DL models (16 studies). Multimodal data used in the

included studies were mostly from private repositories (21 studies). Through this scoping review, we offer new insights for

researchers interested in knowing the current state of knowledge within this research ﬁeld.

Introduction

Over the past decade, digitization of health data have grown tremendously with increasing data repositories spanning the

healthcare sectors

. Healthcare data are inherently multimodal, including electronic health records (EHR), medical imaging,

multi-omics, and environmental data. In many applications of medicine, the integration (fusion) of different data sources has

become necessary for effective prediction, diagnosis, treatment, and planning decisions by combining the complementary

power of different modalities, thereby bringing us closer to the goal of precision medicine2,3.

Data fusion is the process of combining several data modalities, each providing different viewpoints on a common

phenomenon to solve an inference problem. The purpose of fusion techniques is to effectively take advantage of cooperative

and complementary features of different modalities

4,5

. For example, in interpreting medical images, clinical data is often

necessary for making effective diagnostic decisions. Many studies found that missing pertinent clinical and laboratory data

during image interpretation decreases the radiologists’ ability to accurately make diagnostic decisions

. The signiﬁcance of

clinical data to support the accurate interpretation of imaging data is well established in radiology as well as in a wide variety

of imaging-based medical specialties such as dermatology, ophthalmology, and pathology that depend on clinical context to

interpret imaging data correctly7–9.

Thanks to the advances of AI and ML models, one can achieve a useful fusion of multimodal data with high-dimensionality

various statistical properties, and different missing value patterns

. Multimodal ML is the domain that can integrate different

*This is pre-print of paper accepted for publication in Scientiﬁc Reports. Cite the ﬁnal version from Nature Scientiﬁc Reports.

Email: haali2@hbku.edu.qa, zshah@hbku.edu.qa

arXiv:2210.13462v1 [cs.LG] 23 Oct 2022

data modalities. In recent years, multimodal data fusion has gained much attention for automating clinical outcome prediction

and diagnosis. This can be seen in Alzheimer’s disease diagnosis and prediction

12–15

when imaging data were combined

with speciﬁc lab test results and demographic data as inputs to ML models, and better performance was achieved than the

single-source models. Similarly, fusing pathological images with patient demographic data observed an increase in performance

in comparison with single modality models for breast cancer diagnosis16. Several studies found similar advantages in various

medical imaging applications, including diabetic retinopathy prediction, COVID-19 detection, and glaucoma diagnosis17–19.

This scoping review focuses on studies that use AI models to fuse medical images with EHR data for different clinical

applications. Modality fusion strategies play a signiﬁcant role in these studies. In the literature, some other reviews have been

published on the use of AI for multimodal medical data fusion

20–26

; however, they differ from our review in terms of their

scope and coverage. Some previous studies focused on the fusion of different medical imaging modalities

20,21

; they did not

consider the EHR in conjunction with imaging modalities. Other reviews focused on the fusion of omics data with other data

modalities using DL models

22,23

. Another study

focused on the fusion of various internet of medical things (IoMTs) data for

smart healthcare applications. Liu et al.

focused exclusively on integrating multimodal EHR data, where multimodality refers

to structured data and unstructured free texts in EHR, using conventional ML and DL techniques. Huang et al.

discussed

fusion strategies of structured EHR data and medical imaging using DL models emphasizing fusion techniques and feature

extraction methods. Furthermore, their review covered the research till 2019 and retrieved only 17 studies. In contrast, our

review focuses on studies using conventional ML or DL techniques with EHR and medical imaging data, covering 34 recent

studies. Table 1provides a detailed comparison of our review with existing reviews.

The primary purpose of our scoping review is to explore and analyze published scientiﬁc literature that fuses EHR and

medical imaging using AI models. Therefore, our study aims to answer the following questions:

Fusion Strategies: what fusion strategies have been used by researchers to combine medical imaging data with EHR?

What is the most used method?

2. Diseases: For what type of diseases are fusion methods implemented?

Clinical outcomes and ML methods: What types of clinical outcomes are addressed using the different fusion strategies?

What kind of ML algorithms are used for each clinical outcome?

4. Resource: What are the publicly accessible medical multimodal datasets?

We believe that this review will provide a comprehensive overview to the readers on the advancements made in multimodal

ML for EHRs and medical imaging data. Furthermore, the reader will develop an understanding of how ML models could be

designed to align data from different modalities for various clinical tasks. Besides, we believe that our review will help identify

the lack of multimodal data resources for medical imaging and EHR, thus motivating the research community to develop more

multimodal medical data.

Preliminaries

We ﬁrst identify the EHR and medical imaging modalities that are the focus of this review. Then, we present the data fusion

strategies that we use to investigate the studies from the perspective of multimodal fusion.

Data modalities

In this review, we focus on studies that use two primary data modalities:

•

Medical imaging modality: This includes N-dimensional imaging information acquired in clinical practice, such as X-ray,

Magnetic Resonance Imaging (MRI), functional MRI (fMRI), structural MRI (sMRI), Positron Emission Tomography

(PET), Computed Tomography (CT), and Ultrasound.

•

EHR data: This includes both structured and unstructured free-text data. Structured data include coded data such as

diagnosis codes, procedure codes, numerical data such as laboratory test results, and categorical data such as demographic

information, family history, vital signs, and medications. Unstructured data include medical reports and clinical notes.

In our review, we consider studies combining the two modalities of EHR and imaging. However, there exist cases where the

data could contain only multiple EHR modalities (structured and unstructured) or multiple imaging modalities (e.g., PET and

MRI). We consider such data as a single modality, i.e., the EHR modality or imaging modality.

2/20

Previous Reviews Year Scope and Coverage Comparative contribution of our review

A review on multimodal

medical image fusion:

Compendious analysis

of medical modalities,

multimodal databases,

fusion techniques and

quality metrics20

2022

Their review focused on the fusion of differ-

ent medical imaging modalities.

Our review focused on the fusion of medi-

cal imaging with multimodal EHR data and

considered different imaging modalities as

a single modality. The two reviews did not

share any common studies.

Advances in multimodality

data fusion in neuroimag-

ing21

2021

Their review focused on the fusion of dif-

ferent imaging modalities, considering neu-

roimaging applications for brain diseases

and neurological disorders.

Our review focused on the fusion of medical

imaging with EHR data, considering vari-

ous diseases, such as neurological disorders,

cancer, cardiovascular diseases, psychiatric

disorders, eye diseases, and Covid-19. The

two reviews did not share any common stud-

ies.

An overview of deep learn-

ing methods for multimodal

medical data mining22

2022

Their review focused on the fusion of dif-

ferent types of multi-omics data with EHR

and different imaging modalities, only con-

sidering DL models for speciﬁc diseases

(COVID-19, cancer, and Alzheimer’s).

Our review focused on the fusion of medi-

cal imaging with EHR data, considering all

AI models for various diseases, such as neu-

rological disorders, cancer, cardiovascular

diseases, psychiatric disorders, eye diseases,

and Covid-19. The two reviews did not share

any common studies.

Multimodal deep learning

for biomedical data fusion:

a review23

2022

Their review focused on the fusion of dif-

ferent types of multi-omics data with EHR

and imaging modalities, considering only

DL models. Moreover, they did not provide

a summary of the freely accessible multi-

modal datasets and a summary of evaluation

measures used to evaluate the multimodal

models.

Our review focused on the fusion of med-

ical imaging with EHR data, considering

all AI models. Moreover, o ur study pro-

vided a summary of the accessible multi-

modal datasets and a summary of evaluation

measures used to evaluate the multimodal

models. The two reviews only shared two

common studies.

A comprehensive survey on

multimodal medical signals

fusion for smart healthcare

systems24

2021

Their survey did not focus on fusing med-

ical imaging with EHR but rather covered

the fusion of IoMTs data for smart health-

care applications and covered studies pub-

lished untill 2020. Moreover, in their review,

multimodality referred to fusing either dif-

ferent 1D medical signals (such as electro-

cardiogram (ECG) and biosignals), different

medical imaging modalities, or 1D medical

signals with imaging.

Our review focused on the fusion of medical

imaging with EHR (structured and unstruc-

tured) for different clinical applications. It

included 34 studies, most of them published

in 2021 and 2022, with no study common

between the two reviews.

Machine learning for

multimodal electronic

health records-based re-

search: Challenges and

perspectives27

2021

Their review focused on the fusion of struc-

tured and unstructured EHR data and did not

consider medical imaging modalities. More-

over, they did not provide a summary of the

freely accessible multimodal datasets and

a summary of evaluation measures used to

evaluate the multimodal models.

Our review focused on the fusion of medi-

cal imaging with EHR and considered struc-

tured and unstructured data in EHR as a sin-

gle modality. The two reviews did not share

any common studies.

Fusion of medical imag-

ing and electronic health

records using deep learning:

a systematic review and im-

plementation guidelines26

2020

Their review focused on the fusion of struc-

tured EHR data and medical imaging, con-

sidering only DL models, and included only

17 studies published until 2019.

Our review focused on the fusion of medi-

cal imaging with EHR data, considering all

AI models, and included 34 studies, almost

more than half published in 2020 and 2021.

Table 1. Comparison with previous reviews.

3/20

Fusion strategies

As outlined in

, fusion approaches can be categorized into early, late, and joint fusion. These strategies are classiﬁed depending

on the stage in which the features are fused in the ML model. Our scoping review follows the deﬁnitions in

and attempts to

match each study to its taxonomy. In this section, we brieﬂy describe each fusion strategy:

•

Early fusion: It joins features of multiple input modalities at the input level before being fed into a single ML algorithm

for training

. The modality features are extracted either manually or by using different methods such as neural networks

(NN), software, statistical methods, and word embedding models. When NN are used to extract features, early fusion

requires training multiple models: the feature extraction models and the single fusion model. There are two types of joint

fusion: type I and type II. Type I fuses the original features without extracting features, while type II fuses extracted

features from modalities.

•

Late fusion: It trains separate ML models on data of each modality, and the ﬁnal decision leverages the predictions of

each model

. Aggregation methods such as weighted average voting, majority voting, or a meta-classiﬁer are used to

make the ﬁnal decision. This type of fusion is often known as decision-level fusion.

•

Joint fusion: It combines the learned features from intermediate layers of NN with features from other modalities as

inputs to a ﬁnal model during training26. In contrast to early fusion, the loss from the ﬁnal model is propagated back to

the feature extraction model during training so that the learned feature representations are improved through iterative

updating of the feature weights. NNs are used for joint fusion since they can propagate loss from the ﬁnal model to the

feature extractor(s). There are two types of joint fusion: type I and type II. The former is when NNs are used to extract

features from all modalities. The latter is when not all the input modalities’ features are extracted using NNs26.

Methods

In this scoping review, we followed the guidelines recommended by the PRISMA-ScR28.

Search strategy

In a structured search, we searched four databases, including Scopus, PubMed, Embase, and Google Scholar, to retrieve the

relevant studies. We note here that MEDLINE is covered in PubMed . For Google Scholar search results, we selected the ﬁrst

110 relevant studies, as, beyond 110 entries, the search results rapidly lost relevancy and were unmatched to our review’s topic.

Furthermore, we limited our search to English-language articles published in the last seven years between January 1, 2015, and

January 6, 2022. The search was based on abstracts and titles and was conducted between January 3 and January 6, 2022.

In this scoping review, we focused on applying AI models to multimodal medical data-based applications. The term

multimodal refers to combining medical imaging and EHR, as described in Preliminaries section. Therefore, our search string

incorporated three major terms connected by AND:( (“Artiﬁcial Intelligence” OR “machine learning” OR “deep learning”)

AND “multimodality fusion” AND (“medical imaging” OR “electronic health records”)). We used different forms of each term.

We provide the complete search string for all databases in Appendix 1 of the supplementary material.

Inclusion and exclusion criteria

We included all studies that fused EHR with medical imaging modalities using an AI model for any clinical application. As

AI models, we considered classical ML models, DL models, transfer learning, ensemble learning, etc as mentioned in the

search terms in Appendix 1 of the supplementary material. We did not consider studies that use classical statistical models

such as regression in our review. Our deﬁnition of imaging modalities is any type of medical imaging used in clinical practice,

such as MRI, PET, CT scans, and Ultrasound. We considered both structured and unstructured free-text patients’ data for

EHR modalities as described in Preliminaries section . Only peer-reviewed studies and conference proceedings were included.

Moreover, all included studies were limited to English language only. We did not enforce restrictions on types of disorders,

diseases or clinical tasks.

We excluded studies that used a single data modality. Also, we excluded studies that used different types of data from the

same modality, such as studies that only combined two or more imaging types (e.g. PET and MRI), as we considered this

single modality. Moreover, studies that integrated original imaging modalities with extracted imaging features were excluded as

this was still considered a single modality. Also, studies that combined multi-omics data modality were excluded. In addition,

studies that were unrelated to the medical ﬁeld or did not use AI-based models were excluded. We excluded reviews, conference

abstracts, proposals, editorials, commentaries, letters to editors, preprints, and short letters articles. Non-English publications

were also excluded.

4/20

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

ArticialIntelligence-BasedMethodsforFusionofElectronicHealthRecordsandImagingData*FaridaMohsen1,HazratAli1,NadyElHajj1,2,andZubairShah1,*1CollegeofScienceandEngineering,HamadBinKhalifaUniversity,QatarFoundation,34110Doha,Qatar2CollegeofHealthandLifeSciences,HamadBinKhalifaUniversity,QatarFoundation...

展开>> 收起<<

Artiﬁcial Intelligence-Based Methods for Fusion of Electronic Health Records and Imaging Data Farida Mohsen1 Hazrat Ali1 Nady El Hajj12 and Zubair Shah1.pdf

共20页,预览4页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Artiﬁcial Intelligence-Based Methods for Fusion of Electronic Health Records and Imaging Data Farida Mohsen1 Hazrat Ali1 Nady El Hajj12 and Zubair Shah1

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: