3D Matting A Benchmark Study on Soft Segmentation Method for Pulmonary Nodules Applied in Computed Tomography_2

2025-04-28 0 0 2.89MB 21 页 10玖币

侵权投诉

3D Matting: A Benchmark Study on Soft Segmentation Method for

Pulmonary Nodules Applied in Computed Tomography

Lin Wanga,b,c, Xiufen Yea,∗, Donghao Zhangb, Wanji Hec, Lie Jub,c, Yi Luod, Huan Luoe, Xin

Wangc, Wei Fengb,c, Kaimin Songc, Xin Zhaoc, Zongyuan Geb,c,∗

aCollege of Intelligent Systems Science and Engineering, Harbin Engineering University, Harbin, China

bMonash Medical AI Group, Monash University, Clayton, Australia

cBeijing Airdoc Technology Co., Ltd., Beijing, China

dChongqing Hospital of Traditional Chinese Medicine, Chongqing, China

eChongqing Renji Hospital of Chinese Academy of Sciences, Chongqing, China

Abstract

Usually, lesions are not isolated but are associated with the surrounding tissues. For example, the

growth of a tumour can depend on or inﬁltrate into the surrounding tissues. Due to the pathological

nature of the lesions, it is challenging to distinguish their boundaries in medical imaging. However,

these uncertain regions may contain diagnostic information. Therefore, the simple binarization of

lesions by traditional binary segmentation can result in the loss of diagnostic information. In this

work, we introduce the image matting into the 3D scenes and use the alpha matte, i.e., a soft mask,

to describe lesions in a 3D medical image. The traditional soft mask acted as a training trick to

compensate for the easily mislabelled or under-labelled ambiguous regions. In contrast, 3D matting

uses soft segmentation to characterize the uncertain regions more ﬁnely, which means that it retains

more structural information for subsequent diagnosis and treatment. The current study of image

matting methods in 3D is limited. To address this issue, we conduct a comprehensive study of 3D

matting, including both traditional and deep-learning-based methods. We adapt four state-of-the-

art 2D image matting algorithms to 3D scenes and further customize the methods for CT images

to calibrate the alpha matte with the radiodensity. Moreover, we propose the ﬁrst end-to-end deep

3D matting network and implement a solid 3D medical image matting benchmark. Its eﬃcient

counterparts are also proposed to achieve a good performance-computation balance. Furthermore,

there is no high-quality annotated dataset related to 3D matting, slowing down the development

of data-driven deep-learning-based methods. To address this issue, we construct the ﬁrst 3D med-

ical matting dataset. The validity of the dataset was veriﬁed through clinicians’ assessments and

downstream experiments. The dataset and codes will be released to encourage further research1.

Keywords: 3D Matting, Pulmonary nodules, Soft Segmentation, Thoracic CT, Uncertainty

2022 MSC: 00-01, 99-00

Abbreviations and Symbols

The following abbreviations are used in this manuscript:

1Url for codes and dataset: https://github.com/wangsssky/3DMatting.

∗Corresponding author

Email address: zongyuan.ge@monash.edu (Zongyuan Ge)

yexiufen@hrbeu.edu.cn (Xiufen Ye)

wanglin.mailbox@gmail.com (Lin Wang)

Preprint submitted to Computers in Biology and Medicine October 12, 2022

arXiv:2210.05104v1 [eess.IV] 11 Oct 2022

Term Description

3D Three-dimensional

3DMM 3D Medical Matting network

AUROC Area Under the Receiver Operating Characteristic

CF Closed-Form matting

Conn. Connectivity Error

CT Computed Tomography

DL Deep-Learning

FLOPs FLoating-point OPerations

Grad. Gradient Error

GT Ground Truth

HU Hounsﬁeld Unit

IF Information-Flow matting

KNN KNN matting

LB Learning-Based matting

MRI Magnetic Resonance Imaging

MSE Mean Squared Error

PET Positron Emission Tomography

SAD Sum of Absolute Diﬀerences

The following symbols are used in this manuscript:

Term Description

αalpha matte

I2D image

Iidentity matrix

Bbackground

Ddiagonal matrix indexing the position of the constraints

Fforeground

J(·)cost function

Lloss

Lmatting Laplacian matrix

Mbinary mask

Rbbackground region

Rfforeground region

Ruunknown region

Sconstraints of traditional matting methods

V3D volume

δ(·)kronecker delta

a coeﬃcient of regularizer

λweight of the constraints

η, θ loss weighting coeﬃcients

µmean

σvariance

wlocal window

1. Introduction

Due to image noises, the occlusion of human tissues, the principles of medical imaging, and the

anatomical structure characteristics of lesions, fuzzy boundaries are almost inevitable and ubiquitous

in medical images [1, 2, 3, 4]. Binary masks are the most commonly used to describe diseased areas in

segmentation tasks. However, the fuzzy boundary hinders the accurate recognition of the lesion area,

which results in uncertainty in the labelling process, and adversely aﬀects diagnoses and treatments

downstream. Figure 1 shows some examples of fuzziness in medical images. The morphology of the

uncertain regions around the pulmonary nodules is hard to be described by the binarized boundaries

with binary masks, which may result in the loss of information.

Many studies have been put forward to reduce the inﬂuence of ambiguity in medical image

segmentation. Multi-annotated datasets are proposed to reduce labelling bias [5, 6], and probabilistic

models attempt to describe the lesion distribution [7, 8, 1, 9], etc. However, due to the complicated

reasons for ambiguity, the elimination of ambiguity is still challenging.

Instead of identifying perfect lesion boundaries that are diﬃcult to annotate using binary-style

masks, medical matting makes use of the information contained in the ambiguous regions [2]. It

introduces the image matting technique into the ﬁeld of medicine, and regards medical images as

a mixture of lesions and healthy tissues. This mixing factor is known as alpha matte. In such a

way, the fuzzy boundary area is regarded as the transition region from pure lesions to pure healthy

tissues. The alpha matte can be used as a soft mask to describe the anatomical structures of lesions

more comprehensively than a binary mask.

In addition, the fuzzy areas contain important diagnostic information. For example, in thoracic

CT images, the fuzzy area around lung nodules in the lung CT images may refer to two kinds of

(a) (b) (c) (d) (e) (f)

Figure 1: Examples of describing lung nodules [5] with binary masks and alpha mattes. We show two pulmonary

nodule samples (Left) in multiple views (Right). Due to the blurred boundaries of the nodules (indicated by red

arrows), it is challenging for the binary masks labelled by diﬀerent clinicians (b)-(e) to achieve agreement (indicated

by green arrows). Moreover, manual labelling of lesions is done slice by slice, making it diﬃcult to maintain structural

continuity of lesions between slices. However, the ground truth soft masks (f), i.e., the alpha mattes, have a better

ability to represent the details of the lesions, with better continuity in between the slices. The redder the color, the

more likely it is to be part of a lesion.

borders, the indistinct border and amorphous ground-glass shadow, which are vital for the clin-

ical staging of nodules [10]. Therefore, by keeping more details, medical matting provides more

information for downstream tasks (such as nodule grading and precise radiotherapy) than binary

segmentation.

The previous works focused on 2D medical images. The 3D medical image matting research is

very limited in quantity and methodology [11, 12, 13, 14, 15, 16]. To the best of our knowledge,

only [17, 18, 19] have touched upon the problem of the 3D matting, and these methods are all derived

from 2D CF [20]. At present, there is no DL-based method that has been investigated. To address

this, we adapt the matting methods to 3D medical image scenes, including four traditional methods

and a DL-based method, as more accurate approaches for lesion segmentation and description,

especially for fuzzy areas.

As we know, this is the ﬁrst work to explore the possibility of matting to solve the problem of

fuzzy boundaries in 3D medical image segmentation. Furthermore, this is the ﬁrst attempt to deploy

DL-based matting, rather than traditional matting methods, to 3D image data to realize automatic

inference without manual intervention.

Firstly, due to the lack of available datasets in the 3D matting scenario, we created a publicly

accessible and clinically validated dataset of pulmonary nodules based on the LIDC-IDRI [5]. We

hope that it can beneﬁt the 3D matting research community. Using this dataset, we verify that the

alpha matte contains more diagnostic information than the binary mask quantitatively. Furthermore,

four state-of-the-art traditional 2D matting methods are adapted to 3D scenarios and they are further

customized to CT, making the dataset built in a semi-automatic approach. Finally, the 3D medical

matting network, a benchmark DL-based 3D model, is proposed as an end-to-end automatic matting

network for pulmonary nodules. At the same time, we optimize the benchmark model by simplifying

the network structure and introducing the ghost module to achieve a trade-oﬀ between performance

and computation [21].

Our contributions are summarized as follows:

•This is the ﬁrst comprehensive study of matting methods in 3D medical scenes, especially

for lung nodules in CT. Through qualitative clinical evaluation and quantitative downstream

experiments, we have veriﬁed that alpha matte retains more structural and diagnostic infor-

mation of lesions than a binary mask.

•The ﬁrst DL-based trimap-free matting benchmark network (3DMM) is proposed. The 3DMM

contains two auxiliary mask branches that predict the overlap and union of the multi-labelled

binary masks, providing guidance information for alpha matte prediction. Its inference does

not require human participation and is more convenient for real-world applications.

•Moreover, we propose several DL-based methods with higher computational eﬃciency to

achieve a better balance between performance and computation for broader applications.

•Four state-of-the-art traditional 2D matting methods are adapted to 3D scenes with further im-

provements to CT images, which provide the methodological basis for the eﬃcient construction

of 3D medical matting datasets in a semi-automatic way.

•The ﬁrst clinically validated 3D matting dataset speciﬁcally for 3D medical images is proposed

and publicly available to the research community to address the lack of dataset in related

studies.

The rest of the manuscript is organized as follows: Section 2 provides comprehensive background

information on image matting and its applications in medical scenarios. Section 3 introduces the

3D matting benchmark dataset. Section 4 introduces the traditional 3D matting methods and the

optimization speciﬁc for CT images. The trimap-free DL-based methods for 3D medical images

are proposed in Section 5. Section 6 presents the experiments to illustrate the advantages of alpha

mattes to binary masks and investigate the performance of DL-based 3D matting methods. Potential

uses and limitations are discussed in Section 7. Section 8 is a summary of the whole paper.

2. Related Work

2.1. Soft Segmentation

There have been some studies on soft segmentation in the medical ﬁeld. For example, Kats et

al. claimed that pixels around lesions also have diagnostic information and assigned them a soft

label in training to improve segmentation performance [22]. Dai et al. used soft masks in data

augmentation, which mixed the lesion with the image by using a soft coeﬃcient at the boundaries of

the lesions [23]. However, these soft masks can not reﬂect the structural information of the lesions.

2.2. Image Matting

Image matting uses the mixing coeﬃcient α, also known as the alpha matte, to decompose

the image Ito foreground Fand background B, or lesions and its surrounding tissues in medical

images [2, 24, 25, 26, 27, 28, 20, 29, 30, 31]. It can be deﬁned as:

Ii=αiFi+ (1 −αi)Bi.(1)

Image matting is a particular type of image segmentation that uses the alpha matte, a soft mask, to

describe the target. It is beneﬁcial for image/video editing when dealing with the blurred boundaries.

Compared with binary masks, alpha mattes can better depict lesions with more details [2].

There are four terms in Eq. 1. However, only Iis known. Therefore, the matting is an ill-posed

problem and is challenging to be solved directly [32]. A common practice is to reduce the problem

complexity by introducing a prior map called trimap as constraints, indexing the regions of the

foreground Rf, background Rb, and unknown Ru. Therefore, according to whether the trimap is

used, the matting methods can be categorised into trimap-based and trimap-free methods. Com-

pared with trimap-free methods, the trimap-based methods generally achieve better performance as

the trimap provides additional information. However, generating the trimap requires extra-manual

labour and limits practical application [33].

According to the usage of DL techniques, image matting methods can be divided into traditional

methods [24, 26, 27, 20] and DL-based methods [25, 28, 29, 31]. To the best of our knowledge,

almost all of the traditional matting methods are trimap-based, which requires a trimap or scribbles

that provide prior information of the foreground and background. Such information is used to infer

the values of alpha matte in the uncertain areas by deriving pre-deﬁned rules or assumptions. For

instance, the CF assumes that the foreground and background are locally smooth and that the image

in a small window can be represented by them with a linear function [20].

Due to the latest developments in artiﬁcial intelligence, the DL-based methods, driven by ex-

tensive data, tend to have better performance and a more comprehensive range of applications

than traditional methods. Moreover, the trimap-free methods become possible as the prior infor-

mation can be learned by the context-aware networks, which makes matting automatic and easy

to use [34, 33]. However, the traditional methods are still popular because they require no tedious

training and thus have better generalization in new applications with insuﬃcient data. The readers

can refer to [30] for a more comprehensive understanding of image matting.

2.3. Matting in Medical Applications

Image matting can produce ﬁne boundaries as it infers the uncertain regions by the prior infor-

mation from foregrounds and backgrounds. Therefore, most image matting applications in medical

scenarios regarded it as a post-processing method to improve the segmentation performance. Zeng

et al. reﬁned edges of the segmentation results by using the CF [20] directly on each slice of the

PET volume data [11]. The boundary pixels were regarded as a mixture of the foreground (tumours)

and background (normal tissue). Cheng et al. deployed an optimized CF [20] on medical images

of diﬀerent imaging modalities, which improved the segmentation performance [12]. Zhao et al.

utilized the segmentation result to create a trimap via the bi-level thresholding and improved the

segmentation performance of the uncertain regions further by minimizing a local matting loss [13].

Diﬀerent from the methods mentioned above, Medical Matting introduced the alpha matte as

an alternative to the binary mask, which was more expressive and could describe the lesion in more

detail. Moreover, the alpha matte reﬂected the uncertainty of the lesion property, expanding the

scope of matting in medical scenes [2].

2.4. Matting in 3D Scenarios

Matting in 3D is not well studied as the majority of the natural images are 2D images. However,

a large proportion of the images of various modalities in medical scenarios are in 3D, therefore there

is a strong demand for 3D matting-based methods to be proposed in medical applications. The

research on 3D medical matting is limited. All of them are based on traditional methods and are

used as an auxiliary to binary segmentation. For example, Zhong et al. adapted CF [20] to 3D,

and used alpha mattes as probability maps of tumours in calculating the region cost for PET-CT

co-segmentation [17, 18]. Liu et al. also used a 3D CF for organ model extraction for Virtual Human

Project with signiﬁcant eﬃciency improvement [19].

Most existing methods still aim at obtaining a better binary mask but do not touch the root

cause of the matting problem, that is, using soft masks to describe the lesions in more detail. In this

manuscript, by contrast, the matting is used to generate a depictive representation of the lesions.

Compared with the 2D matting methods applied in 3D scenes, we provide a series of native 3D

matting methods rather than a combination of slice-by-slice processing by 2D matting methods.

Therefore, the structural continuities among diﬀerent slices can be maintained. Compared with

the important role of 3D data in medical diagnosis, the current research on 3D matting is not

comprehensive and in-depth. In this work, we investigate the traditional matting methods in 3D

medical images and provide a trimap-free solution based on DL.

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

3DMatting:ABenchmarkStudyonSoftSegmentationMethodforPulmonaryNodulesAppliedinComputedTomographyLinWanga,b,c,XiufenYea,,DonghaoZhangb,WanjiHec,LieJub,c,YiLuod,HuanLuoe,XinWangc,WeiFengb,c,KaiminSongc,XinZhaoc,ZongyuanGeb,c,aCollegeofIntelligentSystemsScienceandEngineering,HarbinEngineeringUniversit...

展开>> 收起<<

3D Matting A Benchmark Study on Soft Segmentation Method for Pulmonary Nodules Applied in Computed Tomography_2.pdf

共21页,预览5页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

3D Matting A Benchmark Study on Soft Segmentation Method for Pulmonary Nodules Applied in Computed Tomography_2

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: