1 JoJoNet Joint-contrast and Joint-sampling-and-reconstruction Network for

2025-04-30 0 0 4.13MB 13 页 10玖币

侵权投诉

JoJoNet: Joint-contrast and

Joint-sampling-and-reconstruction Network for

Multi-contrast MRI

Lin Zhao, Xiao Chen, Eric Z. Chen, Yikang Liu, Dinggang Shen*, Terrence Chen, and Shanhui Sun*

Abstract—Multi-contrast Magnetic Resonance Imaging (MRI)

generates multiple medical images with rich and complementary

information for routine clinical use; however, it suffers from

a long acquisition time. Recent works for accelerating MRI,

mainly designed for single contrast, may not be optimal for

multi-contrast scenario since the inherent correlations among the

multi-contrast images are not exploited. In addition, independent

reconstruction of each contrast usually does not translate to

optimal performance of downstream tasks. Motivated by these

aspects, in this paper we design an end-to-end framework for

accelerating multi-contrast MRI which simultaneously optimizes

the entire MR imaging workﬂow including sampling, reconstruc-

tion and downstream tasks to achieve the best overall outcomes.

The proposed framework consists of a sampling mask generator

for each image contrast and a reconstructor exploiting the inter-

contrast correlations with a recurrent structure which enables

the information sharing in a holistic way. The sampling mask

generator and the reconstructor are trained jointly across the

multiple image contrasts. The acceleration ratio of each image

contrast is also learnable and can be driven by a downstream task

performance. We validate our approach on a multi-contrast brain

dataset and a multi-contrast knee dataset. Experiments show

that (1) our framework consistently outperforms the baselines

designed for single contrast on both datasets; (2) our newly de-

signed recurrent reconstruction network effectively improves the

reconstruction quality for multi-contrast images; (3) the learnable

acceleration ratio improves the downstream task performance

signiﬁcantly. Overall, this work has potentials to open up new

avenues for optimizing the entire multi-contrast MR imaging

workﬂow.

Index Terms—Joint optimization, Learnable acceleration ratio,

Multi-contrast MRI, Reconstruction, Sampling.

I. INTRODUCTION

MAGNETIC Resonance Imaging (MRI) is a widely

used and comprehensive medical imaging technique

that can offer high-quality images with both anatomical and

functional information. For modern MRI nowadays, multi-

contrast images with various and distinctive image contrasts

(see Figure 1(a)) are routinely acquired in practice. These

* Co-corresponding authors. (e-mail: Dinggang.Shen@gmail.com and

shanhui.sun@uii-ai.com)

L. Zhao is with the Department of Computer Science, University of Georgia,

Athens, GA 30602.

X. Chen, E. Chen, Y. Liu, T. Chen and S. Sun are with United Imaging

Intelligence, Cambridge, MA 02140.

D. Shen is with School of Biomedical Engineering, ShanghaiTech Uni-

versity, Shanghai 201210, China, and also with Department of Research

and Development, Shanghai United Imaging Intelligence Co., Ltd., Shanghai

200030, China.

Contribution from L. Zhao was carried out during his internship at United

Imaging Intelligence, Cambridge, MA.

Contrast 1 Contrast 2 Contrast 3 Contrast 4

(a)

Reconstruction

Network

𝑓(∙)

Undersampled

k-space Image

Reconstruction

⋮

⋮⋮

Single-contrast Reconstruction

JoJoNet: Joint-contrast and Joint-sampling-and-reconstruction

Sampling

Mask

Generator

Downstream

T2* Map

⋮

(b) Contrast 1 Contrast 2 Contrast 3 Contrast 4

(a)

Reconstruction

Network

𝑓(∙)

Undersampled

k-space Image

Reconstruction

⋮

⋮⋮

Single Contrast Reconstruction

Joint-contrast Reconstruction

JoJoNet: Joint-contrast and Joint-sampling-and-reconstruction

Sampler

Downstream

T2* Map

(b)

Fig. 1. (a) Example multi-contrast MRI images. (b) Overview of the

proposed JoJoNet. Contrast to a single-contrast scheme, the sampling mask

generator and reconstruction network are jointly optimized across multiple

image contrasts in JoJoNet. The training of the JoJoNet can be driven by

individual reconstruction quality or a downstream task performance.

multiple contrasts can depict and discriminate different tissues

and tissue conditions that reﬂect the underlying physiological

activities, which is commonly used in a complementary man-

ner for diagnosis or as the intermediate images for downstream

tasks such as image synthesis [1], [2]. On the other hand,

MRI is an intrinsically slow imaging modality since data are

collected in a point-by-point manner in k-space, a complex-

valued spatial-frequency space. The slow acquisition not only

limits its availability but also causes patient discomfort as well

as motion artifacts. The situation becomes even worse for the

multi-contrast imaging since it takes longer time to acquire

multiple images in a MRI scan. Therefore, accelerating MRI

is of decisive signiﬁcance and has attracted tremendous efforts

for decades.

A common way for accelerating MRI is acquiring only

partial k-space rather than the full k-space [3]–[5]. How-

ever, undersampling the k-space violates the Nyquist-Shannon

sampling theorem and results in blurring or aliasing arti-

facts. Various methods have been proposed to recover high-

ﬁdelity and artifact-free images from the undersampled k-

space, demonstrating great promises especially for those based

on deep learning [3], [6]–[10]. These efforts can be broadly

arXiv:2210.12548v2 [eess.IV] 27 Oct 2022

classiﬁed into three categories: a) optimizing the sampling

pattern that deﬁnes the set of k-space to be acquired for an easy

artifact removal [11]–[13]; b) optimizing the reconstruction

method for achieving better image quality [3], [8]; c) jointly

optimizing the sampling pattern and the reconstruction method

[5], [7], [14]. It is worth noting that the performance of the

reconstruction method signiﬁcantly depends on the sampling

pattern [7], [15]. A joint optimization of the two can adapt the

reconstruction method to the speciﬁc undersampling pattern

and make the undersampling pattern congruent with the re-

construction method, which can further improve reconstruction

quality.

Despite the remarkable progresses and contributions in

accelerating MRI, most works focused on accelerating single-

contrast imaging while only a few studies considered the

multi-contrast scenario [16], [17]. Although images with

different contrasts have diversiﬁed appearances, correlations

among them do exist. These images share the same underlying

anatomy and in many MR applications the multiple contrasts

record an evolving tissue property where the current contrast

depends on the preceding one. In a joint-contrast optimiza-

tion, these connections among the contrasts can be exploited

for sampling and reconstruction design, where data, either

acquired or recovered, from one contrast can contribute to

another. Furthermore, multi-contrast images are usually used

as intermediate images for downstream tasks such as T2* map

synthesis where contributions of each contrast are not equal

[1]. An optimal strategy would be to collect more data for

those critical contrasts and less on the others given a ﬁxed total

data amount. Thus, a global optimization of the whole multi-

contrast MRI workﬂow can better distribute the resources

among different MR sequences in scanning to achieve the

best overall outcomes (image quality and/or downstream task

accuracy).

In this paper, we propose a novel end-to-end framework,

joint-contrast joint-sampling-and-reconstruction network (Jo-

JoNet) for accelerating multi-contrast MRI which, for the ﬁrst

time, globally optimizes the entire MR imaging workﬂow

including the k-space undersampling, image reconstruction

and downstream task performance across multiple contrasts

(see Figure 1(b)). To do so, we learn the individual un-

dersampling pattern for each image contrast and optimize

them along with a specially designed reconstruction network,

Holistic Recurrent U-Net (HRU-Net), which fully exploits

the inter-contrast correlations to produce high-quality multi-

contrast reconstructions. Beyond image quality, JoJoNet can

also learn to distribute resources (amount of data to be

sampled, a.k.a acceleration ratio) across contrasts for optimal

downstream task outcomes. The proposed framework is exten-

sively evaluated on a multi-contrast brain dataset and a multi-

contrast knee dataset (fastMRI) [18]. The experimental results

show that (1) our multi-contrast MRI acceleration framework

consistently outperforms those designed for single contrast on

both datasets; (2) our HRU-Net demonstrates its effectiveness

and superiority in improving the reconstruction quality by

utilizing the contrast correlations; (3) in T2* map synthesis

downstream task, the scheme with learnable acceleration ratio

improves the accuracy of T2* map signiﬁcantly over the one

with ﬁxed acceleration ratio. The contributions of this paper

are summarized as follows:

•We introduce a framework for accelerating multi-contrast

MRI with global optimization of undersampling and

reconstruction across all contrasts.

•We propose a novel Holistic Recurrent U-Net to per-

form multi-contrast reconstruction with the exploitation

of inter-contrast correlations.

•We optimize MRI downstream task beyond image quality

and conduct extensive experiments, demonstrating the

proposed method’s superiority.

•Our study has the potential to open up new avenues for

optimizing the entire MR imaging workﬂow, providing a

feasible and effective way for clinical multi-contrast MR

applications.

II. RELATED WORKS

k-space sampling pattern. A lot of attempts have been

made in learning the sampling patterns for undersampling k-

space. [5], [7], [11], [12], [19]–[23]. For example, reinforce-

ment learning was adopted to learn the policy for actively

generating the sampling pattern based on greedy search [12]

and Double Deep Q-Networks [11]. [19] trained a progressive

sampler to emulate the policy distribution with a Monte Carlo

Tree Search (MCTS). [5] proposed an evaluator network to

perform the active sampling based on rating the quality gain

of each k-space measurement in reconstruction. Inspired by

the neural network pruning, [20] learned a weight for k-

space measurements and pruned those with less importance.

Other studies turned to the relaxation of binary mask to

make the sampling process differentiable [7], [21], [22]. For

instance, the recent LOUPE framework [7], [22] relaxed the

binarization of a probabilistic mask with sigmoid function to

enable backpropogation. [21] learned a continuous mask to

approximate the gradient of its binary version. It is noted that

those differentiable approaches can simultaneously optimize

the undersampling pattern and reconstruction, improving the

reconstruction quality.

Image reconstruction. Undersampled MRI reconstruction

has been widely studied in the literature. Compressed sensing

(CS) based methods incorporated additional a priori knowl-

edge, e.g., sparsity of medical images, to solve the ill-posed

reconstruction problem [24]–[29]. Recently, deep Convolu-

tional Neural Network (CNN) based methods demonstrated

their superior performances [3], [6], [30]–[32]. [3] presented

a cascaded CNN model with residual connections and a

data-consistency layer that ensures data ﬁdelity to improve

the reconstruction quality, which was widely adopted in the

subsequent studies [5], [32]. Another group of studies used

U-Net architecture [33] and its variants as the anti-aliasing

network [7], [12], [14], [21], [22], [31], [34]. Several studies

integrated the adversarial loss [35] to improve the human

perceptual reconstruction quality such as the sharpness [36]–

[40].

There are a few studies dealing with the multi-contrast MRI

reconstruction problem [16], [17], [41]–[44]. For example,

[16] proposed a reconstruction algorithm based on Bayesian

Recurrent

Cell

Recurrent

Cell



Undersampled

k-space Zero-filled

Images Reconstruction Network

Reconstructed

Images

IFFT











Sampling

Mask





Downstream

T2* Map

rec. loss

map. loss

Probabilistic

Mask



Learnable

Acceleration

Ratio





















Backpropagation





















Sampling Mask Generator



Fig. 2. Illustration of our proposed JoJoNet. The sampling mask generator generates sampling masks to undersample the k-space. After IFFT, the resulted

zero-ﬁlled images are fed into reconstruction network which has a recurrent architecture. The reconstructed images can be used for downstream task such as

synthesizing T2* map. The training of the whole framework can be driven by either the reconstruction loss or the T2* map loss.

CS. [41] reconstructed multiple T1/T2-weighted images of the

same anatomy based on a joint regularization of total variation

(TV) and group wavelet-sparsity. In [43], dictionary learning

was adopted to leverage the correlation between different

contrasts. Besides these CS-based methods, a recent study

proposed a spatial alignment network to register the refer-

ence contrast with the target contrast for better quality [17].

However, all the aforementioned approaches only focused on

the reconstruction. The joint sampling and reconstruction op-

timization and the consideration of multi-contrast downstream

task performance are still missing.

III. BACKGROUND AND NOTATION

Let y∈CN×Nrepresents the complex-valued fully-

sampled k-space. The image x∈CN×Ncan be reconstructed

by applying Inverse Fast Fourier Transform (IFFT) x=

F−1(y), and y=F(x)where Fis the Fast Fourier Transform

(FFT). To accelerate MRI, we only acquire a subset of the

full k-space by undersampling with the Cartesian acquisition

trajectory [6]. The undersampled k-space ˆycan be deﬁned as

ˆy=yM, where Mis a binary sampling mask determining

the sampling pattern, and represents the element-wise

product. The zero-ﬁlled image reconstruction ˆxis obtained

by ˆx=F−1(ˆy), which has blurring or aliasing artifacts due

to undersampling. An anti-aliasing/denoising function A(·)is

applied to get the ﬁnal reconstructed image ˆxrec =A(ˆx).

We parameterize the anti-aliasing function A(·)as a neural

network Aθ(·), i.e., the image reconstruction network. It is

noted that different sampling patterns have a huge impact on

the types of aliasing artifacts, and determine the performance

of reconstruction network implicitly. Thus, we aim to learn a

binary sampling mask Mc

pfor each image contrast to collabo-

rate with the reconstruction network for better reconstruction

performance. The multi-contrast MRI reconstruction problem

is then formulated as the joint optimization of sampling and

reconstruction network across multiple image contrasts:

arg min

θ,p

c=1

Lrec(Aθ(F−1(F(xc)Mc

p), xc)(1)

where Cis the number of image contrasts, xcis the cth

contrast MR image and Lrec is the loss function measuring

reconstruction quality such as the mean squared error (MSE)

or structural similarity index measure (SSIM) [45].

IV. METHOD

Figure 2 illustrates our approach. The framework consists

of a sampling mask generator and a reconstruction network.

The sampling mask generator aims to generate the binary

sampling masks for undersampling k-space for each image

contrast, respectively. The reconstruction network takes the

zero-ﬁlled images and the undersampled k-space as inputs,

and produces the high-ﬁdelity multi-contrast reconstructions.

These reconstructions can be the inputs to a downstream task,

formulated as a function f(·). In the task of T2* map synthesis,

the output of f(·)is the T2* map. The sampling mask

generator and the reconstruction network are jointly optimized

by image reconstruction quality (rec. loss in Figure 2) and/or

downstream task performance (map. loss in Figure 2).

A. Sampling mask generator

For a joint optimization, the sampling mask generator

should be differentiable to enable the backpropagation within

the whole framework. To do so, we extend the sampling

optimization methods in [7], [22] to be compatible with multi-

contrast scenario:

arg min

θ,p

c=1

Lrec(

Aθ(F−1(F(xc)rep(σs(uc≤pc)))), xc),

s.t. 1

dkpck=α

(2)

uc∈CNis a realization of a random vector that is uniformly

distributed on [0,1].pc∈CNis a “probabilistic mask” and

is binarized by uc≤pcwhich sets the value to 1 if the

inequality is satisﬁed and 0 otherwise. rep(uc≤pc)expands

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

1JoJoNet:Joint-contrastandJoint-sampling-and-reconstructionNetworkforMulti-contrastMRILinZhao,XiaoChen,EricZ.Chen,YikangLiu,DinggangShen*,TerrenceChen,andShanhuiSun*AbstractMulti-contrastMagneticResonanceImaging(MRI)generatesmultiplemedicalimageswithrichandcomplementaryinformationforroutineclinical...

展开>> 收起<<

1 JoJoNet Joint-contrast and Joint-sampling-and-reconstruction Network for.pdf

共13页,预览3页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

1 JoJoNet Joint-contrast and Joint-sampling-and-reconstruction Network for

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: