1 ISTA-Inspired Network for Image Super-Resolution Yuqing Liu1 Wei Zhang1 Weifeng Sun2 Zhikai Yu1 Jianfeng Wei1and Shengquan Li1

2025-04-30 0 0 817.89KB 8 页 10玖币

侵权投诉

ISTA-Inspired Network for Image Super-Resolution

Yuqing Liu1, Wei Zhang1,∗, Weifeng Sun2, Zhikai Yu1, Jianfeng Wei1and Shengquan Li1

1Pengcheng Laboratory, Shenzhen, China.

2Dalian University of Technology, Dalian, China.

Abstract—Deep learning for image super-resolution (SR) has

been investigated by numerous researchers in recent years. Most

of the works concentrate on effective block designs and improve

the network representation but lack interpretation. There are

also iterative optimization-inspired networks for image SR, which

take the solution step as a whole without giving an explicit

optimization step. This paper proposes an unfolding iterative

shrinkage thresholding algorithm (ISTA) inspired network for

interpretable image SR. Speciﬁcally, we analyze the problem of

image SR and propose a solution based on the ISTA method.

Inspired by the mathematical analysis, the ISTA block is devel-

oped to conduct the optimization in an end-to-end manner. To

make the exploration more effective, a multi-scale exploitation

block and multi-scale attention mechanism are devised to build

the ISTA block. Experimental results show the proposed ISTA-

inspired restoration network (ISTAR) achieves competitive or

better performances than other optimization-inspired works with

fewer parameters and lower computation complexity.

I. INTRODUCTION

Image super-resolution (SR), as one of the traditional im-

age restoration tasks, has been widely investigated by re-

searchers [1], [2]. Given a low-resolution (LR) image, the task

of image SR is to restore a corresponding high-resolution (HR)

instance with more details. There are numerous applications

considering the image SR, such as video deinterlacing [3] and

compression [4], remote sensing [5]–[7], EGG analysis [8],

and spatiospectral analysis [9].

Deep learning has demonstrated its amazing performance

in image restoration. There are numerous convolutional neu-

ral networks (CNNs) specially designed for image SR. SR-

CNN [10] is the ﬁrst CNN-based method for image SR. After

that, deeper and wider networks show their effectiveness with

better performance, such as VDSR [11], EDSR [12], RDN [13]

and RCAN [14]. Recent image SR networks usually develop

effective blocks for improving the network representation.

IMDN [15] and EFDN [16] utilize information distillation

mechanisms to build an efﬁcient network for fast and accurate

image SR. Cross-SRN [17] builds an edge-preserving network

with cross convolution. However, these works concentrate on

the block designs but lack interpretation, which limits the

performance.

Since image SR can be regarded as a classical optimization

task [18], there are also works considering building the image

SR network from the optimization perspective. IRCNN [18]

provides an iterative solution for the general image restoration

task and designs a CNN-based network to solve the denoising

prior. ISRN [19] develops an iterative network with the help

of the half-quadratic splitting (HQS) strategy. DPSR [20] and

USRNet [21] also achieve good performance on image SR

Fig. 1. An example comparison among different image super-resolution

methods.

inspired by the HQS strategy. There are also works building

the network by alternating direction method of multipliers

(ADMM). Plug-and-Play ADMM [22] regards the denoiser as

a network prior for different image restoration tasks. ADMM-

Net [23] provides an end-to-end network for the compression

sensing task. PSRI-Net [24] considers ADMM for SAR image

SR. Although these works provide an interpretable network

design, the CNN architectures just task the solution step as a

whole, without giving an explicit optimization step on how to

solve the denoising problem.

In this paper, we develop an unfolding network based on the

iterative shrinkage thresholding algorithm (ISTA). Different

from designing the CNN to directly solve the optimization

step, ISTA blocks are specially designed to conduct the image

restoration following the ISTA steps. In the ISTA block,

CNNs are utilized to adaptively learn the functions in the

feature space and speed up the optimization steps. To improve

the network representation, multi-scale exploration (MSE)

and multi-scale attention (MSA) mechanisms are utilized to

build the ISTA block. An ISTA-inspired restoration network

(ISTAR) is developed based on the ISTA block for effective

image SR. Experimental results show the proposed ISTAR can

achieve competitive or better performance than other works.

Compared with other optimization-inspired methods, ISTAR

achieves better performances with much fewer parameters and

lower computation complexity. Figure 1 shows an example

comparison among different image super-resolution methods.

Compared with state-of-the-art methods, our proposed ISTAR

can generate more satisfying textures that close to the HR

image.

The contributions of this paper can be concluded as follows:

arXiv:2210.07818v1 [eess.IV] 14 Oct 2022

•We analyze the image super-resolution task from the

optimization perspective and develop an ISTA block for

image super-resolution.

•We develop the multi-scale exploration and multi-scale

attention mechanism in the ISTA block, which improves

the network representation and boosts the performance.

•Experimental results show the proposed network

achieves competitive or better performance than other

optimization-based works with much fewer parameters

and lower computation complexity.

II. RELATED WORKS

A. Deep Learning for Image Super-Resolution

Deep learning has demonstrated its amazing performance

on various computer vision tasks. There are numerous con-

volutional neural networks (CNNs) specially designed for

image super-resolution (SR). SRCNN [10] is the ﬁrst CNN-

based image SR method composed of three convolution layers,

which follows a sparse-coding manner. After SRCNN, deeper

and wider networks has proposed to improve the restoration

performance. FSRCNN [25] increases the network depth and

decreases the input resolution, which makes the method faster

and more effective. VDSR [11] develops a very deep network

with residual connection to restore the high-resolution (HR)

images. EDSR [12] then utilizes the residual blocks in the

network and improves the network capacity. ESPCN [26]

provides a different upsampling strategy to restore the HR

images, which is more effective than the deconvolution op-

eration. Recently, researchers concentrate more on effective

block design for better restoration performance. RDN [13]

combines the residual connection [27] and densely connec-

tion [28], and develops a residual dense block for image SR.

After that, the researchers introduce the residual-in-residual

design with channel attention [29] for image SR and build an

effective network termed RCAN [14]. RFANet [30] expands

the residual connection and aggregates the residual features for

better information transmission. IMDN [15] and RFDN [16]

build the lightweight networks with the help of an information

distillation mechanism. SHSR [31] and MSRN [32] utilize hi-

erarchical exploration to further investigate the image features.

These works usually concentrate on the effective block designs

but neglect to analyze the image SR from the optimization

perspective.

B. Optimization-Inspired Image Super-Resolution

There are also optimization-inspired networks for inter-

pretable image SR. ADMM-Net [23] provides a good example

of dealing with the image restoration problem by the optimiza-

tion strategy and develops a CNN-based denoiser for plug-

and-play restoration. IRCNN [18] then analyzes the image

restoration with the help of the half-quadratic splitting (HQS)

strategy and recovers the image with a CNN-based denoiser

prior. After IRCNN, there are numerous HQS-based methods

for effective image SR. DPSR [20] proposes a different

observation model for image SR and uses kernel estimation

and CNN denoiser for plug-and-play image SR. USRNet [21]

develops an end-to-end network for different image SR tasks.

ISRN [19] devises an effective network for image SR under the

guidance of HQS and maximum likelihood estimation (MLE).

HSRNet [33] also investigates the HQS strategy and develops

a network for aliasing suppression image SR. However, these

works just take the solution as a whole and calculate it directly

by CNNs, without giving an explicit optimization step for each

iteration.

III. METHODOLOGY

In this section, we ﬁrst analyze the image super-resolution

(SR) from the optimization perspective and propose an itera-

tive solution with the help of ISTA. Then, we introduce the

designed end-to-end network ISTAR. After that, we discuss

the design of the ISTA block. Finally, the network settings are

described in detail.

A. ISTA for Image Super-Resolution

Given an low-resolution (LR) image ILR, the task of image

SR is to ﬁnd a corresponding image ISR, satisfying

ISR = arg min

ILR ||DISR −ILR||2

`+λ||ILR||1,(1)

where Dis the down-sampling matrix, and λis a weighting

factor. The prior term λ||ILR||1is utilized to introduce the

sparsity of the natural image.

To solve this function, we use ISTA to convert it into an

iterative manner. Then, the solution is

ISR

k+1 =Tλαk(ISR

k−αkDT(DISR

k−ILR)),(2)

where αkis the weighting factor for the k-th iteration and

T(·)is the soft-thresholding operation.

It can be found that the right hand side of Equation 2 has

two independent variables ISR

kand ILR. To make it clear for

understanding, we re-write Equation 2 as

ISR

k+1 =Tλαk((E−αkDTD)ISR

k−αkDTILR),(3)

where Eis the identity matrix.

In Equation 3, we can ﬁnd that DTILR is shared for every

iteration. In this point of view, we can calculate this term

before the ISTA optimization, and regard it as an invariant to

speed up the optimization.

B. Network Design

Figure 2 shows the entire network design of our ISTAR.

Firstly, the input image ILR is converted into the feature space

by one convolutional layer as

ILR =Conv(ILR).(4)

Then, two convolutional layers and one ReLU activation are

utilized to calculate the DTILR for ISTA steps, as shown in

the ﬁgure. There are Ksteps for ISTA optimization. For the

k-th step, there is

ISR

k=IST ABlock(DTILR,ˆ

ISR

k−1),(5)

where IST ABlock(·)is the designed ISTA block.

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

1ISTA-InspiredNetworkforImageSuper-ResolutionYuqingLiu1,WeiZhang1;,WeifengSun2,ZhikaiYu1,JianfengWei1andShengquanLi11PengchengLaboratory,Shenzhen,China.2DalianUniversityofTechnology,Dalian,China.AbstractDeeplearningforimagesuper-resolution(SR)hasbeeninvestigatedbynumerousresearchersinrecentyears.M...

展开>> 收起<<

1 ISTA-Inspired Network for Image Super-Resolution Yuqing Liu1 Wei Zhang1 Weifeng Sun2 Zhikai Yu1 Jianfeng Wei1and Shengquan Li1.pdf

共8页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

1 ISTA-Inspired Network for Image Super-Resolution Yuqing Liu1 Wei Zhang1 Weifeng Sun2 Zhikai Yu1 Jianfeng Wei1and Shengquan Li1

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: