Real Image Super-Resolution using GAN through modeling of LR and HR process Rao Muhammad Umer

2025-04-29 0 0 1.53MB 8 页 10玖币

侵权投诉

Real Image Super-Resolution using GAN through modeling of LR and HR

process

Rao Muhammad Umer,

Institute of AI for Health (AIH),

Helmholtz Munich, Germany.

engr.raoumer943@gmail.com

Christian Micheloni,

Department of Mathematics and Computer Science,

University of Udine, Italy.

christian.micheloni@uniud.it

Abstract

The current existing deep image super-resolution meth-

ods usually assume that a Low Resolution (LR) image

is bicubicly downscaled of a High Resolution (HR) im-

age. However, such an ideal bicubic downsampling process

is different from the real LR degradations, which usually

come from complicated combinations of different degrada-

tion processes, such as camera blur, sensor noise, sharp-

ening artifacts, JPEG compression, and further image edit-

ing, and several times image transmission over the internet

and unpredictable noises. It leads to the highly ill-posed

nature of the inverse upscaling problem. To address these

issues, we propose a GAN-based SR approach with learn-

able adaptive sinusoidal nonlinearities incorporated in LR

and SR models by directly learn degradation distributions

and then synthesize paired LR/HR training data to train

the generalized SR model to real image degradations. We

demonstrate the effectiveness of our proposed approach in

quantitative and qualitative experiments.

1. Introduction

Single image super-resolution (SISR) aims to restore the

high-resolution (HR) image from its low-resolution (LR)

image counterpart. SISR problem is a fundamental low-

level vision and image processing problem with various

practical applications in e.g., satellite imaging, medical

imaging, astronomy, remote sensing, surveillance, image

compression, environment and climate change monitoring,

mobile photography, image / video enhancement, and se-

curity and surveillance imaging, etc. With the increasing

amount of HR images / videos data on the internet, there is

a great demand for storing, transferring, and sharing such

large sized data with low cost of storage and bandwidth re-

sources. Moreover, the HR images are usually downscaled

to easily ﬁt into display screens with different resolution

GLR GSR

LR Learning SR Learning

(HR) (fake

LR)

(Real

LR)

(fake

LR)

(Real

HR)

(fake

SR)

Figure 1: The structure of our proposed real-world SR ap-

proach setup. In the LR Learning part, we train the LR

generator network GLR in a GAN framework, where our

goal is to learn the real LR (y) corruptions/degradations.

Then, we use the synthesized paired LR/HR data by the

GLR model to train the generalized SR model GSR in the

SR Learning part. Both the GLR and GSR generators uti-

lize the modiﬁed residual structure (refer to the sections 4

and 5for more details).

while retaining visually plausible information. The down-

scaled LR counterpart of the HR can efﬁciently utilize lower

bandwidth, storage save, and easily ﬁt to various digital dis-

plays. However, some details are lost and sometimes visible

artifacts appear when users downscale and upscale the dig-

ital contents.

Mathematically, SISR is described as a linear forward

observation model [19,21] with the following image degra-

dation process:

y= (H⊗˜

x)↓s+η, (1)

where, yis an observed LR image, His a down-sampling

operator (unknown) that convolves (⊗) with a latent HR

image ˜

xand resizes it by a scaling factor s, and ηis con-

sidered as an i.i.d additive white Gaussian noise (AWGN)

of variance σ2,i.e.,η∼ N 0, σ2. However, in real-

world settings, ηalso accounts for all possible errors dur-

ing the image acquisition process that include inherent sen-

sor noise, stochastic noise, compression artifacts, and the

possible mismatch between the forward observation model

and the camera device. The operator His usually ill-

arXiv:2210.10413v1 [cs.CV] 19 Oct 2022

conditioned or singular due to the presence of unknown

noise realization (η) that turn the SISR to a highly ill-posed

nature of inverse problems. Since, due to ill-posed nature,

there are many possible solutions thus regularization is re-

quired to select the most plausible ones.

Recently, numerous works have been addressed towards

the task of SISR [7,14,32,33,30,13,34,24,20,18,35,12,

5] and real-world SISR [9,28,16,3,19,21,25]. Most of the

SISR methods assume usually bicubic downsampling pro-

cess, which is different from the real LR degradations. The

real-world SISR methods try to solve the problem by uti-

lizing data distribution learning using the GAN [4] frame-

work. However, they do not generalize well to the real

complex degradation, which usually come from the compli-

cated degradation processes, i.e., sensor noise, camera blur,

sharping artifacts, JPEG compression, and further image

editing, and several times image transmission over the in-

ternet. In the most recent works [27,31], the authors aim to

restore general real-world LR images by synthesizing train-

ing pairs with a more practical degradation process. As the

real-world degradation space is much larger/complex, the

synthetic modeling also becomes challenging. Moreover,

the generators (i.e., LR/HR) require a more powerful capa-

bility to model the complex training data, while the gradi-

ents needs to be more accurate for local detail enhancement

with some sophisticated nonlinearities inside the network.

In this work, we proposed the GAN-based real image SR

approach that solves the problem by modeling the LR/HR

process with adaptive sinusoidal activitions (i.e., better rep-

resent the complicated signals) and thus synthesize the more

realistic paired LR/HR data to train the generalized SR

model for the real SR task. The structure of our proposed

real-world SR approach setup is shown in Fig. 1. In the

LR learning, we train the LR network (GLR) with modiﬁed

residual structure (i.e., incorporating the sinusodial non-

linearities) in a GAN-framework [4] to generate the real-

istic LR images as the corruptions/degradations of the real

LR images (y). After that, we use the synthesized paired

LR/HR data to train the generalized SR model in the SR

Learning part. The SR network (GSR) is trained in a GAN-

framework [4] with the modiﬁed residual structure to super-

resolve the LR images.

We evaluate our proposed SR method on the Real-World

Super-resolution (RWSR) dataset [17] to show the effec-

tiveness of our approach through the quantitative and qual-

itative experiments. We summarize our contributions in

three fold as:

1. We propose an end-to-end deep SRResCSinGAN for

the real-world SR task. Instead of using traditional

bicubic downsampling or the existing deep LR degra-

dation methods, we synthesize the paired training data

with a more practical image corruptions/degradations

by modeling the LR/HR process.

2. By exploiting the sinusoidal non-linearities, we em-

ploy the modiﬁed residual network structure incorpo-

rated in both LR and SR learning stages, which bet-

ter models the underlying complex signals i.e., real LR

and HR process.

3. Our proposed approach achieve better quantitative and

visual performance in terms of PSNR/SSIM/LPIPS

(refer to Tables 1and 2).

2. Related Work

2.1. Real World SISR methods

Recently, numerous works [7,14,32,33,30,13,34,24,

20,18,35,12,5] have addressed the task of SISR using

deep CNNs for their powerful feature representation capa-

bilities. The SISR methods mostly rely on the PSNR-based

metric by optimizing the L1/L2losses with blurry results

in a supervised way, while they do not preserve the visual

quality with respect to human perception. Moreover, the

above-mentioned methods are deeper or wider CNN net-

works to learn non-linear mapping from LR to HR with

the ideal bicubic downsampling, while neglecting the real-

world settings.

For the real image SR task, several attempts [9,28,16,

3,19,21,25] have done to solve for realistic LR degrada-

tion. However, the real SR methods still suffer unpleasant

artifacts and challenging for learning ﬁne-grained corrup-

tions/degradations with unpaired data. Our approach takes

into account the real-world settings by increasing its appli-

cability in practical scenarios.

2.2. Blind / Non-Blind degradation models

Classical degradation model (refer to Eq. (1)) is mostly

used in the blind / non-blind deep SISR methods. The

common choice, in the existing SISR degradation models,

usually consist of a sequence of blur kernel (i.e., Gaus-

sian/motion), downsampling (i.e., bicubic, bilinear, nearest-

neighbor), and noise addition (i.e., AWGN). In the existing

deep SISR methods [27,31], they attempt to explicit model

the real-world degradation to super-resolve the real LR im-

ages. But, yet the real-world degradations are too complex

to be explicitly modeled. Therefore, implicit modeling us-

ing GAN framework within the network is a suitable choice

to synthesize more practical degradations.

3. Proposed Method

3.1. Problem Formulation

By referencing to the Eq. (1), the recovery of xfrom

ymostly relies on the variational approach for combining

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

RealImageSuper-ResolutionusingGANthroughmodelingofLRandHRprocessRaoMuhammadUmer,InstituteofAIforHealth(AIH),HelmholtzMunich,Germany.engr.raoumer943@gmail.comChristianMicheloni,DepartmentofMathematicsandComputerScience,UniversityofUdine,Italy.christian.micheloni@uniud.itAbstractThecurrentexistingdeep...

展开>> 收起<<

Real Image Super-Resolution using GAN through modeling of LR and HR process Rao Muhammad Umer.pdf

共8页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Real Image Super-Resolution using GAN through modeling of LR and HR process Rao Muhammad Umer

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: