How Real is Real Evaluating the Robustness of Real-World Super Resolution Athiya Deviyani 1Efe Sinan Hoplamaz 1Alan Savio Paul 1 Abstract

2025-05-06 0 0 6.67MB 9 页 10玖币

侵权投诉

How Real is Real: Evaluating the Robustness of Real-World Super Resolution

Athiya Deviyani * 1 Efe Sinan Hoplamaz * 1 Alan Savio Paul * 1

Abstract

Image super-resolution (SR) is a ﬁeld in com-

puter vision that focuses on reconstructing

high-resolution images from the respective low-

resolution image. However, super-resolution is

a well-known ill-posed problem as most meth-

ods rely on the downsampling method performed

on the high-resolution image to form the low-

resolution image to be known. Unfortunately,

this is not something that is available in real-life

super-resolution applications such as increasing

the quality of a photo taken on a mobile phone.

In this paper we will evaluate multiple state-of-

the-art super-resolution methods and gauge their

performance when presented with various types

of real-life images and discuss the beneﬁts and

drawbacks of each method. We also introduce a

novel dataset, WideRealSR, containing real im-

ages from a wide variety of sources. Finally,

through careful experimentation and evaluation,

we will present a potential solution to alleviate

the generalization problem which is imminent in

most state-of-the-art super-resolution models.

1. Introduction

Super-resolution is a process of reconstructing high-

resolution images from their low-resolution counterparts.

However, it is well-known that super-resolution is an ill-

posed problem as a large amount of current state-of-the-art

methods are trained only to reconstruct images artiﬁcially

downsampled by a simple and uniform degradation (Dong

et al.,2014;Tai et al.,2017;Tong et al.,2017). One of

the most commonly used downsampling methods is bicubic

downsampling. While these state-of-the-art methods pro-

duce impressive results on bicubically downsampled images

(Wang et al.,2018;Dai et al.,2019), they perform poorly on

low-resolution images downsampled by an unknown, realis-

tic image degradation method (Cai et al.,2019a;Lugmayr

et al.,2019a). Such realistic and natural degradations are

Equal contribution

School of Informatics, University of Edin-

burgh, Scotland, United Kingdom.

what we are often presented with through cameras, smart-

phones, TV, etc.

One of the reasons why natural image super-resolution is

a big challenge is that there is a limited number of natu-

ral low-resolution and corresponding high-resolution image

pairs. Therefore, methods such as bicubic downsampling

are employed to artiﬁcially generate LR-HR pairs (Wang

et al.,2020). Other work has also involved using syntheti-

cally generating real low-resolution to high-resolution pairs

through unsupervised learning or blind kernel estimation

(Lugmayr et al.,2019b;Zhou & Susstrunk,2019). Oth-

ers have shifted their focus away from generating LR-HR

pairs to simulating more complex image degradation mod-

els (Lugmayr et al.,2019c). These methods have proven to

improve current super resolution models that were trained

on only bicubically downsampled images to generalise bet-

ter to images found in ‘the wild’. Supervised methods for

“in-the-wild” real images have seen recent signiﬁcant ad-

vancements compared to unsupervised methods, which are

more practical in the real-world setting as it is difﬁcult to

obtain real LR-HR image pairs. We aim to perform detailed

analysis of existing state-of-the-art methods to develop a

better understanding of their strengths and weaknesses.

It is crucial that the trained super-resolution methods are

able to generalise to real-world images, which are natu-

rally occurring low-resolution images that have unknown

and complicated downsampling kernels and noises. Exam-

ples of their applications include facial recognition in video

surveillance, remote sensing, and healthcare applications

such as detecting anomalies in medical images. We hope

that, with this paper, we will be able to contribute to the

development of SR models that are robust to the varied ad-

verse effects (such as noises) of different equipment in these

applications.

In this paper we will compare and contrast the perfor-

mance of several state-of-the-art methods on real-world

low-resolution images, and explore methods that could im-

prove their generalization performance. There are multiple

metrics which indicate how well the network reconstructs

the LR image to its HR version. However, calculating dis-

tortion metrics such as Peak Signal-to-Noise Ratio (PSNR)

and Structural Similarity Index (SSIM) (Wang et al.,2004)

is not possible in some cases if we are using real-world

arXiv:2210.12523v1 [cs.CV] 22 Oct 2022

Evaluating the Robustness of Real-World Super Resolution

LR images for testing because no ground-truth is available.

These measures are objective and do not take any subjective

evaluation of human perception. We will thus use other

perceptual-based metrics for qualitative evaluation on our

dataset which consists of real-world images from a variety

of sources.

Additionally, we note that there is a lack of datasets that con-

tain a wide variety of naturally downsampled, real images

from different types of sources. Having a wide variety of

sources will help obtain more accurate measurements of gen-

eralisability of real-world super-resolution models. Popular

existing datasets such as DPED and RealSR do not achieve

this. DPED contains images taken from 3 smartphone cam-

eras, which together might have different noises compared

to TV streams, CCTV footage and satellite images. RealSR

contains images taken from the same camera and only helps

evaluate the performance of models on the particular down-

sampling method of such cameras. In this paper, we aim to

introduce a carefully curated dataset, WideRealSR, that can

be used as a test set for thoroughly evaluating real-world

performance.

In short, the goal of this paper can be summed as the follow-

ing:

Evaluate the generalisability of various supervised and

unsupervised super-resolution models,

Identify the reasons why the models perform poorly or

well, and

Investigate a method to devise a practical solution to

potentially alleviate the generalization problem.

2. Dataset and task

As mentioned in the previous section, most of the existing

datasets used in the real-world super-resolution ﬁeld lack a

diverse range of sensor noises. The DPED dataset (Ignatov

et al.,2017a), for example, only contains images from 3

different smartphone cameras - iPhone 3GS, BlackBerry

Passport, and Sony Xperia Z - as the Low-Resolution images.

The respective High-Resolution counterparts are captured

using a Canon 70D DSLR. This dataset provided support for

the authors to present an end-to-end deep learning approach

that bridges the gap between ordinary photos into higher-

quality DSLR-like images. The authors have proposed to

learn the translation function using a residual convolutional

neural network that improves both color rendition and image

sharpness (Ignatov et al.,2017b). Since its release, the

DPED dataset has been the base dataset for numerous super-

resolution model proposals, including the award-winning

RealSR model.

A popular alternative to the DPED dataset is DIV2K

(Agustsson & Timofte,2017), containing 800 high-

resolution images and their corresponding low-resolution

images that are obtained artiﬁcially through a variety of

downsampling methods. Recent work has emerged (Cai

et al.,2019b) in the super-resolution ﬁeld which aim to

identify the best image downgrading methods that best gen-

eralise to images in the real world, i.e. images with unknown

sensor noises. The DIV2K dataset is commonly used in this

particular ﬁeld of super-resolution.

To evaluate the performance of existing super-resolution

models, it is not sufﬁcient to sample from a limited num-

ber of sensor noises and downsampling techniques. This

is because, as mentioned previously, images in the ‘real-

world’ inherently come with arbitrary kernels and noises.

Thus, following this limitation, we decided to meticulously

curate a dataset to be used for evaluating several super-

resolution models. We have obtained 1-3 images for 35

different sensor noises, from sources such as (but not lim-

ited to) Google Maps, satellites, drones, microscopes, smart-

phones (iPhone, BlackBerry, Samsung Galaxy), WhatsApp,

Facebook, tablets (iPad, Samsung Galaxy Tab), BBC broad-

casts. To maintain the ‘real-world’ factor, we decided to do

minimal preprocessing and only cropping the image when

necessary. We call this dataset WideRealSR. We have dis-

played sample images from our dataset collected from 10

different sources in Figure 1.

The main task in this paper is to perform real-world super-

resolution using state-of-the-art models on images obtained

with a diversity of sensor noises. From this, we will be able

to identify which models generalise best, and thus performs

real-world super resolution to a satisfactory level. Due to the

lack of ‘ground-truth’ high-resolution images, we will not

be able to use quantitative metrics such as PSNR to evaluate

the performance of the models. Instead, we decided to

perform an extensive user study to obtain a human rating on

the super-resolved images, which was also done for (Rad

et al.,2021). The details of the survey will be discussed in

the proceeding sections.

3. Methodology

3.1. Real-World Super Resolution Methods

RealSR

The authors of the RealSR paper (Cai et al.,2019a) propose

a novel degradation framework based on kernel estimation

and noise injection. The method they propose is mainly

divided into two stages; the ﬁrst stage is to estimate the

degradation from real data and generate realistic LR images,

the second is to train the SR model based on the constructed

data. They take two sets of images as input: a real-world

image set

, and an HR image set

. They estimate the ker-

nel (using KernelGAN (Bell-Kligler et al.,2019)) and noise

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

HowRealisReal:EvaluatingtheRobustnessofReal-WorldSuperResolutionAthiyaDeviyani*1EfeSinanHoplamaz*1AlanSavioPaul*1AbstractImagesuper-resolution(SR)isaeldincom-putervisionthatfocusesonreconstructinghigh-resolutionimagesfromtherespectivelow-resolutionimage.However,super-resolutionisawell-knownill-pose...

展开>> 收起<<

How Real is Real Evaluating the Robustness of Real-World Super Resolution Athiya Deviyani 1Efe Sinan Hoplamaz 1Alan Savio Paul 1 Abstract.pdf

共9页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

How Real is Real Evaluating the Robustness of Real-World Super Resolution Athiya Deviyani 1Efe Sinan Hoplamaz 1Alan Savio Paul 1 Abstract

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: