Super-Resolution by Predicting Offsets An Ultra-Efficient Super-Resolution Network for Rasterized Images

2025-05-02 0 0 9.1MB 16 页 10玖币

侵权投诉

Super-Resolution by Predicting Oﬀsets:

An Ultra-Eﬃcient Super-Resolution Network for

Rasterized Images

Jinjin Gu1,⋆, Haoming Cai2, Chenyu Dong3, Ruofan Zhang3, Yulun Zhang4,

Wenming Yang3, Chun Yuan3,5,⋆

1The University of Sydney, 2University of Maryland, College Park,

3Tsinghua Shenzhen International Graduate School, Tsinghua University,

4ETH Z¨urich, 5Peng Cheng National Laboratory,

jinjin.gu@sydney.edu.au, hmcai@umd.edu,

{dcy20, zrf20}@mails.tsinghua.edu.cn, yulun100@gmail.com,

{yang.wenming, yuanc}@sz.tsinghua.edu.cn

Abstract. Rendering high-resolution (HR) graphics brings substantial

computational costs. Eﬃcient graphics super-resolution (SR) methods

may achieve HR rendering with small computing resources and have

attracted extensive research interests in industry and research communi-

ties. We present a new method for real-time SR for computer graphics,

namely Super-Resolution by Predicting Oﬀsets (SRPO). Our algorithm

divides the image into two parts for processing, i.e., sharp edges and ﬂat-

ter areas. For edges, diﬀerent from the previous SR methods that take

the anti-aliased images as inputs, our proposed SRPO takes advantage

of the characteristics of rasterized images to conduct SR on the raster-

ized images. To complement the residual between HR and low-resolution

(LR) rasterized images, we train an ultra-eﬃcient network to predict the

oﬀset maps to move the appropriate surrounding pixels to the new posi-

tions. For ﬂat areas, we found simple interpolation methods can already

generate reasonable output. We ﬁnally use a guided fusion operation to

integrate the sharp edges generated by the network and ﬂat areas by

the interpolation method to get the ﬁnal SR image. The proposed net-

work only contains 8,434 parameters and can be accelerated by network

quantization. Extensive experiments show that the proposed SRPO can

achieve superior visual eﬀects at a smaller computational cost than the

existing state-of-the-art methods.

1 Introduction

With the popularity of 4K or even 8K display devices, rendering graphics at

ultra-high resolution and high frame rates has attracted extensive research in-

terests in industry and research communities. However, achieving such a goal is

very challenging, as rendering graphics at high resolutions will bring huge compu-

tational costs, which poses severe challenges to graphics algorithms and graphic

⋆Corresponding author.

arXiv:2210.04198v1 [eess.IV] 9 Oct 2022

2 Gu et al.

computing devices. Especially for mobile devices, the huge computational cost

also means high power consumption. Finding ways to enjoy high resolutions and

high frame rates without compromising graphics quality as much as possible has

become an imperative issue.

In addition to compromising graphics eﬀects and texture quality, there is a

new paradigm in recent years to render low-resolution (LR) images ﬁrst and

then super-resolve them to obtain high-resolution (HR) images. NVIDIA Deep

Learning Super Sampling (DLSS) technology and AMD FidelityFX Super Res-

olution (FSR) are two representative solutions that follow this new paradigm.

However, the poor visual eﬀect and computationally unfriendly features of these

super-resolution (SR) methods restrict the application of such techniques. For

example, FSR uses traditional ﬁltering methods to upsample and sharpen LR

images. This method is relatively easy to deploy on various graphics comput-

ing devices, but its visual eﬀect is far from ideal. DLSS uses state-of-the-art

deep learning-based SR methods to obtain better perceptual eﬀects, but deep

networks’ high computational cost makes such a solution only available on ded-

icated acceleration devices called NVIDIA Tensor Core. There is no solution

available for other devices that do not have such powerful neural computing

power.

Deep learning-based SR methods (SR networks) directly map LR rendered

images to HR rendered images with many stacked convolutional layers/ﬁlters.

One of the reasons for its success is to use a large number of learnable parameters

in exchange for suﬃcient model capacity. Some successful SR networks even

contain tens of millions of parameters, e.g., EDSR [20], and RRDBNet [28].

The state-of-the-art eﬃciency-oriented SR network also contains more than 200k

parameters, far short of real-time SR requirements. An eﬃcient deep model

can be very “shallow” for widely deployed graphic devices, i.e., only three or

fewer convolutional layers. However, within this constraint, deep learning has no

precedent for success.

In this paper, we propose an ultra-eﬃcient SR method for computer graph-

ics, especially for mobile platforms and their graphics applications. Some ob-

servations inspire our approach. Firstly, we argue that generating rich, detailed

textures for small SR networks is diﬃcult. In this case, keeping the edges of the

SR image sharp is the most eﬀective way to improve the perceptual eﬀect. Our

method is mainly used to generate such sharp edges. Secondly, SR is generally

performed on anti-aliased images (i.e., FSR) because anti-aliased images typi-

cally have gradient distributions and visual eﬀects similar to natural images. It

is very straightforward to borrow an algorithm that works on natural images

for anti-aliased images. However, the edges of the anti-aliased images have more

complex gradient distributions, and it is diﬃcult for small SR networks to handle

such complex gradient changes. Rasterized images have simpler color changes,

obvious truncation of edges, and similar distributions under various rendering

engines than anti-aliased images. These properties allow us to perform SR on

rasterized images using simple operations without wasting the model’s capacity

to generate sharp edges.

Super-Resolution by Predicting Oﬀsets 3

1-Layer CNN 3-Layer CNN 7-Layer CNN 30-Layer Residual CNN Ground Truth

Diﬀerences

0.00

0.05

0.10

0.15

0.20

Fig. 1. For SR networks, the processing of edges is very important. It can be seen

that the 3-layer network mainly improves the edge compared to the network with only

one layer. Continuing to increase the network to seven layers did not bring signiﬁcant

changes. Until we deepen the network to 30 layers, the network cannot generate sharper

edges. Nevertheless, for the real-time SR problem explored in this paper, how to use

a 3-layer network to generate as sharp, visually pleasing edges as possible is the key

issue.

In the light of the above discussion, we propose to perform SR directly on

rasterized images. The diﬀerence between HR and LR rasterized images usually

occurs in the change of edge pixels. This change is spatially abrupt (the “jagged”

edges), and it is diﬃcult to make up for this diﬀerence with a simple convolutional

network. Our method predicts oﬀsets that shift surrounding similarly colored

pixels to their desired positions to compensate for this sharp diﬀerence. The

new SR rasterized image obtained by this method retains the characteristics of

the rasterized image well and can produce sharp edges. We fuse these edges with

other areas obtained by the interpolation method to get the ﬁnal output image.

With subsequent anti-aliasing (AA) algorithm, we can obtain super-resolved

rendered graphics at a small cost. Our ﬁnal method uses only a three-layer

convolutional neural network, which can also be quantized and accelerated and

run on various devices with 8-bit integer without loss of ﬁnal eﬀects. We present

extensive experiments, and the ﬁnal results show sharp edges and good visual

performance.

2 Related Work

Anti-Aliasing is a longstanding problem in computer graphics. Aliasing oc-

curs when sampling is insuﬃcient during rendering because each pixel can only

belong to a speciﬁc object and receive a unique pixel value, so jagged and saw-

toothed eﬀects can appear at the edges of such an object. Using higher sam-

pling rates is the most straightforward solution to ameliorate aliasing eﬀects,

i.e., super-sample anti-aliasing (SSAA) and multisample anti-aliasing (MSAA).

These approaches are computationally expensive and diﬃcult to be compatible

with modern rendering methods such as deferred lighting/shading, and thus not

4 Gu et al.

1X Rasterized image 2X Rasterized image 3X Rasterized image

Fig. 2. Samples of the generated training

images. We rendered 300 images of ran-

dom geometric objects with random gra-

dient colors, including triangles, circles,

rings, lines and bezier curves.

1X Rasterized image 2X Rasterized image

1X Anti-aliased image 2X Anti-aliased image Differences

Differences

Fig. 3. The diﬀerences between the ras-

terized images and the anti-aliased im-

ages. The residual images are between

the nearest neighbor upsampled LR im-

age and the HR image.

suitable for real-time applications for mobile devices. Another alternative anti-

aliasing paradigm is optimising the visual eﬀects of rasterized images through

image post-processing methods. The most used alternative to MSAA was edge

detection and blurring, which is simple, but the results are signiﬁcantly lower

quality than MSAA. In 2009, Reshetov proposed morphological anti-aliasing

(MLAA) [24] that performs anti-aliasing with rule-based pixel blending accord-

ing to certain patterns in the neighborhood. MLAA has gained extensive atten-

tion and applications and inspired many post-processing AA algorithms, e.g.,

FXAA, SMAA [12], Filmic SMAA and DLAA. Most of these algorithms have

already been applied and created good visual eﬀects. However, AA algorithms

do not increase the resolution of the rendered image, and they can only reduce

jagged artifacts at the native resolution.

Super-Resolution (SR) aims at creating HR images according to the LR

observations. The simplest way to perform SR is through interpolation-based

methods that generate HR pixels by averaging neighboring LR pixels, e.g., bicu-

bic, bilinear and Lanczos. These methods are computationally eﬃcient, but the

results are over-smoothed as the interpolated pixels are locally similar to neigh-

bouring pixels and thus have insuﬃcient visual eﬀects. AMD FSR algorithm ﬁrst

employs a Lanzocs-like upsampling method and uses a sharpening ﬁlter to opti-

mize overly smooth edges. A new fundamental paradigm shift in image process-

ing has resulted from the development of data-driven or learning-based methods

in recent years. Since Dong et. al. [5] introduce the ﬁrst SR network, plenty of

deep learning based SR methods have been proposed, including deeper networks

[6,14,25], recurrent architectures [15,26], residual architectures [20,18,28,19], and

attention networks [32,4,3]. Network-based methods are also used in computer

graphics [13,27]. More related to this work, Xiao et al. [30] propose a dense video

SR network for graphics rendering. These SR networks can achieve impressive

SR results, but the massive parameters and the expensive computational cost

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

Super-ResolutionbyPredictingOffsets:AnUltra-EfficientSuper-ResolutionNetworkforRasterizedImagesJinjinGu1,⋆,HaomingCai2,ChenyuDong3,RuofanZhang3,YulunZhang4,WenmingYang3,ChunYuan3,5,⋆1TheUniversityofSydney,2UniversityofMaryland,CollegePark,3TsinghuaShenzhenInternationalGraduateSchool,TsinghuaUniversi...

展开>> 收起<<

Super-Resolution by Predicting Offsets An Ultra-Efficient Super-Resolution Network for Rasterized Images.pdf

共16页,预览4页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Super-Resolution by Predicting Offsets An Ultra-Efficient Super-Resolution Network for Rasterized Images

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: