Face Super-Resolution with Progressive Embedding of Multi-scale Face Priors Chenggong Zhang Zhilei Liu College of Intelligence and Computing Tianjin University

2025-04-26 0 0 633.93KB 8 页 10玖币

侵权投诉

Face Super-Resolution with Progressive Embedding of Multi-scale Face Priors

Chenggong Zhang, Zhilei Liu*

College of Intelligence and Computing, Tianjin University

Tianjin, China

{zhangchenggong, zhileiliu}@tju.edu.cn

Abstract

The face super-resolution (FSR) task is to reconstruct

high-resolution face images from low-resolution inputs. Re-

cent works have achieved success on this task by utiliz-

ing facial priors such as facial landmarks. Most existing

methods pay more attention to global shape and structure

information, but less to local texture information, which

makes them cannot recover local details well. In this pa-

per, we propose a novel recurrent convolutional network

based framework for face super-resolution, which progres-

sively introduces both global shape and local texture infor-

mation. We take full advantage of the intermediate outputs

of the recurrent network, and landmarks information and

facial action units (AUs) information are extracted in the

output of the ﬁrst and second steps respectively, rather than

low-resolution input. Moreover, we introduced AU clas-

siﬁcation results as a novel quantitative metric for facial

details restoration. Extensive experiments show that our

proposed method signiﬁcantly outperforms state-of-the-art

FSR methods in terms of image quality and facial details

restoration.

1. Introduction

Recent years have witnessed the rapid progress of tech-

nologies for biometrics, especially facial analysis tech-

niques including face recognition and intent recognition

from facial expressions [34]. Nevertheless, most existing

techniques would degrade substantially when given very

low-resolution face images. Face super-resolution (FSR),

also known as face hallucination, aims to estimate the

high-resolution (HR) face images from its degraded low-

resolution (LR) observation and restore details. Therefore,

FSR can be used as an important means of image prepro-

cessing and greatly beneﬁt face-related tasks which desire

high-frequency face details.

As a domain-speciﬁc task of general single image super-

*Corresponding author

resolution (SISR), FSR is an inherently ill-posed problem

since there are always many possible HR counterparts for

every LR image. To reduce the solution space of the prob-

lem and promote the performance of FSR, many researchers

introduce speciﬁc prior information in face images into the

FSR problem. A dense correspondence ﬁeld is employed

in [47] to help restore accurate facial details. Facial land-

marks and parsing maps are introduced in [5] to improve

recovery performance. A deep iterative collaboration net-

work [26] optimizes face recovery and landmark estimation

alternatively. Facial attributes, such as age, gender, and oth-

ers, are also usually exploited in some face hallucination

methods [25, 42, 41]. However, most of the existing meth-

ods do not take full advantage of different scale prior infor-

mation and explore only single scale prior information, i.e.

either global level or local level.

Different faces have distinct distributions in their shape

and texture. For face super-resolution, the texture and shape

information are both crucial. Many previous face SR meth-

ods pay more attention to global shape and structure infor-

mation, but less to local texture information. Because much

prior information such as landmarks and face parsing maps

represents the global facial shape. Facial action units (AUs)

refer to a set of basic facial muscle actions at certain fa-

cial locations deﬁned by the Facial Action Coding System

(FACS) [35]. The activation or intensity of AUs represents

a kind of local semantic and texture information of the face.

Some works [33, 23], which used AU information for face

editing, proved that AU information can affect the local tex-

ture of the face.

To alleviate these above problems, in this paper, we

propose a novel recurrent convolutional network based

face super-resolution method, which introduces multi-scale

facial prior information at different steps progressively.

Firstly, we use a branch to estimate landmarks for the output

of the ﬁrst step and input them to the next step to promote

facial shape restoration. Then, another branch is used to de-

tect AU information from the output of the second step. In

particular, we generate AU attention maps based on the es-

timated landmarks to better boost the local texture recovery

arXiv:2210.06002v1 [cs.CV] 12 Oct 2022

in the following steps. The main contributions of this paper

can be summarized as follows:

• We present a novel FSR framework based on the recur-

rent network that takes full advantage of multi-scale

facial prior information (i.e. landmarks and AUs) to

generate realistic HR face images.

• Global shape information and local texture informa-

tion are embedded into a recurrent network progres-

sively. And we introduced AU classiﬁcation results as

a novel quantitative metric for facial details restoration

of FSR.

• Extensive qualitative and quantitative experiments

demonstrate that, compared with similar state-of-the-

art methods, our proposed framework achieves supe-

rior results in terms of both image quality and facial

details restoration.

2. Related Work

2.1. Single Image Super-Resolution

FSR is a special case of single image super-resolution

(SISR). Recently, due to the excellent learning ability, deep

convolutional neural networks have demonstrated high su-

periority on SISR tasks. Dong et al. [7] ﬁrstly presented

the SRCNN for SISR and achieved promising performance

against traditional methods. Inspired by this pioneering

work, many deep network based SISR methods have been

proposed. Kim et al. [15] designed the VDSR network with

more convolutional layers based on residual learning [11].

Ledig et al. [19] proposed SRGAN for generating photo-

realistic images based on generative adversarial network

(GAN) [9]. Shi and Liu [37] proposed DPA-Net for in-

fant ﬁngerprint super-resolution and enhancement. Some

attention-based methods [46, 6, 22] are also proposed to

further improve the SR performance. However, most of the

above methods have a deep network and hold a lot of pa-

rameters, which may suffer from overﬁtting. To gain better

generalization capability without introducing overwhelm-

ing parameters, the recurrent structure has also been em-

ployed for SISR. Kim et al. [16] ﬁrstly introduced recursive

learning in DRCN for parameter sharing. Later, Tai et al.

designed a recursive block with enhanced residual units in

DRRN [38] and memory blocks with the recursive unit and

gate unit in MemNet [39]. Han et al. [10] presented DSRN

considering a dual-state design to exploit features from both

LR and HR states for ﬁnal predictions. Li et al. [21] devel-

oped a novel feedback block consisting of up- and down-

sampling layers with dense skip connections in SRFBN.

These SISR methods are designed for general images and

most of them only handle up to 4× super-resolution. They

fail to restore the details well for face images, especially

when the scaling factor is large (e.g. 8 ×).

2.2. Face Super-Resolution

Since the concept of face hallucination was ﬁrst pro-

posed by Baker and Kanade [1], many methods were pro-

posed to improve the FSR performance, especially with

the development of deep learning. Yu et al. [43] pro-

posed a GAN-based network URDGN to super-resolve very

low-resolution face images. Huang et al. [12] presented

a wavelet-based method to transform the FSR problem to

wavelet coefﬁcients prediction task. Cao et al. [4] proposed

Attention-FH using reinforcement learning to discover at-

tended patches and then enhance the facial part sequentially.

Dou et al. [8] introduced the incremental orthogonal pro-

jection discrimination in the principal component analysis

subspace to enhance the FSR task. Lu et al. [24] designed

a SISN to reconstruct photorealistic high-resolution facial

images by fusing the features from two paths.

Compared with general images, the face images have

unique prior information which could be utilized. Chen

et al. [5] introduced geometry priors including landmark

heatmaps and parsing maps. Kim et al. [14] used landmark

heatmaps to design an attention loss. Ma et al. [26] intro-

duced a recurrent network to face SR and designed a deep

iterative collaboration framework to optimize face recovery

and landmark estimation alternatively. In addition, facial at-

tributes, such as age and gender, are also usually employed

in some FSR methods [25, 42, 41]. However, most of the

existing methods explore only single-level prior informa-

tion and pay more attention to global shape and structure

information, but less to local texture information.

3. Proposed Method

In this section, we propose a multi-scale prior infor-

mation embedded recursive network (MPENet) for FSR,

which can be unfolded as shown in Figure 1 and consists

of three branches: face super-resolution branch, face align-

ment branch, and AU detection branch. In addition, to gen-

erate photo-realistic face images, we use MPENet as a gen-

erator network Gand introduce a discriminator network D

to build our generative adversarial model MPEGAN.

3.1. Network Architecture

Face Super-Resolution Branch: As shown in Figure 1,

our proposed MPENet can be unfolded into 3 iterations.

The face super-resolution branch in each iteration contains

three parts: a shallow feature extractor (SFE), a prior em-

bedded recurrent block (PERB), and a high resolution re-

construction block (RECB). Given a low-resolution (LR)

input ILR, we use a 3×3 convolutional layer and a pixel

shufﬂe layer to extract shallow feature Ft

sf at t-th iteration

as:

sf =HSF E (ILR),(1)

where HSF E (·)denotes the operations of the SFE. Then,

we use a 1×1 and a 3×3 convolutional layer to fuse shal-

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

FaceSuper-ResolutionwithProgressiveEmbeddingofMulti-scaleFacePriorsChenggongZhang,ZhileiLiu*CollegeofIntelligenceandComputing,TianjinUniversityTianjin,Chinafzhangchenggong,zhileiliug@tju.edu.cnAbstractThefacesuper-resolution(FSR)taskistoreconstructhigh-resolutionfaceimagesfromlow-resolutioninputs.Re...

展开>> 收起<<

Face Super-Resolution with Progressive Embedding of Multi-scale Face Priors Chenggong Zhang Zhilei Liu College of Intelligence and Computing Tianjin University.pdf

共8页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Face Super-Resolution with Progressive Embedding of Multi-scale Face Priors Chenggong Zhang Zhilei Liu College of Intelligence and Computing Tianjin University

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: