Landmark Enforcement and Style Manipulation for Generative Morphing Samuel Price Sobhan Soleymani Nasser M. Nasrabadi West Virginia University

2025-05-03 0 0 6.88MB 10 页 10玖币

侵权投诉

Landmark Enforcement and Style Manipulation for Generative Morphing

Samuel Price*

, Sobhan Soleymani*

, Nasser M. Nasrabadi

West Virginia University

{swp0001, ssoleyma}@mix.wvu.edu, nasser.nasrabadi@mail.wvu.edu

Abstract

Morph images threaten Facial Recognition Systems

(FRS) by presenting as multiple individuals, allowing an

adversary to swap identities with another subject. Morph

generation using generative adversarial networks (GANs)

results in high-quality morphs unaffected by the spatial ar-

tifacts caused by landmark-based methods, but there is an

apparent loss in identity with standard GAN-based morph-

ing methods. In this paper, we propose a novel StyleGAN

morph generation technique by introducing a landmark en-

forcement method to resolve this issue. Considering this

method, we aim to enforce the landmarks of the morph im-

age to represent the spatial average of the landmarks of the

bona ﬁde faces and subsequently the morph images to in-

herit the geometric identity of both bona ﬁde faces. Explo-

ration of the latent space of our model is conducted using

Principal Component Analysis (PCA) to accentuate the ef-

fect of both the bona ﬁde faces on the morphed latent rep-

resentation and address the identity loss issue with latent

domain averaging. Additionally, to improve high frequency

reconstruction in the morphs, we study the train-ability of

the noise input for the StyleGAN2 model.

1. Introduction

Generative Adversarial Networks (GANs) continue to

grow in popularity in areas such as deepfake generation: re-

alistic images generated by a deep neural network (DNN)

[14, 19, 34]. With recent developments in the realistic face

generation abilities of GANs [15, 16], the threat synthesized

images pose to personal reputation, corporate sabotage, and

national security grow concerning [19]. As such, attacks

on Facial Recognition Systems (FRS) mount as their usage

continues to grow as an integral part of national security

and law enforcement to verify identity [8]. Border secu-

rity is a key target as facial recognition is the only biomet-

ric required in electronic Machine-Readable Travel Docu-

ments (eMRTD) approved by the International Civil Avi-

*Authors Contributed Equally.

Figure 1: Subjects are warped toward the average of their

landmarks to produce a warped convex hull of each sub-

ject. The convex hulls are inverted into latent space of

StyleGAN2 using a weighted combination of perceptual

and pixel-wise losses in addition to latent and noise regular-

ization exploring three techniques for blending latent codes.

ation Commission [1]. Facial morph images have proven

a threat to FRS when submitted by a bad actor to attack

the enrollment stage of the biometric system integration

guideline set by the ICAO, passing two safeguards: image

tampering detection and identity veriﬁcation [13]. A facial

morph is an artiﬁcial face image generated by blending two

or more bona ﬁde face images of different individuals. The

contributing subjects can use the morph for veriﬁcation as

FRS would ﬁnd their identities indistinguishable to that of

the morph. If a morph fools both the morph detector and is

identiﬁed as the individual in question, a bad actor can cir-

cumvent these security measures. Using a GAN, our pro-

posed technique generates morphs possessing the identity

of two individuals to fool both human inspectors and FRS.

GAN-based morph generation blends the bona ﬁde im-

ages in the latent space of the model by averaging the latent

arXiv:2210.10182v1 [cs.CV] 18 Oct 2022

representations of contributing subjects [10, 33]. Improve-

ments to early face generating GANs have increased their

threat to FRS [15, 16]. Although beneﬁting from enhanced

visual quality, compared to other face morphing techniques,

GAN-based face morphing falls short when used to attack

FRS compared to landmark-based morphing due to a loss

of identity in the morphed images [2, 33, 37]. As presented

in Figure 1, we address this issue as we augment the la-

tent space projections of the bona ﬁde images by blending

their landmarks before calculating their latent representa-

tions. Our landmark enforcement technique improves the

morphed face’s landmarks, being equidistant from the bona

ﬁde subjects’ landmarks. To construct the latent represen-

tations for the bona ﬁde subjects, we build upon inversion

methods from [6, 16, 3] by incorporating a landmark en-

forcement algorithm to preserve the blended landmarks in

the latent representation. In addition, we adapt the noise

input of our model [16] to derive an improved image inver-

sion algorithm resulting in latent codes with higher levels of

reconstruction quality.

We integrate our proposed inversion algorithm in the

StyleGAN2 to improve the morph generation. We ex-

plore the constructed latent space using Principal Compo-

nent Analysis (PCA) to enhance the blending of latent rep-

resentations and further improve the quality of the morph

images without adding additional optimization steps. This

exploration aims at addressing the known issue with latent

representation averaging which leads to morphs possessing

biased or neither bona ﬁde identities [37]. We examine the

covariance of latent representations using PCA and replace

the latent code averaging with element-wise and vector-

wise blending of PCA projected latent codes. By applying

our image inversion algorithm and exploring latent repre-

sentation blending in the PCA domain, we generate GAN-

based morph images to fool FRS at increased rates while

maintaining high image quality to fool a human inspector.

Our major contributions in this paper are:

• We present a novel StyleGAN2 morphing technique

by enforcing landmarks to improve geometric identity

preservation in the morph.

• We study latent space exploration in the PCA domain

to improve latent code blending by addressing identity-

imbalance issue.

• We study the inﬂuence of the noise input of our model

to improve latent representations and morph image

quality.

2. Related Work

2.1. Landmark Morphing

Facial morphing techniques split into two categories:

landmark-based and GAN-based. GAN-based morphing

operates in the latent space, whereas landmark-based mor-

phing is performed in the image domain [35, 20, 4, 5, 2].

Landmark-based morphing uses landmark predictions of

contributing subjects to warp them toward an equidistant

set of landmarks. The pixel values of the warped images

are alpha blended to complete the morph. Landmark-based

morphing has been the most effective automated morphing

threat to FRS [33]; however, the blending of pixel values

and imperfections in the landmark alignments create arti-

facts surrounding the morphed image eyes, mouth, nose,

and edges around the face due to pasting. This ghosting

effect increases the possibility of a human investigator rec-

ognizing the morph.

2.2. GAN-Based Morphing

Damer et al. [10] introduced GAN-based morphing us-

ing MorGAN to invert images into latent representations via

an encoder, averaging the faces in the latent domain, and

inputting the resultant morph latent representation into the

generator. In a study by Venkatesh et al. [33], MorGAN

morphs were shown to be limited in both image generation

quality and output size of 64×64×3. Morphs generated us-

ing MorGAN fail to pass the size standards set by the ICAO

[1] while also failing to attack the veriﬁcation of FRS.

Prior techniques for GAN-based morphing projects the

average of two bona ﬁde latent representations into the gen-

erator to synthesize the morphed image [6, 33]. The per-

formance of morphs using StyleGAN [15] signiﬁcantly im-

proved when compared to ones generated using MorGAN

[10], but the performance is not comparable to landmark-

based methods. Improvements to morph generation using

StyleGAN include training encoders to estimate the latent

embeddings [24, 31] or by adding new loss functions for

optimization [37]. MIPGAN [37] proposed a hybrid ap-

proach to StyleGAN morphing by using both an encoder to

estimate the latent codes of the bona ﬁde subjects and an op-

timization cycle to improve the averaged latent code. The

novel addition to their optimization cycle was an identity

loss function using a pre-trained FRS model [12] to balance

the identity of the morph between the bona ﬁde subjects.

2.3. PCA For Latent Exploration

Principle Component Analysis (PCA) is widely used to

evaluate correlation between samples in a dataset [32]. The

foundation of PCA calculates the covariance matrix of the

dataset whose eigenvectors represent the variance of the en-

tire dataset. Each eigenvector represents a variable amount

of the variance of the dataset, so by removing the eigenvec-

tors in order of smallest to greatest eigenvalue, the quality

of the restored data degrades exponentially. PCA has been

used to assist in the traversal and disentanglement of the la-

tent space of GANs [22, 36]. We explore applications of

PCA for blending two latent representations for morphing.

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

LandmarkEnforcementandStyleManipulationforGenerativeMorphingSamuelPrice*,SobhanSoleymani*,NasserM.NasrabadiWestVirginiaUniversityfswp0001,ssoleymag@mix.wvu.edu,nasser.nasrabadi@mail.wvu.eduAbstractMorphimagesthreatenFacialRecognitionSystems(FRS)bypresentingasmultipleindividuals,allowinganadversaryto...

展开>> 收起<<

Landmark Enforcement and Style Manipulation for Generative Morphing Samuel Price Sobhan Soleymani Nasser M. Nasrabadi West Virginia University.pdf

共10页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Landmark Enforcement and Style Manipulation for Generative Morphing Samuel Price Sobhan Soleymani Nasser M. Nasrabadi West Virginia University

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: