Boosting Adversarial Robustness From The Perspective of Eective Margin Regularization Ziquan Liu1and Antoni B. Chan1

2025-04-30 0 0 1.58MB 20 页 10玖币

侵权投诉

Boosting Adversarial Robustness From The

Perspective of Eﬀective Margin Regularization

Ziquan Liu1and Antoni B. Chan1

Department of Computer Science, City University of Hong Kong

ziquanliu2-c@my.cityu.edu.hk, abchan@cityu.edu.hk

Abstract. The adversarial vulnerability of deep neural networks (DNNs)

has been actively investigated in the past several years. This paper in-

vestigates the scale-variant property of cross-entropy loss, which is the

most commonly used loss function in classiﬁcation tasks, and its im-

pact on the eﬀective margin and adversarial robustness of deep neural

networks. Since the loss function is not invariant to logit scaling, increas-

ing the eﬀective weight norm will make the loss approach zero and its

gradient vanish while the eﬀective margin is not adequately maximized.

On typical DNNs, we demonstrate that, if not properly regularized, the

standard training does not learn large eﬀective margins and leads to ad-

versarial vulnerability. To maximize the eﬀective margins and learn a

robust DNN, we propose to regularize the eﬀective weight norm dur-

ing training. Our empirical study on feedforward DNNs demonstrates

that the proposed eﬀective margin regularization (EMR) learns large

eﬀective margins and boosts the adversarial robustness in both stan-

dard and adversarial training. On large-scale models, we show that EMR

outperforms basic adversarial training, TRADES and two regularization

baselines with substantial improvement. Moreover, when combined with

several strong adversarial defense methods (MART [48] and MAIL [26]),

our EMR further boosts the robustness.

1 Introduction

One major challenge to the security of computer vision systems is that deep

neural networks (DNNs) often fail to achieve a satisfactory performance under

adversarial attacks [45]. Since the phenomenon is observed, various adversarial

attacks [15,5,10] and defense methods [23,33,53] have been proposed and the

understanding into the adversarial vulnerability of DNNs is improved [3,46,20].

Denote the DNN as fθ:x7→ l, with x∈RDand l∈RK. The model is optimized

by algorithm Athat minimizes empirical risk Lover training set Dtr ,

θ∗=A(fθ,Dtr,L).(1)

There are generally four direct methods to improve the robustness of DNNs.

First, the function space fθcan be designed to accommodate the need for ad-

versarial robustness. For example, replacing the piecewise linear activation with

a smooth activation function improves the performance of adversarial training

arXiv:2210.05118v1 [cs.LG] 11 Oct 2022

2 Liu and Chan

Effective Margin

Epoch Epoch

Effective Margin

!"# $ %

!& '

!"%

!& '

!"%

!& ( )'

𝑚"!

𝑚""

Epoch

(a) Illustration (b) Comparison

Fig. 1. The problem of cross-entropy loss in maximizing eﬀective margins and the

proposed ERM’s performance in terms of increasing the eﬀective margins on MNIST

test set. (a) The gradient of the sigmoid function σ(·) vanishes if we scale up the

input logit fθ(x) by α > 1. However, its distance to the decision boundary (dashed

red vertical line), i.e., the eﬀective margin ˜mdeﬁned in (3) remains the same. This

property shows that only training with cross-entropy loss does not eﬀectively increase

the actual margin. Our work aims to push the decision boundary away from the sample

so that the ˜mis increased, e.g., fθ(x) + βwith β > 0. (b) The means (solid lines) and

standard deviations (background shadow) of the eﬀective margins of an MLP on the

MNIST test set. The proposed EMR achieves an ˜mand adversarial robustness that is

comparable with adversarial training, and outperforms standard training with weight

decay or L-Softmax. See Table 1 for details. (c) and (d) Eﬀective margin on CIFAR10

and MNIST when diﬀerent λW D are used. Training without weight decay (WD) or

with small λW D leads to smaller eﬀective margins.

[49], and some architectural conﬁgurations are better than others in terms of

adversarial robustness [19]. Second, the algorithm Acan be incorporated with

inductive biases to learn a function with some speciﬁc properties, such as low

model complexity [22], local linearization [38] and feature alignment [25]. Third,

the training set Dtr can be shifted by adversarial perturbation [33,53] or other

data augmentation [16,39] to enhance the robustness. Finally, a carefully de-

signed loss Lcan be used to improve the robustness, such as Max Mahalanobis

center loss [36]. Besides the direct adversarial defenses, indirect adversarial de-

fenses are also investigated, e.g., adversarial examples detection [35,30,51,42] and

obfuscated gradient defenses [2,32,4,12,17,50,44,43].

This work falls into the second category (inductive bias) where the neural

network is trained with regularization to boost the adversarial robustness. We

consider the most popular loss function for the classiﬁcation task, the cross-

entropy (XE) loss,

XEi=−

k=1

yik log exp(lik)

Pjexp(lij ),(2)

where the logit lik =f(k)

θ(xi) is the k-th output of the neural network for the i-th

sample. One property of the network is that the prediction for the i-th sample,

i.e., ˆyi= arg maxk∈[K]lik, is invariant to scaling the logit vector li= [lik]kby a

positive constant α. In other words, the classiﬁcation accuracy will not change

if we scale liup to αliwhere α > 1. However, the XE loss will vanish if we scale

up the logit.

Boosting Adversarial Robustness From The Perspective of EMR 3

This phenomenon brings a problem in optimization with XE loss, since the

training only aims to minimize the loss without maximizing the eﬀective mar-

gin, which is deﬁned as the normalized logit diﬀerence (see Equation 3) and is

invariant to the weight magnitudes. Once a sample is correctly classiﬁed, the

scale-variant property of XE loss can be exploited by SGD to minimize the loss

while the distance to the decision boundary (in the input space) remains small

(see Fig. 1a). In a homogeneous NN [31,14,29], such as multi-layer perceptron

(MLP) and convolutional neural network (CNN) without residual connections or

normalization layers, the logit magnitude scales with weight norms so the train-

ing algorithm can minimize the loss by increasing weight norms. In a ResNet [18],

the ﬁnal classiﬁcation layer can be scaled up to minimize the cross-entropy loss.

Weight decay [22] is a common strategy to increase the eﬀective margin by con-

trolling the squared l2norm of weights in the DNN. However, it is known that

adversarial robustness cannot be achieved by only using weight decay (WD),

especially in deeper networks [45]. The most popular way to robustify DNNs

is adversarial training (AT) [33], which explicitly perturbs samples to be on the

desired margin from the original samples, and then trains on the perturbed sam-

ples. However, AT incurs increased computational cost for training due to the

generation of the adversarial training samples in each iteration.

In this paper, we propose eﬀective margin regularization (EMR) to push the

decision boundary away from the samples by controlling the eﬀective weight

norms of the samples. We ﬁrst show that traditional regularization such as

weight decay and large-margin loss (e.g., [28]) cannot train a DNN with sat-

isfactory robustness. Then the proposed method is compared with WD, large-

margin softmax and adversarial training, where we show its strength at maxi-

mizing the eﬀective margin and thus improving adversarial robustness. Finally,

on large-scale DNNs, we propose an approximation to EMR and demonstrate

that when combined with adversarial training, EMR achieves competitive results

compared with basic adversarial training and two recent regularization methods

for improving adversarial training, i.e., Input Gradient Regularization (IGR) [41]

and Hypersphere Embedding (HE) [37]. Note that our EMR is complementary

to adversarial training (AT) – EMR pushes the decision boundary away from

the training samples so as to increase the eﬀective margin, while AT generates

training samples on the desired margin. Thus EMR and AT can be combined to

further improve adversarial robustness.

2 Related Work

Adversarial Defense. The standard way to train an adversarially robust DNN

is to use adversarial training [33]. The clean examples are deliberately perturbed

to approach the desired margin distance, so that the eﬀective margin is produced

during training. Based on adversarial training, regularization approaches are

proposed to learn a DNN with desired properties. [41] proposes to regularize

the norm of the loss gradient with respect to input (IGR). In contrast, our work

proposes to regularize the gradient of logit with respect to input to maximize the

4 Liu and Chan

eﬀective margin. Locally Linear Regularization (LLR) [38] is proposed to learn

a more linear loss function at each training sample, while our paper controls the

local logit function’s weight norm for training samples. Hypersphere embedding

(HE) [37] proposes to normalize the features and classiﬁcation layer to alleviate

the inﬂuence of weight norms. In our empirical study, we demonstrate that EMR

achieves better robustness than IGR and HE on large-scale neural networks.

Margin Regularization. The hinge loss [7] is a classical loss to induce a large

margin in SVM [7]. On DNNs, several losses are proposed to induce large mar-

gins, such as Large-Margin Softmax [28], A-Softmax [27] and AM-Softmax [47].

These large-margin losses still have problems to learn large eﬀective margins

since the scale of features and weights aﬀects the loss values. On both MLP

and CNN, we demonstrate that training with L-Softmax loss improves the eﬀec-

tive margin compared with the standard cross-entropy loss, while EMR learns a

larger eﬀective margin than L-Softmax since EMR considers the scale problem

in the loss function. [31,6] study the normalized margin of homogeneous DNNs

trained with gradient descent from a theoretical perspective and prove that the

normalized margin is maximized by the gradient descent. Our work empirically

investigates the normalized margin in DNNs trained with stochastic gradient de-

scent and its inﬂuence on the adversarial robustness. We show that by controlling

the eﬀective weight norm and increasing the eﬀective margin, the adversarial ro-

bustness can be improved over vanilla training with SGD and WD. The attack

method DeepFool [34] moves an input sample to cross its decision boundary by

treating the model as a linear classiﬁer at each optimization step, which is related

to the margin of a classiﬁer. In contrast, our paper proposes to defend against

adversarial attacks by increase the eﬀective margin during training. We did not

evaluate the DeepFool since it is not a standard attack method in adversarial

defense literature [8] and our experiment shows that DeepFool is not as eﬀec-

tive as PGD at attacking large-scale models. Max-Margin Adversarial training

(MMA) [13] proposes to approximate the margin by pushing input samples to

cross the classiﬁcation boundary with PGD and recording the moved distance.

In contrast, EMR proposed to maximize the eﬀective margin by regularizing

the eﬀective weight matrix norm, and can boost the adversarial robustness of

both standard training and adversarial training. Since the performance of MMA

is worse than vanilla PGD and TRADES according to AutoAttack benchmark

[10], we do not include the comparison with MMA in the experiment.

3 Regularizing Eﬀective Weight Norm Improves Eﬀective

Margin and Adversarial Robustness

We use the notation for a DNN in Section 1. A general DNN, such as MLP,

CNN and evaluation-mode Resnet with piece-wise linear activation functions

(e.g., ReLU and LeakyReLU), can be expressed as a linear function for each

input sample, i.e., fθ(xi) = W(xi)xi+b(xi). The weight matrix Wi,W(xi)

and bias bi,b(xi) are determined by input samples since the activations will

change with the input. Similar to the normalized margin in [31], we deﬁne the

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

BoostingAdversarialRobustnessFromThePerspectiveofEectiveMarginRegularizationZiquanLiu1andAntoniB.Chan1DepartmentofComputerScience,CityUniversityofHongKongziquanliu2-c@my.cityu.edu.hk,abchan@cityu.edu.hkAbstract.Theadversarialvulnerabilityofdeepneuralnetworks(DNNs)hasbeenactivelyinvestigatedinthepas...

展开>> 收起<<

Boosting Adversarial Robustness From The Perspective of Eective Margin Regularization Ziquan Liu1and Antoni B. Chan1.pdf

共20页,预览4页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Boosting Adversarial Robustness From The Perspective of Eective Margin Regularization Ziquan Liu1and Antoni B. Chan1

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: