Stability Via Adversarial Training of Neural Network Stochastic Control of Mean-Field Type Julian Barreiro-Gomez Salah Eddine Choutri Boualem Djehiche

2025-05-03 0 0 951KB 6 页 10玖币

侵权投诉

Stability Via Adversarial Training of Neural Network Stochastic

Control of Mean-Field Type

Julian Barreiro-Gomez Salah Eddine Choutri Boualem Djehiche

Abstract— In this paper, we present an approach to neural

network mean-ﬁeld-type control and its stochastic stability

analysis by means of adversarial inputs (aka adversarial at-

tacks). This is a class of data-driven mean-ﬁeld-type control

where the distribution of the variables such as the system

states and control inputs are incorporated into the problem.

Besides, we present a methodology to validate the feasibility of

the approximations of the solutions via neural networks and

evaluate their stability. Moreover, we enhance the stability by

enlarging the training set with adversarial inputs to obtain a

more robust neural network. Finally, a worked-out example

based on the linear-quadratic mean-ﬁeld type control problem

(LQ-MTC) is presented to illustrate our methodology.

Index Terms— Neural networks, data-driven control, sta-

bility, robustness, supervised machine learning, adversarial

training

I. INTRODUCTION

Mean-ﬁeld type control is a topic that attracted a lot

of attention since the introduction of mean-ﬁeld games by

Lasry and Lions in their seminal work [1] and by Caines,

Huang and Malham´

e in [2]. Andersson and Djehiche in

[3] introduced a stochastic mean-ﬁeld type control problem

in which the state dynamics and the performance criterion

depend on the moments of the state, see also [4] and [5].

Carmona and Delarue [6], and Buckdahn et al. [7], later,

generalized the problem to include the probability law of the

state dynamics. For applications related to mean-ﬁeld type

control and games problems we cite, among many others,

[8, Chapter 16], [9]–[11] and the references therein.

This class of problems is non-conventional since both the

evolution of the state and often the performance functional

are inﬂuenced by terms that are not directly related to the

state or to the control of the decision maker. In a sense,

they model a very large number of agents behaving, all,

similarly to a representative agent. The latter is impacted by

the aggregation of all agents due to the large number. The

aggregation effect is modelled as a mean-ﬁeld term such as,

among others, the law of the state, the expectation of the

state or its variance.

Solving this problem, analytically is rather challenging

as there are no general analytic methods for this purpose.

Therefore the use of numerical methods is often needed to

Julian Barreiro-Gomez and Salah Eddine Choutri are with NYUAD

Research Institute, New York University Abu Dhabi, PO Box 129188,

Abu Dhabi, United Arab Emirates. (e-mails: jbarreiro@nyu.edu,

choutri@nyu.edu).

Boualem Djehiche works at the Department of Mathematics, KTH,

Stockholm, Sweden. (e-mails: boualem@kth.se).

We gratefully acknowledge support from Tamkeen under the NYU Abu

Dhabi Research Institute grant CG002. We thank Hatem Hajri for the

discussions on adversarial training.

provide approximations to the solutions and several methods

have been suggested for ﬁnite horizon mean-ﬁeld type con-

trol problems (see e.g. [12], [13]). Furthermore, the recent

progress on machine learning technologies made it easier

to test and provide more efﬁcient approximations of the

solutions to complex mean-ﬁeld type control problems.

The link between deep learning and mean-ﬁeld type

control and games was recently studied by, among others,

Lauri`

ere, Carmona and Fouque in series of papers (see e.g.

[14]–[17]), where the authors (jointly and/or independently)

proposed algorithms for the solution of mean-ﬁeld type

optimal control problems based on approximations of the

theoretical solutions by neural networks, using the software

package TensorFlow with its ‘Stochastic Gradient Descent’

optimizer designed for machine learning. However, the sta-

bility of neural networks associated to the mean-ﬁeld type

control problems was not considered in the literature so far.

In deep learning, there is an increasing interest in studying

and improving the robustness and stability of the trained

neural networks see e.g. [18], where it has been reported

that a simple modiﬁcation in the input data might fool a

well-trained neural network, returning a wrong output. For

instance, a picture that is previously well-classiﬁed by a

trained neural network could be incorrectly classiﬁed once

we perturb one or more pixels in it. Such perturbations are

known as adversarial attacks and can help to characterize

how robust and stable a network is. The contribution of

this paper is summarized in three points: training, stability

evaluation, and stability improvement.

Training: we present an indirect and simple method to

train neural networks to learn optimal controls based on data.

Inspired by the work in [15] and [16], we ﬁrst illustrate how

data can be generated by computationally solving a ﬁnite-

time horizon optimal control problem with decision variables

given by the output of the neural network. Then, we design a

data-driven (model-free) mean-ﬁeld-type control using neural

networks in a supervised learning fashion. The idea is to

design an ofﬂine controller (once trained, only a simple

forward run is required) that is more time-efﬁcient than

the conventional online optimization-based control approach,

which can be time consuming depending on the complexity

of the problem. In real life, one can use the data provided by

a trafﬁc application such as google maps to train the neural

network to give optimal paths.

Stability: we borrow the idea of an adversarial attack from

the topic of image classiﬁcation and draw an analogy in

the context of stochastic dynamical systems. An adversarial

attack, in our sense, is an initial condition that might make

arXiv:2210.00874v1 [math.OC] 27 Sep 2022

the closed-loop neural network control system unstable. This

concept enabled us to study the stability of the neural

network mean-ﬁeld-type control and empirically characterize

the corresponding forward invariant basin of attraction for the

closed-loop system composed of the stochastic dynamics and

the optimal control/strategies.

Stability improvement: we improve the stability of the

neural network mean-ﬁeld-type control by enlarging the

training set using adversarial data (attacks) generated from

the previous phase (Stability). We compare and discuss the

resulting data-enhanced closed-loop system and a suitably

modiﬁed neural network architecture which potentially en-

hances the stability of the closed-loop system.

The reminder of this paper is organized as follows. In

section II, we formulate the mean-ﬁeld type control problem.

In section III, we present an approach that solves ﬁrst an

optimization problem taking as decision variables the output

of the neural network, and then it solves the neural network

training. In section IV, we deﬁne the stochastic stability

concept for our mean-ﬁeld type neural network by means

of adversarial inputs or attacks to the closed-loop neural

network. We illustrate our stability results in Section V

through numerical examples. Section VI concludes the paper.

II. MEAN-FIELD-TYPE CONTROL PROBLEM

We consider a ﬁnite-horizon stochastic control problem

where the state process is governed by a stochastic differ-

ential equation (SDE) of mean-ﬁeld type. The drift here

depends on the state and control as well as their respective

probability laws. For a ﬁxed time horizon T > 0, let

(Ω,F,(Ft)0≤t≤T,P)be a ﬁltered probability space satis-

fying the usual conditions, on which we deﬁne a standard

Brownian motion B:= (B(t),0≤t≤T). We assume

that F:= (Ft,0≤t≤T)is the natural ﬁltration of B

augmented by P-null sets of F. The action space, U, is a

non-empty, closed and convex subset of R, and Uis the class

of measurable, F-adapted and square integrable processes

taking values in U. For any control u∈ U, we consider the

following SDE







dx(t) = f(t, x(t),E[x(t)], u(t),E[u(t)])dt

+σdB(t),

x(0) = x0, x0∼µ0,

(1)

where, f: [0, T ]×R×R×U×R→R,σ > 0.The expected

cost is given by

J(u) = E[ZT

`(t, x(t),E[x(t)], u(t),E[u(t)])dt (2)

+ψ(x(T),E[x(T)])],

where, `: [0, T ]×R×R×U×R→R,and ψ:R×R→R.

The mean-ﬁeld-type control problem is as follows:

PMFTC := (min

u∈U

J(u),

subject to (1),and x0∼µ0.

Next, we present the proposed approach to solve PMFTC.

III. NEURAL NETWORKS FOR MEAN-FIELD-TYPE

PROBLEMS

In this section we deﬁne, rigorously, what we mean by a

neural network, then we show how it can be solved our mean-

ﬁeld control problem. A neural network is usually deﬁned by

an architecture, which is essentially, the number of hidden

layers, the number of neurons per layer and the activation

functions. We deﬁne the set of layer functions with input

dimension d1, output dimension d2, and activation functions

hj:R→R,j∈ {0, . . . , n},by

Lhj

d1,d2={φ:Rd1→Rd2|∃b∈Rd2,∃W∈Rd2×d1,

∀i∈ {1, . . . , d2}, φ(x)i=hj(bi+X

k=1

Wikzk)},

and we denote the set of neural networks with nhidden

layers and one output layer by

UNN ={gθ:Rd0→Rdn+1 |∀j∈ {0...,n},∃φj∈Lhj

dj,dj+1 ,

gθ=φn◦φn−1◦ · · · ◦ φ0}.

The vector band matrix Wfor each layer, are called the

parameters of the neural network, which we usually seek to

optimize through training. We denote them by

θ:= {W(0), b(0), W (1), b(1), . . . , W (n−1), b(n−1)},∀n∈N>2,

and we denote their set by Θ.

A. Neural Network Training as Data-Driven Control

We ﬁrst discretize time as follows. For a ﬁnite T > 0

and NT∈N+, let ∆t=T/NTand tk=k∆t, k ∈

{0, . . . , NT−1}. The discretized version of problem PMFTC

is given by the state dynamics

x(tk+1) = x(tk)(3)

+f(x(tk),E[x(tk)], u(tk),E[u(tk)])∆t+σBk,

Bk:= Btk+1 −Btk∼ N (0,∆t), k ∈ {0, . . . , NT−1},

and the associated cost function

J(u) = E[

NT−1

k=0

`(x(tk),E[x(tk)], u(tk),E[u(tk)])∆t

+ψ(x(tNT),E[x(tNT)])],(4)

In order to deal computationally with (4), we replace the

expectations by empirical averages over N-dimensional sam-

ple of state trajectories, with initial conditions independently

drawn from some distribution µ0. The same is done for the

corresponding control trajectories, i.e., the cost functional

becomes

JN(u) = 1

i=1 ψ(xi(tNT), µx(tNT))

NT−1

k=0

`(tk, xi(tk), µx(tk), ui(tk), µu(tk))∆t,

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

StabilityViaAdversarialTrainingofNeuralNetworkStochasticControlofMean-FieldTypeJulianBarreiro-GomezSalahEddineChoutriBoualemDjehicheAbstractInthispaper,wepresentanapproachtoneuralnetworkmean-eld-typecontrolanditsstochasticstabilityanalysisbymeansofadversarialinputs(akaadversarialat-tacks).Thisisac...

展开>> 收起<<

Stability Via Adversarial Training of Neural Network Stochastic Control of Mean-Field Type Julian Barreiro-Gomez Salah Eddine Choutri Boualem Djehiche.pdf

共6页,预览2页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Stability Via Adversarial Training of Neural Network Stochastic Control of Mean-Field Type Julian Barreiro-Gomez Salah Eddine Choutri Boualem Djehiche

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: