Less Emphasis on Hard Regions Curriculum Learn- ing of PINNs for Singularly Perturbed Convection- Diffusion-Reaction Problems

2025-05-02 0 0 4.87MB 22 页 10玖币

侵权投诉

Less Emphasis on Hard Regions: Curriculum Learn-

ing of PINNs for Singularly Perturbed Convection-

Diffusion-Reaction Problems

Yufeng Wang 1, Cong Xu 2, Min Yang 1 ,*and Jin Zhang3

1School of Mathematics and Information Sciences, Yantai University, Yantai,

China

2School of Computer Science and Technology, East China Normal University,

Shanghai, China

3Department of Mathematics, Shandong Normal University, Jinan, China

Abstract. Although Physics-Informed Neural Networks (PINNs) have been suc-

cessfully applied in a wide variety of science and engineering ﬁelds, they can fail

to accurately predict the underlying solution in slightly challenging convection-

diffusion-reaction problems. In this paper, we investigate the reason of this fail-

ure from a domain distribution perspective, and identify that learning multi-scale

ﬁelds simultaneously makes the network unable to advance its training and easily

get stuck in poor local minima. We show that the widespread experience of sam-

pling more collocation points in high-loss layer regions hardly help optimize and

may even worsen the results. These ﬁndings motivate the development of a novel

curriculum learning method that encourages neural networks to prioritize learning

on easier non-layer regions while downplaying learning on harder layer regions.

The proposed method helps PINNs automatically adjust the learning emphasis and

thereby facilitate the optimization procedure. Numerical results on typical bench-

mark equations show that the proposed curriculum learning approach mitigates

the failure modes of PINNs and can produce accurate results for very sharp bound-

ary and interior layers. Our work reveals that for equations whose solutions have

large scale differences, paying less attention to high-loss regions can be an effective

strategy for learning them accurately.

AMS subject classiﬁcations: 35Q68, 68T07, 68W25

Key words: physics-informed neural networks, convection-diffusion-reaction, boundary lay-

ers, interior layers, curriculum learning

∗Email addresses: zytuyufengwang@163.com (Y. Wang), congxueric@gmail.com (C. Xu),

yang@ytu.edu.cn (M. Yang),jinzhangalex@hotmail.com (J. Zhang)

arXiv:2210.12685v2 [cs.LG] 24 Mar 2023

1 Introduction

Convection-diffusion-reaction problems appear in the modeling of various modern

complicated processes, such as ﬂuid ﬂow at high Reynolds numbers [14], drift diffu-

sion in semiconductor device modeling [26], and chemical reactor theory [24]. Very

often the size of diffusion is characterized by a parameter e, which could be smaller

by several orders of magnitude compared to the size of convection and/or reaction,

resulting narrow boundary or interior layers in which the solution changes extremely

rapidly [28]. Classical numerical methods use layer-adapted meshes or introduce care-

fully designed artiﬁcial stability terms to solve these challenging problems [2,5,30,33,

34].

In recent years, there has been a surge of interest in applying neural networks in

traditional scientiﬁc modeling (e.g. partial differential equations), which yields the

so-called physics-informed neural networks (PINNs) [6,9,12,16,18,20,27,31]. The

main idea of PINNs is to include physical domain knowledge as soft constraints in

the empirical loss function and then use existing machine learning methodologies

such as stochastic optimization, to train the model. As an interesting alternative to

traditional numerical solvers, PINN has the advantage of ﬂexibility in dealing with

high-dimensional PDEs in complicated geometry and easy incorporation of available

data information. Moreover, well-trained PINNs can have good generalization ability

and can quickly predict solutions outside the computational area.

However, as reﬂected in some recent studies on the ”failure modes” of PINNs

[1,7,18], it has been found that PINNs can fail to converge to the correct solution even

for relatively simple convection-diffusion problems. Approaches to improve the accu-

racy of PINNs in solving convection-diffusion problems can be broadly classiﬁed into

two categories. The ﬁrst category borrows theories and concepts from conventional

numerical methods. For example, Mojgani et al. [25] rewrote the original equation into

a Lagrangian form on the characteristic curves and then applied a two-branch neural

network to solve the reformulated form. However, the approach is only applicable to

time-dependent problems and not to steady-state equations. Recently, inspired by the

theory of singular perturbation and asymptotic expansions, Arzani et al. [1] used sep-

arate neural networks to learn the different levels on the inner and outer layer regions,

respectively. The second category emphasizes machine learning techniques, such as

the design of loss functions, sample selection, and learning strategies. He et al. [13]

used a weighted sum of residual losses and showed that in order to obtain an accurate

solution of the advection-dispersion equation, the weights of the initial and boundary

conditions should be larger than the PDE residuals. Daw et al. [7] proposed an evo-

lutionary sampling algorithm in which the collocation points evolve gradually with

training to prioritize high-loss regions while maintaining a background distribution

of uniformly sampled points. Krishnapriyan et al. [18] argued that the PDE-based soft

constraints make the loss landscapes difﬁcult to optimize, and proposed a curriculum

approach that sets the PINN loss term starting with a simple equation regularization

and progressively become more complex as the network gets trained. But for strong

singular perturbation problems, the approach can be computationally overburdened

due to the need to learn many intermediate subproblems.

The existing studies mainly considered the relatively simple cases where the vis-

cosity/diffusivity is about a scale of 10−4. Singularly perturbed problems contain-

ing extremely sharp layers (strong vanishing viscosity/diffusivity limit) remains an

urgent target for PINNs. This paper aims to unravel the failure modes of PINNs

from some new perspectives and to further advance the approximation performance

of PINNs. We show that simultaneously learning multi-scale solutions in layer and

non-layer regions makes the network difﬁcult to advance its training and easily get

stuck in poor local minima. We demonstrate that in such case, prioritizing layer re-

gions (sampling more collocation points in high-loss regions) can make the training

more difﬁcult and worsen the performance. This surprising ﬁnding is contrary to

the majority of existing studies on PINNs. While most previous studies have em-

phasized high-loss regions, our investigation indicates that for problems containing

samples with extreme scale differences, it seems not a good idea to emphasize high-

loss regions. We argue that this is because collocation points from layer regions are

signiﬁcantly more challenging to learn than those from non-layer regions. To allevi-

ate the learning difﬁculties, we propose a novel curriculum learning approach that

can automatically adjust the sample weights to emphasize easier non-layer regions,

thereby improving the approximation accuracy of the network for strongly singular

perturbation problems. We empirically demonstrate the efﬁciency of the proposed ap-

proach in a variety of typical convection-diffusion-reaction problems. We show that

the proposed curriculum learning algorithm can mitigate the failure modes of vanilla

PINNs and well capture the sharp boundary or interior layers even in the cases of

very small diffusivity (e=10−9). Our approach successfully learns solutions contain-

ing very sharp layers, using only one neural network, without learning any inter-

mediate solutions. More importantly, we provide a new perspective to understand

the failure modes of PINNs and reveal that for equations whose solutions have large

scale differences, paying less attention to high-loss regions could be a feasible strat-

egy for learning them accurately. The source code built on PyTorch is available at

https://github.com/WYu-Feng/CLPINN to enable other researchers to reproduce and

extend the results.

The remainder of the paper is organized as follows. Section 2gives the problem

under study and introduces the basic notation of PINNs. A toy example is used in

Section 3to explore the possible reason for the failure mode of PINNs in solving sin-

gularly perturbed equations. In Section 4, we design a curriculum learning approach

to improve the performance of PINNs. Section 5gives comprehensive experimental

results to demonstrate the efﬁciency of the proposed method. Finally, the conclusion

is drawn in Section 6.

2 Problem Setup

Consider the following singularly perturbed equation:

Lu:=eL2u+L1u+L0u=f(x),x∈Ω, (2.1)

where Ωis a physical domain in Rd,Lkrepresents the differential operator of order k,

k=0,1,2, f(x)denotes the source term, and the diffusion coefﬁcient satisﬁes 0<e≤1.

Further assume that the solution u(x)satisﬁes the following boundary condition

Bu=g(x),x∈∂Ω. (2.2)

where Bis a well-deﬁned differential operator for determining the condition on the

admissible boundary ∂Ω. When the diffusion coefﬁcient eis very small, the latent

solution of the equation changes rapidly within some thin layers, posing a great chal-

lenge to the numerical simulation [5,23].

For PINNs, the solution u(x)is approximated by a neural network uθ(x), where θ

denotes the parameters of the network. Let

Lphys(θ) = 1

∑

i=1

phys(xi;θ) = 1

∑

i=1

[Luθ(xi)−f(xi)]2(2.3)

be the mean-squared physical residual loss of Ntraining sample points in Ω, and

Lbc(θ) = 1

∑

i=1

bc(xi;θ) = 1

∑

i=1

[Buθ(xi)−g(xi)]2(2.4)

be the mean-squared boundary loss of Mtraining sample points on ∂Ω. All the sam-

ples constitute a training set Xtrain.

The neural network approximation uθ(x)can be determined by solving the follow-

ing optimization objective

min

θLphys(θ)+λLbc(θ), (2.5)

where λis a hyperparameter to balance the weights of the two loss terms.

Although PINNs have been successfully applied in solving many types of differen-

tial equations, their performance for relatively simple convection-diffusion equations

are far from satisfactory. In the next section, we are to analyze the dilemma encoun-

tered by PINNs.

3 Analysis of Failure Mode

Consider the following one-dimensional convection-diffusion problem as an example:

−euxx +(x−2)ux=f(x),x∈(0,1),

u(0) = u(1) = 0.

(3.1)

where the diffusion coefﬁcient eis set as 10−3, and the source term f(x)is determined

by the exact solution u(x) = cos(πx/2)(1−exp(2x/e)). This problem has a boundary

layer at x=0.

Consider a four-layer fully connected neural network uθ(x), where each interme-

diate layer has 20 neurons and Tanh is used as the activation function. The training

set Xtrain consists of 2500 points uniformly sampled from the domain (0,1).

Different initializations and optimizations. The network parameters are initial-

ized by Normal Xaiver or Uniform Xaiver methods [8]. Two mainstream optimizers,

Stochastic Gradient Descent (SGD) [4] and Adam [17], are utilized to solve the opti-

mization objective (2.5), where the balance parameter λis set to 1.

0.0 0.2 0.4 0.6 0.8 1.0

Computational domain

−0.4

−0.2

0.0

0.2

0.4

0.6

0.8

1.0

Solution

Exact solution

Xavier Normal

Xavier Uniform

(a) SGD

0.0 0.2 0.4 0.6 0.8 1.0

Computational domain

−0.4

−0.2

0.0

0.2

0.4

0.6

0.8

1.0

Solution

Exact solution

Xavier Normal

Xavier Uniform

(b) Adam

Figure 1: Predictions of PINN under two parameter initializations using SGD and

Adam optimizers, respectively.

It can be observed from Figure 1that the prediction uθ(x)has very large errors

throughout the computational domain, regardless of the initial or training methods

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

LessEmphasisonHardRegions:CurriculumLearn-ingofPINNsforSingularlyPerturbedConvection-Diffusion-ReactionProblemsYufengWang1,CongXu2,MinYang1,*andJinZhang31SchoolofMathematicsandInformationSciences,YantaiUniversity,Yantai,China2SchoolofComputerScienceandTechnology,EastChinaNormalUniversity,Shanghai,Ch...

展开>> 收起<<

Less Emphasis on Hard Regions Curriculum Learn- ing of PINNs for Singularly Perturbed Convection- Diffusion-Reaction Problems.pdf

共22页,预览5页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Less Emphasis on Hard Regions Curriculum Learn- ing of PINNs for Singularly Perturbed Convection- Diffusion-Reaction Problems

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: