Linear attention coupled Fourier neural operator for simulation of three-dimensional turbulence Wenhui Peng 彭文辉123 Zelong Yuan 袁泽龙12

2025-05-03 0 0 2.65MB 36 页 10玖币

侵权投诉

Linear attention coupled Fourier neural operator for simulation of

three-dimensional turbulence

Wenhui Peng (彭文辉)1,2,3, Zelong Yuan (袁泽龙)1,2,

Zhijie Li (李志杰)1,2, and Jianchun Wang (王建春)1,2∗

1Department of Mechanics and Aerospace Engineering,

Southern University of Science and Technology, Shenzhen 518055, China

2Guangdong-Hong Kong-Macao Joint Laboratory for

Data-Driven Fluid Mechanics and Engineering Applications,

Southern University of Science and Technology, Shenzhen 518055, China and

3Department of Computer Engineering,

Polytechnique Montreal, H3T1J4, Canada

(Dated: November 28, 2022)

Abstract

Modeling three-dimensional (3D) turbulence by neural networks is diﬃcult because 3D turbu-

lence is highly-nonlinear with high degrees of freedom and the corresponding simulation is memory-

intensive. Recently, the attention mechanism has been shown as a promising approach to boost

the performance of neural networks on turbulence simulation. However, the standard self-attention

mechanism uses O(n2) time and space with respect to input dimension n, and such quadratic com-

plexity has become the main bottleneck for attention to be applied on 3D turbulence simulation.

In this work, we resolve this issue with the concept of linear attention network. The linear atten-

tion approximates the standard attention by adding two linear projections, reducing the overall

self-attention complexity from O(n2) to O(n) in both time and space. The linear attention coupled

Fourier neural operator (LAFNO) is developed for the simulation of 3D isotropic turbulence and

free shear turbulence. Numerical simulations show that the linear attention mechanism provides

40% error reduction at the same level of computational cost, and LAFNO can accurately recon-

struct a variety of statistics and instantaneous spatial structures of 3D turbulence. The linear

attention method would be helpful for the improvement of neural network models of 3D nonlinear

problems involving high-dimensional data in other scientiﬁc domains.

∗wangjc@sustech.edu.cn

arXiv:2210.04259v2 [physics.flu-dyn] 24 Nov 2022

I. INTRODUCTION

With the rising of deep learning techniques, neural networks (NNs) have been extensively

explored to complement or accelerate the traditional computational ﬂuid dynamics (CFD)

modeling of turbulent ﬂows [1,2]. Applications of deep learning and machine learning tech-

niques to CFD include approaches to the improvements of Reynolds averaged Navier–Stokes

(RANS) and large eddy simulation (LES) methods. These eﬀorts have mainly focused on us-

ing NNs to learn closures of Reynolds stress and subgrid-scale (SGS) stress and thus improve

the accuracy of turbulence modeling [3–5].

Deep neural networks (DNNs) have achieved impressive performance in approximating

the highly non-linear functions[6]. Guan et al. proposed the convolutional neural network

model to predict the SGS forcing terms in two-dimensional decaying turbulence [7].Yang

et al. incorporated the vertically integrated thin-boundary-layer equations into the model

inputs to enhance the extrapolation capabilities of neural networks for large-eddy-simulation

wall modeling [8]. Some recent works aim to approximate the entire Navier-Stokes equations

by deep neural networks [9–17]. Once trained, the “black-box” NN models can make infer-

ence within seconds on modern computers, thus can be extremely eﬃcient compared with

traditional CFD approaches [18]. Xu et al. employed the physics-informed deep learning by

treating the governing equations as a parameterized constraint to reconstruct the missing

ﬂow dynamics[19]. Wang et al. further applied the physical constraints into the design

of neural network, and proposed a grounded in principled physics model: the turbulent-

ﬂow network (TF-Net). The architecture of TF-Net contains trainable spectral ﬁlters in

a coupled model of Reynolds-averaged Navier-Stokes simulation and large eddy simulation,

followed by a specialized U-net for prediction. The TF-Net oﬀers the ﬂexibility of the learned

representations, and achieves state-of-the-art prediction accuracy [20].

Most neural network architectures aim to learn the mappings between ﬁnite-dimensional

Euclidean spaces. They are good at learning a single instance of the governing equation,

but they can not generalize well once the given equation parameters or boundary conditions

change [21–24]. Li et al. proposed the Fourier neural operators (FNO), which learns an

entire family of partial diﬀerential equations (PDEs) instead of a single equation [25]. The

FNO mimics the pseudo-spectral methods [26,27]: it parameterizes the integral kernel in the

Fourier space, thus directly learns the mapping from any functional parametric dependence

to the solution [25]. Beneﬁted from the expressive and eﬃcient architecture, the FNO

outperforms the previous state-of-the-art neural network models, including U-Net [28],TF-

Net [20] and ResNet [29]. The FNO achieves 1% error rate on prediction task of two-

dimensional (2D) turbulence at low Reynolds numbers.

Direct numerical simulation (DNS) of the three-dimensional turbulent ﬂows is memory

intensive and computational expensive, due to the highly nonlinear characteristics of tur-

bulence associated with the large number of degrees of freedom. In recent years, there has

been extensive works dealing with the spatio-temporal reconstruction of two-dimensional

turbulent ﬂows [28,30–39]. These works reduce the reconstruction error mainly through

adopting advanced neural network models [25,35,40–44] or incorporating the prior physical

knowledge into the model [20,21,27,45–48].

However, modeling of three-dimensional turbulence with deep neural networks is more

challenging. The size and dimension of simulation data increases dramatically from 2D

to 3D [49,50]. In addition, modeling the non-linear interactions of such high-dimensional

data requires suﬃcient model complexity and huge number of parameters with hundreds of

layers not being uncommon [51]. Training such models can be computationally expensive

because of the sheer amount of parameters involved. Further, these models also take up a

lot of memory which can be a major concern during training, since deep neural networks

are typically trained on graphical processing units (GPUs), where the available memory is

often constrained.

Arvind et al. ﬁrst designed and evaluated two NN models for 3D homogeneous isotropic

turbulence simulation [52]. In their work, they proposed two deep learning models: the

convolutional generative adversarial network (C-GAN) and the compressed convolutional

long-short-term-memory (CC-LSTM) network. They evaluated the reconstruction quality

and computational eﬃciency of the two diﬀerent approaches. They employed convolutional

layers in GANs (CGANs) to handle the high dimensional 3D turbulence data. The pro-

posed CGANs model consists of an eight-layer discriminator and a ﬁve-layer generator. The

generator takes a latent vector that is sampled from the uniform distribution as an input

and produces a cubic snapshot (of the same dimensions as the input) as an output [52].

The CGANs model has an acceptable accuracy in modelling the velocity features of individ-

ual snapshots of the ﬂow, but has diﬃculties in modelling the probability density functions

(PDFs) of the passive scalars advected with the velocity [52].

Another model adopts a convolutional LSTM (ConvLSTM) network, which embeds the

convolution kernels in a LSTM network to simultaneously model the spatial and tempo-

ral features of the turbulence data [53]. However, the major limitation of ConvLSTM is

the huge memory cost due to the complexity of embedding a convolutional kernel in an

LSTM and unrolling the network [53], especially when dealing with the high-dimensional

turbulence data. The authors resolved this challenge of large-size data memory by train-

ing the ConvLSTM on the low dimensional representation (LDR) of turbulence data. They

used a convolutional autoencoder (CAE) to learn compressed, low dimensional ‘latent space’

representations for each snapshot of the turbulent ﬂow. The CAE contains multiple convo-

lutional layers, greatly reducing the dimensionality of the data by utilising the convolutional

operators [52]. The convolution ﬁlters are chosen since they can capture the complex spatial

correlations and also reduce the number of weights due to the parameter-sharing mechanism

[6]. The ConvLSTM takes the compressed low dimensional representations as input, and

predicts future instantaneous ﬂow in latent space which is then ‘decompressed’ to recover

the original dimension [52]. The CC-LSTM is able to predict the spatio-temporal dynamics

of ﬂow: the model can accurately predict the large scale kinetic energy spectra, but diverges

in the small scale range.

Nakamura et al. applied the CC-LSTM framework to three-dimensional channel ﬂow

prediction task [54]. Despite that the convolutional autoencoder (CAE) can accurately re-

construct the three-dimensional DNS data through the compressed latent space, the LSTM

network fails to accurately predict the future instantaneous ﬂow ﬁelds [54]. Accurate predic-

tion of three-dimensional turbulence is still one of the most challenging problems for neural

networks.

In recent years, attention mechanism has been widely used in boosting the performance

of neural networks on a variety of tasks, ranging from nature language processing to com-

puter vision [55–57]. The ﬂuid dynamics community has no exception. Wu et al. introduced

the self-attention into a convolution auto-encoder to extract temporal feature relationships

from high-ﬁdelity numerical solutions [58]. The self-attention module was coupled with the

convolutional neural network to enhance the non-local information perception ability of the

network and improve the feature extraction ability of the network. They showed that the

self-attention based convolutional auto-encoder reduces the prediction error by 42.9%, com-

pared with the original convolutional auto-encoder [58]. Deo et al. proposed an attention-

based convolutional recurrent autoencoder to model the phenomena of wave propagation.

They showed that the attention-based sequence-to-sequence network can encode the input

sequence and predict for multiple time steps in the future. They also demonstrated that

attention based sequence-to-sequence network increases the time-horizon of prediction by

ﬁve times compared to the plain sequence-to-sequence model [59]. Liu et al. used a graph

attention neural network to simulate the 2Dﬂow around a cylinder. They showed that

the multi-head attention mechanism can signiﬁcantly improve the prediction accuracy for

dynamic ﬂow ﬁelds [60]. Kissas et al. coupled the attention mechanism with the neural

operators towards learning the partial diﬀerential equations task. They demonstrated that

the attention mechanism provides more robustness against noisy data and smaller spread of

errors over testing data [61]. Peng et al. proposed to model the nonequilibrium feature of

turbulence with the self-attention mechanism [40]. They coupled the self-attention module

with the Fourier neural operator for the 2Dturbulence simulation task. They reported

that the attention mechanism provided 40% prediction error reduction compared with the

original Fourier neural operator model [40].

The attention mechanism has shown itself to be very successful at boosting the neural

networks performance for turbulence simulations, and therefore bringing new opportunities

to improve the prediction accuracy of 3Dturbulence simulation. However, extending the

attention mechanism to 3Dturbulence simulation is a non-trivial task. The challenge comes

from the computational expense of the self-attention matrix: the standard self-attention

mechanism uses O(n2) time and space with respect to input dimension n[55]. On the

other hand, neural networks are often trained on GPUs, where the memory is constrained.

Such quadratic complexity has become the main bottleneck for the attention mechanism

to be extended to 3D turbulence simulations. Detailed computational cost and memory

consumption are discussed in section III A.

Recently, Wang et al. demonstrated that the self-attention mechanism can be approxi-

mated by a low-rank matrix [62]. They proposed the linear attention approximation, which

reduces the overall self-attention complexity from O(n2) to O(n) in both time and space [62].

The linear attention approximation performs on par with standard self-attention, while be-

ing much more memory and time eﬃcient [62], allowing attention module to be applied on

high-dimensional data. In this work, we couple the linear attention module with the Fourier

neural operator, for the 3Dturbulence simulation task.

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

LinearattentioncoupledFourierneuraloperatorforsimulationofthree-dimensionalturbulenceWenhuiPeng(彭文辉)1;2;3,ZelongYuan(袁泽龙)1;2,ZhijieLi(李志杰)1;2,andJianchunWang(王建春)1;21DepartmentofMechanicsandAerospaceEngineering,SouthernUniversityofScienceandTechnology,Shenzhen518055,China2Guangdong-HongKong-MacaoJo...

展开>> 收起<<

Linear attention coupled Fourier neural operator for simulation of three-dimensional turbulence Wenhui Peng 彭文辉123 Zelong Yuan 袁泽龙12.pdf

共36页,预览5页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Linear attention coupled Fourier neural operator for simulation of three-dimensional turbulence Wenhui Peng 彭文辉123 Zelong Yuan 袁泽龙12

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: