1 TRANSFORMER -BASED CONDITIONAL GENERATIVE ADVERSARIAL NETWORK FOR MULTIVARIATE TIME

2025-04-28 0 0 648.97KB 11 页 10玖币

侵权投诉

TRANSFORMER-BASED CONDITIONAL GENERATIVE

ADVERSARIAL NETWORK FOR MULTIVARIATE TIME

SERIES GENERATION

Abdellah Madane

madane@lipn.univ-paris13.fr Mohamed-Djallel DILMI

dilmi@lipn.univ-paris13.fr Florent Forest

forest@lipn.univ-paris13.fr

Hanane AZZAG

azzag@lipn.univ-paris13.fr Mustapha Lebbah

Mustapha.lebbah@uvsq.fr

Je´roˆme

Lacaille

jerome.lacaille@safrangroup.com

ABSTRACT

Conditional generation of time-dependent data is a task that has much interest,

whether for data augmentation, scenario simulation, completing missing data, or

other purposes. Recent works proposed a Transformer-based Time series genera-

tive adversarial network (TTS-GAN) to address the limitations of recurrent neu-

ral networks. However, this model assumes a unimodal distribution and tries to

generate samples around the expectation of the real data distribution. One of its

limitations is that it may generate a random multivariate time series; it may fail

to generate samples in the presence of multiple sub-components within an overall

distribution. One could train models to fit each sub-component separately to over-

come this limitation. Our work extends the TTS-GAN by conditioning its gener-

ated output on a particular encoded context allowing the use of one model to fit a

mixture distribution with multiple sub-components. Technically, it is a conditional

generative adversarial network that models realistic multivariate time series under

different types of conditions, such as categorical variables or multivariate time se-

ries. We evaluate our model on UniMiB Dataset, which contains acceleration data

following the XYZ axes of human activities collected using Smartphones. We

use qualitative evaluations and quantitative metrics such as Principal Component

Analysis (PCA), and we introduce a modified version of the Frechet inception

distance (FID) to measure the performance of our model and the statistical sim-

ilarities between the generated and the real data distributions. We show that this

transformer-based CGAN can generate realistic high-dimensional and long data

sequences under different kinds of conditions.

INTRODUCTION

Conditional generative adversarial networks have attracted significant interest recently (Hu et al.,

2021; Liu & Yin, 2021; Liu et al., 2021). The quality of generated samples by such models is im-

proving rapidly. One of their most exciting applications is multivariate time series generation, par-

ticularly when considering contextual knowledge to carry out this generation. Most published works

address this challenge by using recurrent architectures (Lu et al., 2022), which usually struggle with

long time series due to vanishing or exploding gradients. One other way to process sequential data

is via Transformer-based architectures. In the span of five years, Transformers have repeatedly ad-

vanced the state-of-the-art on many sequence modeling tasks (Yang et al., 2019)(Radford et al.,

2019)(Conneau & Lample, 2019). Thus, it was a matter of time before we could see transformer-

based solutions for time series (Yoon et al., 2019)(Wu et al., 2020)(Mohammadi Farsani & Pazouki,

2020), and particularly for multivariate time series generation (Li et al., 2022)(Leznik et al., 2021).

These studies showed promising results. Consequently, whether Transformer-based techniques are

suitable for conditional multivariate time series generation is an interesting problem to investigate.

Our work extends the TTS-GAN (Li et al., 2022) by conditioning its generated output on a par-

ticular encoded context allowing the use of one model to fit a mixture distribution with multiple

sub-components. Our contributions are summarized as follows:

•

We designed a transformer-based conditional generative adversarial network architecture

for conditional multivariate time series generation using different type of conditions.

•

We introduce and study a new parameter, alpha α, which controls the percentage of the de-

sired noise/variability and the percentage of relevance given to the context in a conditional

generative adversarial network.

•

We rigorously evaluate our approach on the UniMiB SHAR dataset using different qualita-

tive and quantitative evaluation methods.

•

We introduce MTS-FID, a version of FID suitable for evaluating generative models dealing

with multivariate time series. We show that MTS-FID exhibits the same behavior as FID

and that the results obtained correlate with other evaluations performed.

•

Using a data augmentation study, we demonstrate that our Transformer-based CGAN could

accurately model the mixture distribution of the classes given as a condition and generate

precise multivariate time series for each class.

RELATED WORKS

Conditional Generative Adversarial Networks (CGANs) (Mirza & Osindero, 2014) are an exten-

sion of vanilla GANs (Goodfellow et al., 2014). They allow to obtain a certain degree of control

over the generated samples. A CGAN model initially sets a condition that the generated data must

meet. This condition can take different forms: it can be the class of the image we want to generate

in the case of a GAN model trained for an image generation task, or some context encoding for a

time series generative model. CGAN extends the vanilla GAN architecture by incorporating a con-

dition in the form of a vector c concatenated with the noise vector z at the input of the generator and

provided as an input to the discriminator. Thus, this corresponds to conditioning the G(z) and D(x)

distributions. Therefore, the standard loss function of CGAN is as follows:

 



data

















Transformers, as presented in the seminal paper Vaswani et al. (2017), is an architecture relying

entirely on the attention mechanism to learn global dependencies between input and output, without

any notion of recurrence. Following this work, we realized that we could build architectures using

only attention mechanisms, which proved to be sufficient to understand and extract features from the

input given to the model. The subtlety here, and what makes a difference compared to a recurrent

network (RNN), is that Transformers process every sequence elements simultaneously, in parallel.

Also, the attention mechanism ensures the possible use of any element of the sequence whenever we

compute attention for it. Thus, it does not lose any relevant information whatsoever.

Transformer-based Generative Adversarial Networks

TransGAN (Jiang et al., 2021) introduces, for the first time, a generative adversarial network built

using solely Transformers that generates synthetic images. Like other GANs, the TransGAN con-

sists of two parts: a generator and a discriminator; first, the generator takes a one-dimensional noise

vector as an input and gradually increases the resolution of the feature map computed using trans-

former encoder blocks until a synthetic image with the required resolution is generated. As for the

discriminator, the authors adopted the exact model for Vision Transformer (ViT) (Dosovitskiy et al.,

2020a), an image classifier based on Transformers.

TTS-GAN (Li et al., 2022) is a transformer-based GAN where the generator and the discriminator

adopt the Trans-encoder architecture. It generates synthetic multidimensional time series for data

augmentation purposes. The process followed was similar to the TransGAN for image generation.

Authors of TTS-GAN consider a multivariate time series as an image of height equal to one and

length equal to the number of time steps and the number of variables in the multivariate time series

as the number of channels.

MTS-CGAN: MULTIVARIATE TIME SERIES CONDITIONAL GENERATIVE

ADVERSARIAL NETWORK

We propose a conditional generative adversarial network where the generator and the discriminator

are purely transformer-based neural networks. This model generates multivariate time series given

a context. This context could be categorical variables or a multivariate time series. The model is

discussed and evaluated in detail in the following sections.

3.1

ARCHITECTURE

MTS-CGAN architecture is inspired by the models introduced in Jiang et al. (2021); Dosovitskiy

et al. (2020b); Li et al. (2022). Like other GANs, it consists in a generator (G) and a Discriminator

(D), as shown in Figure 1.

Figure 1: MTS-CGAN architecture.

The conditional generator has two inputs: the random noise vector of dimension dz and the en-

coded context to condition the generation. The latent dimension is a hyper-parameter depending on

the dataset. Also, the context is encoded to fit a latent space of dim dz to facilitate its concatenation

with the noise vector z. We then apply linear transformations to the concatenated vectors to obtain

a vector of size equal to the target sequence length and with c channels, where c needs to be tuned.

Finally, we encode the position of the elements of this vector. The resulting vector passes through

the consecutive layers of a Transformer encoder. Each has a multi-head self-attention layer that ex-

tracts the contextual inter-dependencies between the generated signal and the provided context. The

final output of those layers passes through a

(1, 1)

-convolution layer with a filter size equal to the

dimension of the time series we are aiming to generate.

Alpha α : Every modeling deals with the object of interest – in our case, a multivariate time series

– as a composition of two components: a regular and an irregular one. Contextual knowledge

and prior information encode evidence in favor of some propositions and help to characterize the

regular component of the time series while a random variable models the irregular one. Thus, it

seems natural to associate a weight α with the random variable that models the lack of knowledge.

This hyper-parameter 0

1 defines the percentage of the desired noise/variability and the

percentage of relevance given to the context by complementary. Therefore, we multiply the noise z

by α and the context vector by 1 − α.

The conditional discriminator aims at classifying the time series as real or synthetic and takes as

input either a real time series with its corresponding context, or a generated one. First, it concatenates

the input vectors and applies a linear transformation. Then, the resulting embedding is broken down

into multiple patches associated with their respective positions and a classification token. The whole

set is then fed to the consecutive layers of the Transformer’s encoder, as was the case in the generator.

Finally, a binary classifier uses the information embedded into the classification token to distinguish

between the real and fake signals.

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

1TRANSFORMER-BASEDCONDITIONALGENERATIVEADVERSARIALNETWORKFORMULTIVARIATETIMESERIESGENERATIONAbdellahMadanemadane@lipn.univ-paris13.frMohamed-DjallelDILMIdilmi@lipn.univ-paris13.frFlorentForestforest@lipn.univ-paris13.frHananeAZZAGazzag@lipn.univ-paris13.frMustaphaLebbahMustapha.lebbah@uvsq.frJe´roˆm...

展开>> 收起<<

1 TRANSFORMER -BASED CONDITIONAL GENERATIVE ADVERSARIAL NETWORK FOR MULTIVARIATE TIME.pdf

共11页,预览3页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

1 TRANSFORMER -BASED CONDITIONAL GENERATIVE ADVERSARIAL NETWORK FOR MULTIVARIATE TIME

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: