Continuous Simulation Data Stream A dynamical timescale-dependent output scheme for simulations_2

2025-05-01 0 0 1.62MB 20 页 10玖币

侵权投诉

Continuous Simulation Data Stream:

A dynamical timescale-dependent output scheme for simulations

Lo¨

ıc Hausammann1,∗

Laboratoire d’Astrophysique, Ecole Polytechnique F´ed´erale de Lausanne (EPFL), 1290 Sauvergny, Switzerland

Pedro Gonnet

Google Switzerland, 8002 Z¨urich, Switzerland

Matthieu Schaller

Lorentz Institute for Theoretical Physics, Leiden University, PO Box 9506, NL-2300 RA Leiden, The Netherlands

Leiden Observatory, Leiden University, PO Box 9513, NL-2300 RA Leiden, The Netherlands

Abstract

Exa-scale simulations are on the horizon but almost no new design for the output has been proposed in recent

years. In simulations using individual time steps, the traditional snapshots are over resolving particles/cells

with large time steps and are under resolving the particles/cells with short time steps. Therefore, they are

unable to follow fast events and use eﬃciently the storage space. The Continuous Simulation Data Stream

(CSDS) is designed to decrease this space while providing an accurate state of the simulation at any time.

It takes advantage of the individual time step to ensure the same relative accuracy for all the particles. The

outputs consist of a single ﬁle representing the full evolution of the simulation. Within this ﬁle, the particles

are written independently and at their own frequency. Through the interpolation of the records, the state of the

simulation can be recovered at any point in time. In this paper, we show that the CSDS can reduce the storage

space by 2.76x for the same accuracy than snapshots or increase the accuracy by 67.8x for the same storage

space whilst retaining an acceptable reading speed for analysis. By using interpolation between records, the

CSDS provides the state of the simulation, with a high accuracy, at any time. This should largely improve the

analysis of fast events such as supernovae and simplify the construction of light-cone outputs.

Keywords: methods: numerical, software: simulations, simulations: I/O, galaxies: evolution

1. Introduction

With the arrival of the era of exa-scale comput-

ing, scientists across all domains have been focused

on improving the performances of their software. They

have been mostly looking at the scaling of their own

physical computation (examples in astrophysics in-

clude Schaller et al. 2016; Jetley et al. 2008; Potter

∗Corresponding author

Email address: loic.hausammann@protonmail.com

(Lo¨

ıc Hausammann)

et al. 2017; Adams et al. 2009; M¨

uller et al. 2019)

with, so far, little attention given to the way of writ-

ing the outputs. Indeed, in recent years, the HDF5

format (The HDF Group 1997) has become the de-

facto standard with some applications also relying

on custom-made binary ﬁles (e.g. Maksimova et al.

2021; Potter et al. 2017). Scientists rely often solely

on the improvements within the HDF5 library itself

or on the hardware for their own code. This approach

is generally successful with simulations writing data

in parallel from 1000s of inter-connected nodes. This

Preprint submitted to Elsevier October 4, 2022

arXiv:2210.00835v1 [astro-ph.IM] 3 Oct 2022

format is particularly well adapted for snapshots that

simply write the state of the simulation at a few dis-

crete times (e.g. Nelson et al. 2015; Norman et al.

2007) containing up to trillions of particles Potter

et al. (2017); Maksimova et al. (2021); Bocquet et al.

(2016). Whilst some studies or software develop-

ments have been focused on improving the perfor-

mances of the I/O (e.g. Xiao et al. 2012; Ross et al.

2008; Ma et al. 2006; Mitra et al. 2005; Ragagnin

et al. 2017), or on increasing the level of abstraction

(e.g. Godoy et al. 2020; L¨

uttgau et al. 2018; Zheng

et al. 2013; Abbasi et al. 2009), little work has been

done on rethinking the general framework used to

store data from simulations.

An interesting case where such rethinking was

done is the construction of light-cone data. Such out-

puts reproduce the behavior of observations, where

looking further away corresponds to looking back in

time. As only the particles contained within the light

cone surface are interesting, techniques have been

developed to increase the performance of the simula-

tion (e.g. Garaldi et al. 2020) and to write the outputs

in a more eﬃcient way (e.g. Evrard et al. 2002). This

last technique consists in writing a type of snapshot

where the particles are written only when they cross

the light cone surface. While this approach needed

to rethink the outputs, it is not a general solution for

astrophysics.

In cosmological simulations (but also more gen-

erally in other simulations where many time-scales

are coupled), the gravity is producing large diﬀer-

ence of time scales through the simulated volume.

As it would require too much computational time

to use a single, global, time-step size, a multi (lo-

cal) and individual time step approach was designed

where each particle or cell is evolved at its own time

scale. This approach ensures that all the particles

or cells are evolved at the same relative accuracy

(Aarseth 1963; Springel 2005). As an example of

this, in Fig. 1, we show the distribution of time steps

within a cosmological simulation run with the EA-

GLE model (Schaye et al. 2015). Only an almost

negligible fraction of the particles actually needs a

small time step and the simulation analysis often fo-

cuses on them. They are located in the most active

regions of the simulation Even if this example sim-

ulation is relatively small (107particles), already a

2022242628210

Time step

100

101

102

103

104

105

106

Number of particles

Figure 1: Distribution of time-step sizes (normalized to the

smallest time-step) extracted from a cosmological simulation

run with the EAGLE model within a periodic box of 12 Mpc

at z=3. In this simulation, the ratio between the largest and

smallest time steps is already above 1000 and only a small frac-

tion of the particles really need a high time resolution.

ratio of 1000 can be observed between the smallest

and largest time-step sizes. For the outputs, it means

that writing all the particles/cells together can poten-

tially provide a far too high accuracy for particles/-

cells within regions using long time-steps (e.g. in

voids) and far too low for particles/cells using very

short steps (e.g. within galaxies). A consequence of

this is that the traditional snapshot model is unable

to follow accurately fast events such as supernovae

feedback eﬀects and render their interpretation more

complicated. Only by writing an impossibly large

number of snapshots could such events be properly

followed in large-scale simulations. To remedy this

issue, many codes are relying on an additional set

of ﬁles that describes such fast events (e.g. writing

all the supernovae into a ﬁle). While this approach

is suﬃcient in many cases, it lacks a complete de-

scription of the event’s environment and its impact

through time. It does also require an a priori knowl-

edge of what events will be of interest.

In this paper, we present a novel approach, the

Continuous Simulation Data Stream (CSDS), which

allows us to recover the state of the simulation at

any time with higher accuracy and a lower storage

cost than snapshots. To simplify the discussion, in

this paper, we will focus on particle based codes, but

this method could certainly be adapted for Adaptive

Mesh Reﬁnements (AMR) or to any other general

method. In practice, the CSDS is implemented as a

stand-alone module around the open-source cosmo-

logical code Swift1(Schaller et al. 2016, 2018).

The paper is organized as follows. In section 2,

we describe in details the theory behind the CSDS.

In section 3, the implementation within Swift is pro-

vided followed by section 4 where the reading strat-

egy is explained. Our results are separated in two

sections (5 and 7). The ﬁrst one presents the eﬃ-

ciency of the CSDS and the second one shows some

examples of applications.

The implementation presented in this paper is fully

open-source and included as part of the Swift repos-

itory2.

2. The Continuous Simulation Data Stream

The Continuous Simulation Data Stream (CSDS)

is a new output strategy and mechanism aimed at re-

placing or supplementing, at least partially, the tra-

ditional snapshot system. Its main advantage resides

in the writing of each individual resolution element

in their own mini logbook. It hence allows to cap-

ture their evolution according to their own time-scale

and not only at the ﬁxed time resolution set globally

by traditional snapshots. Thus, it perfectly adapts

to simulations using individual localized time-step

sizes and can achieve extremely high time resolu-

tion while keeping the output size reasonable. Cap-

turing the detailed progression of the elements using

the smallest time-step bin would require a prohibitive

number of ﬁxed-time snapshots; whilst our mecha-

nism allows us to track this evolution precisely.

2.1. Overview of the diﬀerent components

To achieve this high resolution, the CSDS uses a

single ﬁle, called the logﬁle, that describes the whole

evolution of the simulation. Within this ﬁle, all the

1www.swiftsim.com

2For the csds-reader, we used version 1.5 (available

at https://gitlab.cosma.dur.ac.uk/lhausammann/

csds-reader) in this paper. More speciﬁcally, all the plots

and numbers shown in this paper were obtained using commit

d078e2dd of Swift and commit c073baf0 of the reader code.

information is written in the form of per-particle records

(except for the header containing some general meta-

data). This approach allows to write the particles

individually and at their own timescale without any

need to synchronize them during the simulation. When

reading the CSDS for analysis, a synchronization is

still required and done through an interpolation.

To construct the logﬁle, a general header is writ-

ten during the initialization of the simulation. Then,

a time record for the initial step is written followed

by a particle record for all the particles in the vol-

ume (i.e. this corresponds to the initial conditions).

The particles then carry a clock (see below) giving

them an individual dump frequency. At each step of

the simulation, the CSDS writes a time record and a

particle record for the particles whose timer is up.

At the end of the simulation, all the particles are

written one ﬁnal time and a time record concludes

the logﬁle. Users can hence reconstruct the entire

history of a given particle by hoping from record to

record through the logﬁle. If a user requests details

about a given particle in-between two time-records,

interpolation is used. The accuracy (faithfulness) of

the whole mechanism is entirely dictated by the fre-

quency of the dump of each particle. We note, for in-

stance, that for the simulation shown on Fig 1, a large

fraction of particles would not need any records be-

tween their initial and ﬁnal dump as their trajectories

can be well recovered just by interpolation.

To simplify the reading of the logﬁle, the CSDS

uses a set of index ﬁles that describe the state of the

simulation at a given time. While this mimics the be-

havior of snapshots, they tend to be smaller as they

only contain, for each particle, the last known loca-

tion within the logﬁle and IDs of the particles. The

index ﬁles contain also the history of the particles

created and deleted during the simulation (e.g. for

MPI exchanges, star formation and black hole inter-

actions for cosmological simulations). Theses ﬁles

are generated after the simulation and, from our ex-

perience, a small number (∼10) of index ﬁles is suf-

ﬁcient to achieve good analysis performance.

Finally, we also include a metadata ﬁle that con-

tains details on the simulation (e.g. units, cosmolog-

ical model, subgrid parameters). This ﬁle will not be

discussed further as the details are not important to

the CSDS mechanism.

As a summary, the diﬀerent objects used in the

CSDS and their relations are shown in Fig. 2.

2.2. Description of the records

Each record in the logﬁle is made of a small header

followed by the data, as shown in the zoom-in at the

bottom of Fig. 2. In the header, we store a mask and

an oﬀset.

The mask is a simple bit mask describing the ﬁelds

contained in the data (coordinates, velocities, inter-

nal energies, etc.). This means that based only on

the bit mask, a record can be identiﬁed as contain-

ing either the current time or a particle ﬁeld. It also

means that we can have diﬀerent writing frequencies

for diﬀerent particle ﬁelds (e.g. metallicities evolve

much more smoothly than temperatures, eﬀectively

requiring fewer entries). The oﬀset is the distance

in the ﬁle to the previous record of the particle. It

means that the evolution of a particle can be quickly

followed backwards thanks to this oﬀset.

As usually we are interested in moving forward

in time, the oﬀset of the records are reversed during

the ﬁrst read of a logﬁle after a simulation has com-

pleted 3.

In the data part of a record, we simply copy the

ﬁelds corresponding to the mask into the ﬁle. As

the diﬀerent ﬁelds are written the one after the other

without any information in between, the order when

writing and reading a record must be respected. This

order is indicated in the logﬁle header.

2.3. Description of the logﬁle

The logﬁle starts with a header that contains the

version, the direction of the oﬀset (in order to know if

they need to be reversed), the size of the strings, the

number of diﬀerent masks, all the available masks

(name and data size), and ﬁnally the masks for each

particle types (in the writing order). It could be ex-

tended with more data without risk as we also store

the position of the ﬁrst record. This general structure

is shown on Fig. 2.

When the simulation starts, the initial time is writ-

ten followed by a record for each particle presents in

the initial conditions. As the CSDS requires some

3This operation is one of the slowest. Thus we might drop it

in the future and encourage the users to move backward in time.

interpolations, this step is required in order to avoid

any extrapolation. Then at each simulation step, the

CSDS writes a time record followed by as many par-

ticle records as required by their individual write fre-

quencies (see section 2.6 for a discussion about the

writing criterion). The fraction of particles written

during a single step can largely diﬀer between diﬀer-

ent steps. On some steps, the CSDS writes 0 parti-

cle due to a low number of active particles and/or a

lack of active particles requiring a log. The opposite

case also exists where all the particles are written.

At the end of the simulation, we write the ﬁnal time

followed by all the particles in order to avoid any ex-

trapolation and conclude with a copy of the last time

record as a sentinel marking the end of the ﬁle.

2.4. Restart Strategy

HPC software should expect to be killed at any

time due to a system failure or to be limited in the

amount of time they can run. In order to restart the

simulation, diﬀerent strategies exist, but all relies on

dumping at least part of their memory into so-called

check-point ﬁles and restarting from them.

The CSDS is no diﬀerent from the rest of the

software package in this situation. While the sim-

ulation would be written to the check-point ﬁels as

usual, the CSDS simply dumps the position of the

last record written in the logﬁle. When restarting,

the CSDS starts writing after the position of the last

record and thus automatically get rid of the unneces-

sary data written between the last restart ﬁle and the

crash. We emphasize that we do not intend the CSDS

mechanism to be used as a check-pointing mecha-

nism by itself.

2.5. Description of the index ﬁle

The index ﬁles are totally optional as they contain

information that exists in the logﬁle. They, however,

allow for an eﬃcient usage of the logﬁle. For exam-

ple, if the simulation at the ﬁnal time is requested,

without the index ﬁles, it would be required to read

the whole logﬁle in order to ﬁnd all the existing par-

ticles and then update them until reaching the ﬁnal

time. To solve this problem, the index ﬁles con-

tain enough information to reconstruct quickly the

state of a simulation at a given time (like a traditional

snapshot would) using the logﬁle.

文档加载中……请稍候！
如果长时间未打开，您也可以点击刷新试试。

下载文档到电脑，查找使用更方便

10 玖币 0人已下载

立即下载

摘要：

ContinuousSimulationDataStream:Adynamicaltimescale-dependentoutputschemeforsimulationsLo¨cHausammann1,Laboratoired'Astrophysique,EcolePolytechniqueF´ed´eraledeLausanne(EPFL),1290Sauvergny,SwitzerlandPedroGonnetGoogleSwitzerland,8002Z¨urich,SwitzerlandMatthieuSchallerLorentzInstituteforTheoreticalP...

展开>> 收起<<

Continuous Simulation Data Stream A dynamical timescale-dependent output scheme for simulations_2.pdf

共20页,预览4页

还剩页未读，继续阅读

声明：本站为文档C2C交易模式，即用户上传的文档直接被用户下载，本站只是中间服务平台，本站所有文档下载所得的收益归上传人(含作者)所有。玖贝云文库仅提供信息存储空间，仅对用户上传内容的表现方式做保护处理，对上载内容本身不做任何修改或编辑。若文档所含内容侵犯了您的版权或隐私，请立即通知玖贝云文库，我们立即给予删除！

Continuous Simulation Data Stream A dynamical timescale-dependent output scheme for simulations_2

相关推荐

开通VIP享超值会员特权

作者详情

相关内容

热门标签

举报选择: