File sizes in WarpX

[Tissot11]

I asked this question in #5131 but I spilt these questions in separate threads. According to the documentation, only for format=chk all data has to be saved which should results in large files (>TB) for my simulations. However, using the Input Deck in #5131, I see significant large file sizes in (> 500 GB) for a smaller run. Using a CPU only code, I could manage (same grid, resolution, coarsening ratio) 50-60 GB. What is the source of this large file size? I could only reduce file sizes to few GB (expected) if I declare diagCombined.write_species = 0. But then I can not use particle tracking and save <diagName>.variables.

Also, if I use <diagName>.variables = 'x', 'ux' then I only expect to save these two variables. But WarpX saves also 'y' for the 2D simulations. I would expect WarpX to save only the quantities that I declared in the diagnostics?

[Tissot11]

After clarification in #5460, it is clear why the file sizes are large. However, if I only want to save <diagName>.fields_to_plot, even then I need to declare in the Full diagnostic <diagName>.write_species = 0 otherwise the file sizes remain huge. Please clarify what is actually saved by default in a diagnostic if it is declared as a Full diagnostic? This should then answer most of my questions.

[n01r]

Hi @Tissot11,

The current default for our FullDiagnostics is to write out all fields and all particle species with all variables.
If you want to have only field output in your diagnostics, you need to specify write_species = 0.
This does not hinder you from writing fields reduced from particle data, e.g., the average momentum of electrons per cell, from only forward-moving electrons:

diagInst.fields_to_plot = none
diagInst.write_species = 0
diagInst.particle_fields_to_plot = u
diagInst.particle_fields.u(x,y,z,ux,uy,uz) = sqrt(ux*ux + uy*uy + uz*uz)
diagInst.particle_fields_species = electrons
diagInst.particle_fields.do_average = 1
diagInst.particle_fields.u.filter(x,y,z,ux,uy,uz) = uz > 0

This will produce files with

(warpx-gpu) mgarten@nid001009:/pscratch/sd/m/mgarten/...> h5ls -r diags/diagInst/openpmd_000099.h5
/                        Group
/data                    Group
/data/99                 Group
/data/99/fields          Group
/data/99/fields/u_electrons Dataset {128, 96}

For only raw particle data, you need to set fields_to_plot to none. Particle-derived fields are turned off by default.
Specify <diag_name>.<species> = <species_name_0> <species_name_1> ... to limit which species will be written out. When specifying <diag_name>.<species_name>.variables, you can select any of the positions, and/or momenta, and/or the weighting w. It looks like id will always be written out. We could make that optional.

[Tissot11]

Many thanks for the clear explanations!

[n01r]

Thank you for your questions, @Tissot11!

If you have not done so yet, please consider adding yourself/your group or institute to the BLAST Community Map:

[Tissot11]

I'll do so after I get my other questions answered and I could reproduce my old results using WarpX!