+
+ +
+

Archiving Data on Kronos

+

Kronos +is the mass storage system at OLCF. Each user has a directory of the form:

+
/nl/kronos/olcf/<projectID>/users/<userID>
+
+
+

and data can be transferred there using standard Unix commands.

+
+

Note

+

You need to be logged into dtn.olcf.ornl.gov to access kronos. It is +not visible directly from Frontier or Andes.

+
+

A submission / shell script pair that automates the transfer of data is available in +workflow/job_scripts/hpss as:

+
    +

  • olcf_kronos.submit : the slurm submission script

    • +

  • kronos_process.sh : a BASH script that finds output and automates the archiving.

    • +
+

You submit the job from the directory containing the plotfiles you wish to archive. +It will then:

+
    +

  • tar up the diagnostic files, inputs, and other metadata into a file with the +date-stamp in the file name and copy that to kronos

    • +

  • find all of the plotfiles and tar them directly to kronos. If the tar is successful, +it will move the plotfile into a plotfiles/ subdirectory and add a .processed +file so the script knows it was archived already.

    • +

  • find the checkpoint files matching a pattern (currently defaults to every 5000 steps) +and archive those in the same fashion, moving them to a checkfiles/ subdirectory +once archived.

    • +

  • loop, looking for new output files

    • +
+

By default, it will not transfer the last file, in case it is actively being written to.

+
+

Tip

+

The olcf_kronos_once.submit can be used to just transfer without the loop +waiting for new files.

+
+
+ + +
+