Performance statistics, where?

[rbtoscan]

Is your feature request related to a problem? Please describe. For generic questions use Q&A section in the Discussions forum above.

hello,
it is said that as of version 3.14 there is performance statistics available. where, exactly? I am interested in the final runtime and amount of ram memory used for the process of metaspades assembly. thank you

Describe the solution you'd like

i guess a simple line in the log file would be enough.

Describe alternatives you've considered

No response

Additional context

No response

[asl]

Could you please elaborate what exactly and where is "said"?

[rbtoscan]

Readme.md file, line 158, last line on 'SPAdes performance', just before 'Installation'.

[asl]

You can certainly measure whatever performance statistics are relevant for your case and for your platform (as e.g. memory consumption could be really OS-dependent) as well as runtime.

Sill, lots of statistics were always readily available from the log, e.g.:

  0:00:02.286     8M / 317M  INFO    General                 (pipeline.cpp              : 287)   SPAdes finished
  0:00:02.287     1M / 317M  INFO    General                 (main.cpp                  : 136)   Assembling time: 0 hours 0 minutes 2 seconds

Here one can see the runtime of one SPAdes iteration as well as current and maximum RAM consumption. Overall running time could be obtained via system-standard time command.

[rbtoscan]

Dear Anton,

thank you for the reply. I just ran an assembly and produced a spades.log.

$ cat spades.log  | grep General | tail -n 5
  8:33:50.138   108G / 208G  INFO    General                 (contig_output_stage.cpp   :  98)   Outputting contigs to /storage-matrix/projects/smaft_polish_soil_microbiome/test_assembly/assembly_test2/K55/scaffolds.fasta
  8:34:13.143   108G / 208G  INFO    General                 (contig_output_stage.cpp   : 104)   Outputting FastG paths to /storage-matrix/projects/smaft_polish_soil_microbiome/test_assembly/assembly_test2/K55/scaffolds.paths
  8:34:30.420   108G / 208G  INFO    General                 (contig_output_stage.cpp   : 111)   Populating GFA with scaffold paths
  8:34:44.117   103G / 208G  INFO    General                 (pipeline.cpp              : 287)   SPAdes finished
  8:36:40.424     1M / 208G  INFO    General                 (main.cpp                  : 136)   Assembling time: 8 hours 36 minutes 40 seconds

Can I conclude based on this information that the peak use of memory was of 208 GB and the assembly time of was of 8hours and 36 mins? That is precisely the information that I need.

Thank you very much.

[asl]

SPAdes is a pipeline, so it runs several tools in a row (including external, e.g. pigz if available). Also, there might be multiple iterations of these tools. This shows one particular iteration.

Also, "peak use of memory" is not something very well defined. As at different times one might need RSS or virtual memory consumption. Or something else. What is shown is the amount asked from OS by SPAdes' internal memory allocator. OS-side of view might be different as it might include different overheads.

[rbtoscan]

I observed the entries and perhaps and I see it varies, so I propose the following one-liner to do the job of fetching the memory peak.

cat spades.log  | grep General | awk '{print $2$3$4}' | sed "s/\//\n/g" |  grep G | sort | uniq | sort -n | tail -n 1
248G

[asl]

There might be different log categories, not only General. You'd better just grab the very last message (Assembling time...). Also, if the memory consumption is less than 1 GiB, than M suffix will be used.