Mesh setting up message at the beginning of output

[xueyang94]

I sometimes randomly (but rarely) receive this message at the beginning of the output when running a moose input file. With this message, the simulation takes a long time to run. But when I ran the same input file for a second time, I don't receive the mesh setting message and everything is normal. Where does this message comes from and how to make sure the future run does not have this message? Thanks. -Xueyang

[GiudGiud]

Hello

These are messages from the live perf graph. It's outputting to let you know it's not stuck.
https://mooseframework.inl.gov/source/utils/PerfGraph.html
I imagine the Reading mesh ... is always there and the Updating mesh ... message is only sometimes there?
Are you running with a distributed mesh?

Guillaume

[GiudGiud]

if you can share your inputs I can see if I can reproduce this. But the fact that this affects the whole simulation and not parts of it points to an issue with either your machine or how moose is being ran rather than a buggy feature which would only be present in parts of the simulation.
Are you running threads or MPI?

[xueyang94]

I'm running MPI. On both cluster and my local machine, with that message, the simulation is slowed down. My input file imports mesh from an exodus file which is 25GB. How would you prefer me to share both the exodus and input file? Thanks for your help.

[GiudGiud]

That message appears because the simulation is slow I think, not the other way around.
With MPI a classic 2x slowdown is if two processes get bound to the same core. Or if you are using all the cores but one of the cores is busy doing something else for the system (often an issue on HPC).

How many processes are you using and how many cores are there on the machines?
If you can share those I can see if it's reproducible on INL HPC.

[xueyang94]

On my university HPC, for one simulation I request 8 nodes and 32 MPI processes. Each node has 2 CPU and 32 cores. So I use 16 CPU and 256 cores in total.

My local machine has 1 CPU and 8 cores in total. I use all cores to execute a simulation: mpirun -np 8 -opt -i.

[GiudGiud]

So you could send me the inputs by email. My email is on my github profile.
What application do I need to run it? just moose?
I ll see if I can reproduce the slowdown.

[GiudGiud]

@xueyang94 any luck finding the culprit?

[xueyang94]

@GiudGiud I have emailed you back days ago. The printing of mesh setting messages and the subsequent slow down is most likely due to the machine being busy. Thanks for the help.

[GiudGiud]

oh sorry. Glad this is settled and thanks for letting me know