Memory error: Falcon Moose

[pranayasai]

Hello Everyone,

I am trying to run a big mesh (4Million cells) simulation using Falcon/Kerstel (MOOSE) on Sawtooth (at INL HPC) but the simulation keeps crashing.
Sometimes it crashes without finishing any iterations and sometimes it runs one or two iterations and then crashes.
I have tried running them on the compute node and also using PBS script but the error still persists.

The errors and number of processors used are listed below.

I have tried running the simulation in multiple combinations for the number of processors requested/used:
7200 requested: 3600 used (did not work, crashes with Error 1 (see below))
1000 requested: 500 used (did not work, crashes with Error 1)
1000 requested: 100 used (did not work, crashes with Error 1)
500 requested: 100 used (did not work crashes with Error 1)
500 requested: 75 used (works up to one iteration and then crashes with Error 2(see below))
144 requested: 50 used (works up to one iteration and then crashes with Error 3(see below))

Error 1:
[asaipran][~/Desktop/Forge/DFN/full2]> mpirun -np 150 ~/projects/kestrel/kestrel-opt -i fullDFN_PA_tran_PT.i

Building mesh .......................... [ 27.39 s]
Caching mesh information .... [ 5.20 s]
Initializing equation system ........... [ 12.94 s]
Caching mesh information .... [ 5.62 s]
Copying variables from Exodus ... [ 4.82 s]

Running App: main

Framework Information:
MOOSE Version: git commit e3fb9a7 on 2020-08-10
LibMesh Version: 8d10d4aff7dc56cb100ab27450120fd522449f40
PETSc Version: 3.12.5
SLEPc Version: 3.12.1
Current Time: Wed Feb 3 09:37:41 2021
Executable Timestamp: Thu Sep 24 16:05:15 2020

Parallelism:
Num Processors: 150
Num Threads: 1

Mesh:
Parallel Type: replicated
Mesh Dimension: 3
Spatial Dimension: 3
Nodes:
Total: 2050401
Local: 15289
Elems:
Total: 2000000
Local: 13315
Num Subdomains: 1
Num Partitions: 150
Partitioner: metis

Nonlinear System:
Num DOFs: 4100802
Num Local DOFs: 30578
Variables: { "pressure" "temperature" }
Finite Element Types: "LAGRANGE"
Approximation Orders: "FIRST"

Auxiliary System:
Num DOFs: 18000000
Num Local DOFs: 119835
Variables: { "vel_x" "vel_y" "vel_z" "fluid_density" "viscosity" "porosity" "perm_x"
"perm_y" "perm_z" }
Finite Element Types: "MONOMIAL"
Approximation Orders: "CONSTANT"

Execution Information:
Executioner: Transient
TimeStepper: SolutionTimeAdaptiveDT
Solver Mode: NEWTON
PETSc Preconditioner: asm
MOOSE Preconditioner: SMP

LEGACY MODES ENABLED:
This application uses the legacy material output option: material properties are output only on TIMESTEP_END, not INITIAL. To remove this message, set 'use_legacy_material_output' to false in this application. If there are gold output files that contain material property output for which output occurs on INITIAL, then these will generate diffs due to zero values being stored, and these tests should be re-golded.

Caching mesh information .... [ 5.33 s]
[cli_131]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 131
[asaipran][~/Desktop/Forge/DFN/full2]>

Error 2:

Computing initial residual ..... [ 6.09 s]
0 Nonlinear |R| = 1.826232e+09

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 125496 RUNNING AT r6i2n19.ib0.sawtooth.inl.gov
= EXIT CODE: 9
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES

[proxy:0:1@r6i2n22] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:911): assert (!closed) failed
[proxy:0:1@r6i2n22] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
[proxy:0:1@r6i2n22] main (pm/pmiserv/pmip.c:202): demux engine error waiting for event
[proxy:0:2@r6i2n20] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:911): assert (!closed) failed
[proxy:0:2@r6i2n20] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status
[proxy:0:2@r6i2n20] main (pm/pmiserv/pmip.c:202): demux engine error waiting for event
[mpiexec@r6i2n19] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:75): one of the processes terminated badly; aborting
[mpiexec@r6i2n19] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion
[mpiexec@r6i2n19] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion
[mpiexec@r6i2n19] main (ui/mpich/mpiexec.c:340): process manager error waiting for completion

Error 3:

Computing initial residual ...... [ 7.05 s]
0 Nonlinear |R| = 6.311744e-01
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Petsc has generated inconsistent data
[0]PETSC ERROR: is not a subset of lis
[0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.12.5, Mar, 29, 2020
[0]PETSC ERROR: /home/asaipran/projects/kestrel/kestrel-opt on a named r2i3n26 by asaipran Tue Feb 2 22:58:16 2021
[0]PETSC ERROR: Configure options --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-x=0 --with-mpi=1 --with-ssl=0 --with-openmp=1 --with-debugging=0 --with-64-bit-indices=1 --with-cxx-dialect=C++11 --with-shared-libraries=1 --with-fortran-bindings=0 --with-cxxlib-autodetect=0 --download-hypre=1 --download-metis=1 --download-slepc=1 --download-ptscotch=1 --download-parmetis=1 --download-scalapack=1 --download-fblaslapack=1 --download-superlu_dist=1 AR=ar CC=mpicc CXX=mpicxx FC=mpifort FC=mpifort FC=mpifort --prefix=/apps/moose/stack/petsc-3.12.5_mvapich2-2.3.3_gcc-9.2.0
[0]PETSC ERROR: #1 PCSetUp_ASM() line 378 in /tmp/tmp.yuC9vViIq5/petsc-3.12.5/src/ksp/pc/impls/asm/asm.c
[0]PETSC ERROR: #2 PCSetUp() line 894 in /tmp/tmp.yuC9vViIq5/petsc-3.12.5/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #3 KSPSetUp() line 376 in /tmp/tmp.yuC9vViIq5/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #4 KSPSolve() line 703 in /tmp/tmp.yuC9vViIq5/petsc-3.12.5/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #5 SNESSolve_NEWTONLS() line 225 in /tmp/tmp.yuC9vViIq5/petsc-3.12.5/src/snes/impls/ls/ls.c
[0]PETSC ERROR: #6 SNESSolve() line 4482 in /tmp/tmp.yuC9vViIq5/petsc-3.12.5/src/snes/interface/snes.c
[cli_0]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0

A similar input deck runs perfectly fine with a smaller mesh size.
I am not entirely sure where does the problem lie.
Can you please provide me any guidance to fix or to be able to run this simulation?

Thank you
Pranay

[GiudGiud]

Hi Pranay

You should request a number of nodes directly, and specify the maximum amount of memory available per node.
Can you paste your PBS script here? The part where you are requesting ressources especially.

Guillaume

[pranayasai]

Hello Guillaume,

I believe the available memory per node is around 4GB

Here is the PBS script.

#!/bin/bash
#PBS -N full5
#PBS -l select=5:ncpus=48:mpiprocs=48
#PBS -l walltime=48:00:00
#PBS -j oe

module load use.moose PETSc

echo
echo PBS_NODEFILE is $PBS_NODEFILE
echo PBS_QUEUE is $PBS_QUEUE
echo
hostname
NPROCS=wc -l < $PBS_NODEFILE
echo
echo NPROCS: $NPROCS
echo

cd /home/asaipran/Desktop/Forge/DFN/full5/

mpirun -np 150 ~/projects/kestrel/kestrel-opt -i fullDFN_PA_tran_PT.i

-Pranay

[GiudGiud]

Falcon nodes should have closer to 120 GB on-node ( so close to 4GB per core ). I dont know the exact number.
And they have 2 x 18 cores per node, with no hyper-threading I think.
We don't recommend using more MPI ranks than CPUs on Falcon.
So to run 150 MPI processes, I would recommend asking for 5 nodes, and using 30 cores on each node.

So #PBS -l select=5:ncpus=180:mpiprocs=150:mem=115gb
180 (=5 x 36) cores is to book the nodes entirely. You can also indicate you want exclusivity with another flag (saw it in discussions earlier on, dont use it myself) or by requesting all the memory.

Now an issue that may come up is that the MPI processes are not placed equitably on each node. To help with that, you have to pass a hostfile to mpirun / mpiexec to tell it which hosts to spawn processes on.

If your job is too memory intensive, you should first update to the latest moose since there was a memory reducing PR lately, and you can try booking more and more nodes to spread your problem over.

[pranayasai]

I just checked, Sawtooth has 192GB on the node with 48 processors. No hyper-threading.

[GiudGiud]

oh my bad, sorry I thought you meant Falcon the cluster! Ok Same advice, just adapt the numbers.