Floating Point Exception in Debug after Update

[Flolaffel]

Check these boxes if you have followed the posting rules.

• Q&A General is the most appropriate section for my question
• I have consulted the posting Guidelines on the Discussions front page
• I have searched the Discussions forum and my question has not been asked before
• I have searched the MOOSE website and the documentation does not answer my question
• I have formatted my post following the posting guidelines (screenshots as a last resort, triple back quotes around pasted text)

Question

Hello,

I updated MOOSE and Conda today and now I can't use the debug build anymore. I followed these steps from the New Users page to update.

cd ~/projects/moose
git fetch origin
git rebase origin/master

conda activate base
conda env remove -n moose
conda create -n moose moose-dev=2024.07.19
conda activate moose

cd ~/projects/YourAppName
make clobberall
make 

The input doesn't matter. Top opt build is working fine. But as soon as I try to debug, this happens:

Time Step 0, time = 0

Time Step 1, time = 0.05, dt = 0.05

Floating point exception signaled (invalid floating point operation)!
libMesh terminating:

To track this down, compile in debug mode, then in gdb do:
  break libmesh_handleFPE
  run ...
  bt
Stack frames: 35
0: libMesh::print_trace(std::ostream&)
1: libMesh::MacroFunctions::report_error(char const*, int, char const*, char const*, std::ostream&)
2: /home/.local/ruef/miniforge/envs/moose/libmesh/lib/libmesh_dbg.so.0(+0x222e6e3) [0x7fb88922e6e3]
3: /lib/x86_64-linux-gnu/libc.so.6(+0x3bf90) [0x7fb87f6ccf90]
4: fedisableexcept
5: libMesh::enableFPE(bool)
6: /home/.local/ruef/projects/moose/framework/libmoose-dbg.so.0(+0x20421ec) [0x7fb88ee421ec]
7: NonlinearSystemBase::computeResidualTags(std::set<unsigned int, std::less<unsigned int>, std::allocator<unsigned int> > const&)
8: FEProblemBase::computeResidualTags(std::set<unsigned int, std::less<unsigned int>, std::allocator<unsigned int> > const&)
9: FEProblemBase::computeResidualInternal(libMesh::NumericVector<double> const&, libMesh::NumericVector<double>&, std::set<unsigned int, std::less<unsigned int>, std::allocator<unsigned int> > const&)
10: FEProblemBase::computeResidual(libMesh::NumericVector<double> const&, libMesh::NumericVector<double>&, unsigned int)
11: FEProblemBase::computeResidualSys(libMesh::NonlinearImplicitSystem&, libMesh::NumericVector<double> const&, libMesh::NumericVector<double>&)
12: ComputeResidualFunctor::residual(libMesh::NumericVector<double> const&, libMesh::NumericVector<double>&, libMesh::NonlinearImplicitSystem&)
13: libmesh_petsc_snes_residual
14: SNESComputeFunction
15: /home/.local/ruef/miniforge/envs/moose/petsc/lib/libpetsc.so.3.20(+0xf6937f) [0x7fb88316937f]
16: SNESSolve
17: libMesh::PetscNonlinearSolver<double>::solve(libMesh::SparseMatrix<double>&, libMesh::NumericVector<double>&, libMesh::NumericVector<double>&, double, unsigned int)
18: libMesh::NonlinearImplicitSystem::solve()
19: TimeIntegrator::solve()
20: NonlinearSystem::solve()
21: FEProblemBase::solve(unsigned int)
22: FEProblemSolve::solve()
23: FixedPointSolve::solveStep(double&, double&, std::set<unsigned long, std::less<unsigned long>, std::allocator<unsigned long> > const&)
24: FixedPointSolve::solve()
25: TimeStepper::step()
26: Transient::takeStep(double)
27: Transient::execute()
28: MooseApp::executeExecutioner()
29: MooseApp::run()
30: ./simp-dbg(+0x3e80) [0x5582b63c4e80]
31: main
32: /lib/x86_64-linux-gnu/libc.so.6(+0x2718a) [0x7fb87f6b818a]
33: __libc_start_main
34: ./simp-dbg(+0x3079) [0x5582b63c4079]
[0] ../src/base/libmesh.C, line 139, compiled Jul 19 2024 at 13:27:54

--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------

What should I do?

[milljm]

Looks like you installed our OpenMPI variant packages, which is a bit under-tested. Can you do the following and see if this solves your issue?

conda deactivate
conda activate base
conda env remove -n moose
conda create -n moose moose-dev=2024.07.19 mpich
conda activate moose
cd ~/projects/YourAppName
make clobberall
make -j 6

We're in the process of updating our instructions right now for this very reason, ref: #28195

[pbehne]

Alright, I'll try to replicate on one of our linux systems. If that doesn't work, I have some Ubuntu and Windows systems at home I can try.

[pbehne]

I was able to emulate an Ubuntu 22.04 environment and reproduce the failure. I'll dig into it now.

[GiudGiud]

@pbehne i just remembered this isn't the first time we hit this (this being the difference between Mac and Linux, not the specific error here) btw.
I seem to recall having those FP issues on Linux and macOS was not hitting them

It may be worth finding what happened then, something like two years ago

[pbehne]

I haven't gotten anywhere with finding the cause of the FPE, but the backtraces I have found have piqued @roystgnr's interest enough that he is now trying to replicate it.

[GiudGiud]

opened #28350 so we do not lose track of it