Running into issue with update libmesh script on HPC

[TheGreatCid]

Check these boxes if you have followed the posting rules.

• Q&A General is the most appropriate section for my question
• I have consulted the posting Guidelines on the Discussions front page
• I have searched the Discussions forum and my question has not been asked before
• I have searched the MOOSE website and the documentation does not answer my question
• I have formatted my post following the posting guidelines (screenshots as a last resort, triple back quotes around pasted text)

Question

I am trying to run update_and_rebuild_libmesh.sh on an HPC and I am running into the following error:

I for src/utilities/include/timpi/timpi_config.h.tmp defines
config.status: executing depfiles commands
config.status: error: in `/ascldap/users/detorre/raccoon/moose/libmesh/build/contrib/timpi':
config.status: error: Something went wrong bootstrapping makefile fragments
    for automatic dependency tracking.  If GNU make was not used, consider
    re-running the configure script with MAKE="gmake" (or whatever is
    necessary).  You can also try re-running configure with the
    '--disable-dependency-tracking' option to at least be able to build
    the package (albeit without support for automatic dependency tracking).
See `config.log' for more details
configure: WARNING: cache variable ac_cv_env_timpi_CPPFLAGS_value contains a newline
configure: error: ../../../contrib/timpi/configure failed for contrib/timpi
Running make -j 15...
make: *** No targets specified and no makefile found.  Stop.

I have tried both of the suggestions from the error message to no avail.

This is the output of diagnostics.sh

##################################################################################################
Influential Environment Variables

CC=mpicc
CONDA_CHANNEL=https://conda.software.inl.gov/public
CONDA_EXE=/home/detorre/mambaforge3/bin/conda
CONDA_PYTHON_EXE=/home/detorre/mambaforge3/bin/python
CONDA_SHLVL=0
CURL_CA_BUNDLE=
CXX=mpicxx
F77=mpif77
F90=mpif90
FC=mpif90
http_proxy=http://user:pass@proxy.sandia.gov:80
https_proxy=http://user:pass@proxy.sandia.gov:80
LD_LIBRARY_PATH=/opt/toss/openmpi/4.1/gnu/lib:/usr/lib64/nvidia:/projects/netpub/gnu/11.3.0-toss3/lib64:/projects/netpub/gnu/11.3.0-toss3/lib::
MODULESHOME=/usr/share/lmod/lmod
MOOSE_JOBS=15
PATH=/opt/toss/openmpi/4.1/gnu/bin:/projects/cmake/toss4/3.25.2/bin:/projects/aue/hpc/builds/x86_64/rhel8/ba17d7f2/tooling/install/linux-rhel8-x86_64/gcc-10.3.0/python-3.9.16-a7hvvwm/bin:/projects/netpub/gnu/11.3.0-toss3/bin:/home/detorre/mambaforge3/condabin:/ascldap/users/detorre/mambaforge3/envs/gmsh2/bin:/usr/lpp/mmfs/bin:/opt/ibm/MCStore/bin:/opt/ibm/MCStore/scripts:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
REQUESTS_CA_BUNDLE=
SSL_CERT_FILE=

##################################################################################################
Compiler(s) (CC CXX FC F77 F90):

CC=/opt/toss/openmpi/4.1/gnu/bin/mpicc
CC -show:
gcc -I/opt/toss/openmpi/4.1/gnu/include -pthread -L/opt/toss/openmpi/4.1/gnu/lib -Wl,-rpath -Wl,/opt/toss/openmpi/4.1/gnu/lib -Wl,--enable-new-dtags -lmpi
CC version:	gcc (GCC) 11.3.0

CXX=/opt/toss/openmpi/4.1/gnu/bin/mpicxx
CXX -show:
g++ -I/opt/toss/openmpi/4.1/gnu/include -pthread -L/opt/toss/openmpi/4.1/gnu/lib -Wl,-rpath -Wl,/opt/toss/openmpi/4.1/gnu/lib -Wl,--enable-new-dtags -lmpi
CXX version:	g++ (GCC) 11.3.0

FC=/opt/toss/openmpi/4.1/gnu/bin/mpif90
FC -show:
gfortran -I/opt/toss/openmpi/4.1/gnu/include -pthread -I/opt/toss/openmpi/4.1/gnu/lib -L/opt/toss/openmpi/4.1/gnu/lib -Wl,-rpath -Wl,/opt/toss/openmpi/4.1/gnu/lib -Wl,--enable-new-dtags -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi
FC version:	GNU Fortran (GCC) 11.3.0

F77=/opt/toss/openmpi/4.1/gnu/bin/mpif77
F77 -show:
gfortran -I/opt/toss/openmpi/4.1/gnu/include -pthread -I/opt/toss/openmpi/4.1/gnu/lib -L/opt/toss/openmpi/4.1/gnu/lib -Wl,-rpath -Wl,/opt/toss/openmpi/4.1/gnu/lib -Wl,--enable-new-dtags -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi
F77 version:	GNU Fortran (GCC) 11.3.0

F90=/opt/toss/openmpi/4.1/gnu/bin/mpif90
F90 -show:
gfortran -I/opt/toss/openmpi/4.1/gnu/include -pthread -I/opt/toss/openmpi/4.1/gnu/lib -L/opt/toss/openmpi/4.1/gnu/lib -Wl,-rpath -Wl,/opt/toss/openmpi/4.1/gnu/lib -Wl,--enable-new-dtags -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi
F90 version:	GNU Fortran (GCC) 11.3.0

OK

##################################################################################################
Python Sanity Checks

Verify `/usr/bin/env python3 --version` (reporting as: Python 3.9.16),
matches versions for: `which python3 && which python`

OK

##################################################################################################
Python Modules (TestHarness, run-ability)

WARNING: python module: `yaml` not available

Warning Python packages may cause miscellaneous runtime issues.

Either install the above packages, or perhaps you have yet to activate your
moose environment: `conda activate moose`.

I see the error with yaml but that seems unrelated to this issue. Though I imagine I'll have to address that eventually.

Any insight into the issue would be appreciated. Thanks!

[GiudGiud]

Could you please attach the config.log file for more details?
It ll be in libmesh/

[TheGreatCid]

Sorry for the delay. I had some issues with the hpc that I needed to sort out.
Where do I insert skip-submodule-update? I put it into the configure libmesh file but it seems to not a be a recognized option

[GiudGiud]

It goes after update_and_rebuild_libmesh.sh, to prevent it from updating and removing your changes to libmesh

[TheGreatCid]

./update_and_rebuild_libmesh.sh --skip-submodule-update
Is this correct? The makefile is still being reset after running.

[GiudGiud]

the makefile must be generated automatically.

[TheGreatCid]

I did try a fresh install as well, but I am running into the same issue

[GiudGiud]

I think it might be time to get help from @roystgnr on this

[roystgnr]

Could we see the config.log from TIMPI? It'll be in contrib/timpi/config.log

Could you try just running ./bootstrap in the contrib/timpi directory, and let us know the output?

That newline is weird, but it's also a warning and I think it might be a red herring.

[roystgnr]

Well, hopefully it worked. Seeing both /usr/include/boost1.78/boost and /usr/include/boost on the same system is scary. Hopefully the latter is just a symbolic link to the former?

If so, then you should be good to go. I'll see if I can make such a symlink on my laptop to replicate the problem and fix our boost m4 script for that case.

If they're not just the same directory via links in the filesystem, but they are the same version or ABI-compatible versions, then you should also be good to go, at least until your package manager decides to upgrade one but not the other.

If they're ever ABI-incompatible versions ... something could break, in theory? Dependency A detects /usr/include/boost1.78 and creates a struct according to one definition, then passes a pointer it to dependency B which detects /usr/include/boost and looks for member offsets according to a different definition, and then you segfault if you're lucky or you get weird corruption if you're not. I've never had this happen with boost before, but I have seen it with other libraries, and it's even less fun to diagnose than this configure issue was. If that /usr/include/boost isn't just a symlink then I'd bring it up with your HPC guys; you pretty much never want to have multiple versions of software in system directories where they're likely to be autodetected.

[TheGreatCid]

It does appear that /usr/include/boost1.78/boost and /usr/include/boost are indeed different directories. However, I'm unsure how to check the compatibility of the versions.

I do see what you are saying. They both certainly should not exist.
Looks like this whole debacle came from an issue specific to my HPC rather than anything with MOOSE.
I really appreciate the help in narrowing this down!

[roystgnr]

There's a version.hpp you could check.

[TheGreatCid]

Sorry for the last response.
Looking at the headers, one appears to be 1.66 and the other 1.78. So definitely a conflict there

[roystgnr]

Oh, yikes. So if any other dependency puts a -I/usr/include in your compiler flags (which is really common) then that potential conflict can become an actual conflict.

You might try --with-boost=/usr and see if that brings in the /usr/include/boost, since it's much less likely that another dependency will be demanding /usr/include/boost1.78/boost. If that works, you're safe.

If that doesn't work? I'd keep going with --with-boost=/usr/include/boost1.78, but be very watchful for either weird compiler errors or weird crashes.