libmkl_intel_thread.so: undefined reference to many kmpc thread on RHEL 9

[SomePersonSomeWhereInTheWorld]

Below are the modules loaded on RHEL 9 cluster. Is there some incompatibility with Intel oneAPI 2024 mkl? I've posted on the OneAPI discussion board no replies. I've tried multiple versions of the MUMPS source code, same errors.

scotch/7.0.4, parmetis/4.0.3, scalapack/2.2.0, oneapi/mkl/2024.0, oneapi/mpi/2021.11

Warning: Type mismatch between actual argument at (1) and actual 
argument at (2) (REAL(8)/INTEGER(4)).
ar vr ../lib/libzmumps.a zana_aux.o zana_aux_par.o zana_lr.o 
zfac_asm_master_ELT_m.o zfac_asm_master_m.o zfac_front_aux.o 
zfac_front_LU_type1.o zfac_front_LU_type2.o zfac_front_LDLT_type1.o 
zfac_front_LDLT_type2.o zfac_front_type2_aux.o zfac_sispointers_m.o 
zfac_lr.o zfac_mem_dynamic.o zfac_omp_m.o zfac_par_m.o 
zfac_sol_l0omp_m.o zlr_core.o zlr_stats.o zlr_type.o 
zmumps_comm_buffer.o zmumps_config_file.o zmumps_load.o 
zmumps_lr_data_m.o zmumps_ooc_buffer.o zmumps_ooc.o zmumps_sol_es.o 
zmumps_save_restore.o zmumps_save_restore_files.o zmumps_struc_def.o 
zomp_tps_m.o zsol_lr.o zsol_omp_m.o zstatic_ptr_m.o zmumps_mpi3_mod.o 
zini_driver.o zana_driver.o zfac_driver.o zsol_driver.o zsol_distrhs.o 
zsol_distsol.o zend_driver.o zana_aux_ELT.o zana_dist_m.o 
zana_LDLT_preprocess.o zana_reordertree.o zarrowheads.o zbcast_int.o 
zfac_asm_ELT.o zfac_asm.o zfac_b.o zfac_diag.o 
zfac_distrib_distentry.o zfac_dist_arrowheads_omp.o zfac_distrib_ELT.o 
zfac_lastrtnelind.o zfac_mem_alloc_cb.o zfac_mem_compress_cb.o 
zfac_mem_free_block_cb.o zfac_mem_stack_aux.o zfac_mem_stack.o 
zfac_process_band.o zfac_process_blfac_slave.o 
zfac_process_blocfacto_LDLT.o zfac_process_blocfacto.o 
zfac_process_bf.o zfac_process_end_facto_slave.o 
zfac_process_contrib_type1.o zfac_process_contrib_type2.o 
zfac_process_contrib_type3.o zfac_process_maprow.o 
zfac_process_master2.o zfac_process_message.o 
zfac_process_root2slave.o zfac_process_root2son.o 
zfac_process_rtnelind.o zfac_root_parallel.o zfac_scalings.o 
zfac_determinant.o zfac_scalings_simScaleAbs.o 
zfac_scalings_simScale_util.o zfac_sol_pool.o zfac_type3_symmetrize.o 
zini_defaults.o
ar: creating ../lib/libzmumps.a
a - zana_aux.o
a - zana_aux_par.o
a - zana_lr.o
a - zfac_asm_master_ELT_m.o
a - zfac_asm_master_m.o
a - zfac_front_aux.o
a - zfac_front_LU_type1.o
a - zfac_front_LU_type2.o
a - zfac_front_LDLT_type1.o
a - zfac_front_LDLT_type2.o
a - zfac_front_type2_aux.o
a - zfac_sispointers_m.o
a - zfac_lr.o
a - zfac_mem_dynamic.o
a - zfac_omp_m.o
a - zfac_par_m.o
a - zfac_sol_l0omp_m.o
a - zlr_core.o
a - zlr_stats.o
a - zlr_type.o
a - zmumps_comm_buffer.o
a - zmumps_config_file.o
a - zmumps_load.o
a - zmumps_lr_data_m.o
a - zmumps_ooc_buffer.o
a - zmumps_ooc.o
a - zmumps_sol_es.o
a - zmumps_save_restore.o
a - zmumps_save_restore_files.o
a - zmumps_struc_def.o
a - zomp_tps_m.o
a - zsol_lr.o
a - zsol_omp_m.o
a - zstatic_ptr_m.o
a - zmumps_mpi3_mod.o
a - zini_driver.o
a - zana_driver.o
a - zfac_driver.o
a - zsol_driver.o
a - zsol_distrhs.o
a - zsol_distsol.o
a - zend_driver.o
a - zana_aux_ELT.o
a - zana_dist_m.o
a - zana_LDLT_preprocess.o
a - zana_reordertree.o
a - zarrowheads.o
a - zbcast_int.o
a - zfac_asm_ELT.o
a - zfac_asm.o
a - zfac_b.o
a - zfac_diag.o
a - zfac_distrib_distentry.o
a - zfac_dist_arrowheads_omp.o
a - zfac_distrib_ELT.o
a - zfac_lastrtnelind.o
a - zfac_mem_alloc_cb.o
a - zfac_mem_compress_cb.o
a - zfac_mem_free_block_cb.o
a - zfac_mem_stack_aux.o
a - zfac_mem_stack.o
a - zfac_process_band.o
a - zfac_process_blfac_slave.o
a - zfac_process_blocfacto_LDLT.o
a - zfac_process_blocfacto.o
a - zfac_process_bf.o
a - zfac_process_end_facto_slave.o
a - zfac_process_contrib_type1.o
a - zfac_process_contrib_type2.o
a - zfac_process_contrib_type3.o
a - zfac_process_maprow.o
a - zfac_process_master2.o
a - zfac_process_message.o
a - zfac_process_root2slave.o
a - zfac_process_root2son.o
a - zfac_process_rtnelind.o
a - zfac_root_parallel.o
a - zfac_scalings.o
a - zfac_determinant.o
a - zfac_scalings_simScaleAbs.o
a - zfac_scalings_simScale_util.o
a - zfac_sol_pool.o
a - zfac_type3_symmetrize.o
a - zini_defaults.o
ranlib ../lib/libzmumps.a
make[2]: Leaving directory '/path/to/rk3199/MUMPS_5.7.3/src'
make[1]: Leaving directory '/path/to/rk3199/MUMPS_5.7.3/src'
cd examples; make z
make[1]: Entering directory '/path/to/rk3199/MUMPS_5.7.3/examples'
mpif90 -O -std=legacy -DBLR_MT -fopenmp -fallow-argument-mismatch  -I. 
-I../include -I../src -I../libseq -c zsimpletest.F -o zsimpletest.o
mpif90 -o zsimpletest -O  -fopenmp -fallow-argument-mismatch 
zsimpletest.o  -L../lib -lzmumps -lmumps_common 
-L/insomnia001/shared/apps/metis/5.1.0/lib/ -lmetis -L../PORD/lib/ 
-lpord -L/insomnia001/shared/apps/scotch/7.0.4/lib -lesmumps -lscotch 
-lscotcherr -L/insomnia001/shared/apps/oneapi/mkl/2024.0 
-lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -L../libseq -lmpiseq 
 -L/usr/lib64 -lblas -lpthread
/usr/bin/ld: zsimpletest.o: in function `MAIN__':
zsimpletest.F:(.text+0x4f): undefined reference to `zmumps_'
/usr/bin/ld: zsimpletest.F:(.text+0x5f4): undefined reference to `zmumps_'
/usr/bin/ld: zsimpletest.F:(.text+0x628): undefined reference to `zmumps_'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_critical_with_hint'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_end_critical'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_for_static_init_8'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_atomic_float8_add'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_dispatch_next_4'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_master'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_for_static_fini'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_dispatch_init_4u'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_end_reduce'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_flush'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_dispatch_next_8u'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_for_static_init_8u'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_for_static_init_4'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_fork_call'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_dispatch_next_8'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_push_num_threads'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_ordered'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_omp_task_alloc'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_dispatch_init_8u'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_barrier'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_atomic_cmplx8_add'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_omp_task'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_global_thread_num'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_dispatch_init_4'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_critical'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_end_serialized_parallel'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_atomic_fixed4_add'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_omp_task_with_deps'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_reduce_nowait'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_atomic_float4_max'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_end_master'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_single'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_dispatch_fini_8'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_for_static_init_4u'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_dispatch_init_8'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_atomic_fixed8_add'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_atomic_cmplx4_add'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_ok_to_fork'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_end_single'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_dispatch_next_4u'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_end_ordered'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_end_reduce_nowait'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_serialized_parallel'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_atomic_float4_add'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_atomic_float8_max'
/usr/bin/ld: 
/path/to/apps/oneapi/mkl/2024.0/lib/libmkl_intel_thread.so: undefined 
reference to `__kmpc_reduce'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:41: zsimpletest] Error 1
make[1]: Leaving directory '/path/to/me/MUMPS_5.7.3/examples'
make: *** [Makefile:32: z] Error 2

[scivision]

what commands are you using? It looks like

cmake -Bbuild -DBUILD_SINGLE=yes -DBUILD_DOUBLE=yes -DBUILD_COMPLEX=yes -DBUILD_COMPLEX16=yes

cmake --build build

I tried this on RHEL 9 with oneAPI IntelLLVM 2024.2.1. It may be another option that you're using that I didn't try.