Build / test UFS WM with Intel oneAPI v2024.2.1 (LLVM)-based stacks

[rickgrubin-noaa]

Description

Build and test UFS WM against spack-stack@release/1.8.0 built with Intel oneAPI v2024.2.1 (LLVM) compilers.

Intel oneAPI v2024.2.1 is the last oneAPI release to provide two FORTRAN compilers: ifx (new) and ifort (old / primary compiler in previous releases).

Build and test against stacks built with ifx and ifort, respectively.

Solution

stack environments

hera: /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs

• ifx

  • ue-llvm-ifx-2024.2.1
  • gsi-llvm-ifx-2024.2.1

• ifort

  • ue-llvm-ifort-2024.2.1
  • gsi-llvm-ifort-2024.2.1

Will also require addon environments for each compiler for a known-to-work combination of MAPL / ESMF.

Once the MAPL / ESMF addon envs are ready, steps to load required modules, e.g. for the ifx compiler:

module purge
module use /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.8.0-llvm/envs/ue-llvm-ifx-2024.2.1/install/modulefiles/Core
module load stack-oneapi/2024.2.1
module load stack-intel-oneapi-mpi/2021.13
module load ufs-weather-model-env/1.0.0
module load geos-gcm-env/1.0.0

Alternatives

Currently N/A

Related to

See also: Upgrade to Intel LLVM Compilers #2025 and associated PR Support IntelLLVM compiler #2224

[rickgrubin-tomorrow]

ufs-weather-model @ develop and spack-stack @ release/1.8.0 built with oneAPI compilers (v2024.2.1); one stack built with ifx, and another stack built with ifort. These stacks include:

jasper          = 2.0.32
zlib-ng         = 2.1.6
libpng          = 1.6.37
hdf5            = 1.14.3
netcdf-c        = 4.9.2
netcdf-fortran  = 4.6.1
parallelio      = 2.6.2
esmf            = 8.6.1
fms             = 2024.02
bacio           = 2.4.1
crtm            = 2.4.0.1
g2              = 3.5.1
g2tmpl          = 1.13.0
ip              = 5.0.0
sp              = 2.5.0
w3emc           = 2.10.0
gftl-shared     = 1.9.0
mapl            = 2.46.3-esmf-8.6.1
scotch          = 7.0.4

Test control_c48 successfully compiles with both compilers in non-DEBUG mode.

Test control_c48 / ifort successfully completes.
Test control_c48 / ifx fails to run to completion.

See below for location of test runs.

Building a separate spack-stack based on release/1.8.0 and ifx@2024.2.1, reverting versions of various packages to meet what were used in Support LLVM compiler:

jasper          = 2.0.32
zlib-ng         = 1.2.13
libpng          = 1.6.37
hdf5            = 1.14.0
netcdf-c        = 4.9.2
netcdf-fortran  = 4.6.1
parallelio      = 2.5.10
esmf            = 8.6.0
fms             = 2024.01
bacio           = 2.4.1
crtm            = 2.4.0.1
g2              = 3.5.1
g2tmpl          = 1.13.0
ip              = 4.3.0
sp              = 2.5.0
w3emc           = 2.10.0
gftl-shared     = 1.6.1
mapl            = 2.40.3-esmf-8.6.0
scotch          = 7.0.4

fails in the same manner as the failure above; again, see below for location of test runs.

The common denominator is ifx @ 2024.2.1. It's known that the UFS WM, when modified to work with oneAPI compilers, was tested with an earlier version of these compilers; per Support LLVM compiler, it appears to be 2023.x

Confirmed, I think:

PR: Support LLVM compiler

file modulefiles/ufs_hera.intelllvm.lua

[...]
stack_intel_ver=os.getenv("stack_intel_ver") or "2021.5.0"
load(pathJoin("stack-intel", stack_intel_ver))

stack_impi_ver=os.getenv("stack_impi_ver") or "2021.5.1"
load(pathJoin("stack-intel-oneapi-mpi", stack_impi_ver))

load("gnu")
load("intel/2023.2.0")
[...]

shows the details; and at the command line on hera:

# module load intel/2023.2.0
# which ifx
/apps/oneapi/compiler/2023.2.0/linux/bin/ifx
# ifx --version
ifx (IFX) 2023.2.0 20230622

---

UFS WM Regression Testing results

ifort

• ifort@2024.2.1 (successful compile (non-DEBUG) and successful model run)
  • ESMF@8.6.1 and MAPL@2.46.3 and FMS@2024.02
  • /scratch2/NCEPDEV/stmp1/role.epic/ufswm-test/stmp2/role.epic/FV3_RT/rt_2455064

ifx

• ifx@2024.2.1 (successful compile (non-DEBUG) and failed model run)

  • ESMF@8.6.1 and MAPL@2.46.3 and FMS@2024.02
  • /scratch2/NCEPDEV/stmp1/role.epic/ufswm-test/stmp2/role.epic/FV3_RT/rt_2949966

• ifx@2024.2.1 (failed compile (DEBUG))

  • ESMF@8.6.1 and MAPL@2.46.3 and FMS@2024.02
  • /scratch2/NCEPDEV/stmp1/role.epic/ufswm-test/stmp2/role.epic/FV3_RT/rt_2377353

• ifx@2024.2.1 (successful compile (non-DEBUG) and failed model run)

  • ESMF@8.6.0 and MAPL@2.40.3 and FMS@2024.01
  • /scratch2/NCEPDEV/stmp1/role.epic/ufswm-test/stmp2/role.epic/FV3_RT/rt_1323858