GFDL MOM6 model aborts with netcdf-fortran-4.6.0. Works with netcdf-fortran-4.5.3. (SUSPECT MOM6 ISSUE BUT OPENING AT REQUEST OF NETCDF SUPPORT)

[GeorgeVandenberghe-NOAA]

The GFDL MOM6 ocean model fails with a field name error when using netcdf-fortran-4.6.0. It runs normally with netcdf-fortran-4.5.3. I strongly suspect this is not a NetCDF issue but am submitting this issue at the request of NetCDF support. Will try to replicate this when I get a chance on gaea C5 and report the error message which has been lost on C5. The rest of the software is netcdf-c-4.7.4 (also fails with 4.9.1) and hdf5/1.12.2.

[GeorgeVandenberghe-NOAA]

The error message from MOM6
8416: FATAL from PE 0: NetCDF: Name contains illegal characters: netcdf_add_variable: file:MOM6_OUTPUT/ocean_geometry.ncvariable:lath

A second came from the ATM model after 24 hours in an ATM only run
0: FATAL from PE 0: NetCDF: Name contains illegal characters: netcdf_add_variable: file:RESTART/20200602.060000.fv_core.res.ncvariable:xaxis_1

I will report these to EMC.

[WardF]

Thank you!

[edwardhartnett]

What are the illegal names being attempted?

That is, what is the software trying to use for a variable name?

[GeorgeVandenberghe-NOAA]

From the issue I reported - The error message from MOM6 - 8416: FATAL from PE 0: NetCDF: Name contains illegal characters: netcdf_add_variable: file:MOM6_OUTPUT/ocean_geometry.ncvariable:*lath* - - A second came from the ATM model after 24 hours in an ATM only run - 0: FATAL from PE 0: NetCDF: Name contains illegal characters: netcdf_add_variable: file:RESTART/20200602.060000.fv_core.res.ncvariable *:xaxis_1* *I think these are valid names but a memory issue overwriting some table defining them. * - - I will report these to EMC. - "

…

On Tue, Mar 28, 2023 at 2:16 PM Edward Hartnett ***@***.***> wrote: What are the illegal names being attempted? That is, what is the software trying to use for a variable name? — Reply to this email directly, view it on GitHub <#395 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDS4FQAC2MDR2TPT522J5DW6MTHNANCNFSM6AAAAAAWJKYNJA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- George W Vandenberghe *Lynker Technologies at * NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 ***@***.*** 301-683-3769(work) 3017751547(cell)

[edwardhartnett]

Is a '*' valid in a variable name in netCDF?

[GeorgeVandenberghe-NOAA]

I don't know. I suspect not. If not it's definitely a model issue

…

On Wed, Mar 29, 2023 at 1:03 PM Edward Hartnett ***@***.***> wrote: Is a '*' valid in a variable name in netCDF? — Reply to this email directly, view it on GitHub <#395 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDS4FWLXVTTJE7EB7KHSP3W6QXKBANCNFSM6AAAAAAWJKYNJA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- George W Vandenberghe *Lynker Technologies at * NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 ***@***.*** 301-683-3769(work) 3017751547(cell)

[edwardhartnett]

According to this answer, asterisks should be allowed in names: https://www.unidata.ucar.edu/support/help/MailArchives/netcdf/msg10684.html

[DennisHeimbigner]

I believe that is correct.

[edwardhartnett]

Oh, duh, it's right there in front of us:

Name contains illegal characters:
   netcdf_add_variable: file:RESTART/20200602.060000.fv_core.res.ncvariable
   *:xaxis_1*

There's a "/" there. Is that part of the variable name? Because that it an illegal character for netCDF names, IIRC...

[GeorgeVandenberghe-NOAA]

I thought the file description valid and the actual variable was :xaxis_1* but I haven't looked at MOM6 source (which EMC doesn't control in any way).

…

On Wed, Mar 29, 2023 at 3:58 PM Edward Hartnett ***@***.***> wrote: Oh, duh, it's right there in front of us: Name contains illegal characters: netcdf_add_variable: file:RESTART/20200602.060000.fv_core.res.ncvariable *:xaxis_1* There's a "/" there. Is that part of the variable name? Because that it an illegal character for netCDF names, IIRC... — Reply to this email directly, view it on GitHub <#395 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDS4FUIRZIJQBR5DAZB4S3W6RL3XANCNFSM6AAAAAAWJKYNJA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- George W Vandenberghe *Lynker Technologies at * NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 ***@***.*** 301-683-3769(work) 3017751547(cell)

[GeorgeVandenberghe-NOAA]

I can start looking at this after I get the benchmark off my desk and out the door.

[edwardhartnett]

Maybe the variable is named "file:RESTART/20200602.060000.fv_core.res.ncvariable :xaxis_1"? Can you ask the programmers?

Surely at some point this must have worked, yet as far as I know there has been no change in illegal name characters since @DennisHeimbigner changed everything to allow unicode, and that was like 15 years ago. Dennis have there been changes to what's legal in a name since then?

[GeorgeVandenberghe-NOAA]

I have asked FV3 ATM and MOM6 people to look into it at GFDL. Meanwhile since I believe this is a model issue UCAR and NETCDF development shouldn't lose any sleep over it.

…

On Wed, Mar 29, 2023 at 4:09 PM Edward Hartnett ***@***.***> wrote: Maybe the variable is named "file:RESTART/20200602.060000.fv_core.res.ncvariable *:xaxis_1*"? Can you ask the programmers? Surely at some point this must have worked, yet as far as I know there has been no change in illegal name characters since @DennisHeimbigner <https://github.com/DennisHeimbigner> changed everything to allow unicode, and that was like 15 years ago. Dennis have there been changes to what's legal in a name since then? — Reply to this email directly, view it on GitHub <#395 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDS4FUAPUQI7L6VRBYEHG3W6RNCBANCNFSM6AAAAAAWJKYNJA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- George W Vandenberghe *Lynker Technologies at * NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 ***@***.*** 301-683-3769(work) 3017751547(cell)

[DennisHeimbigner]

There have been no changes as far as I know. But it is still the case that the '/' character is not legal.

[bensonr]

GFDL models are running just fine with the cray-netcdf modules on the Gaea C5 partition. The versions appear to be C-4.9.0 and Fortran-4.5.3 (according to nc- and nf-config). I have asked an FMS engineer to run unit tests on an internal system at GFDL with C-4.9.0 and Fortran-4.6.0.

[GeorgeVandenberghe-NOAA]

I have only tested netcdf fortran 4.5.3 and netcdf-fortran 4.6.0. I get solid failures with netcdf-fortran 4.6.0 using both netcdf-c/4.7.4 and netcdf-c/4.9.1 with hdf5/1.10.6 and with netcdf-c/4.7.4, netcf-fortran/4.6.0 using hdf5/1.14.0. netcdf-c/4.9.1 does not build with hdf5/1.14.0. The common denominator seems to be netcdf-fortran/4.6.0 I have not tried netcdf-c/4.9.2 yet

…

On Wed, Mar 29, 2023 at 6:50 PM Rusty Benson ***@***.***> wrote: GFDL models are running just fine with the cray-netcdf modules on the Gaea C5 partition. The versions appear to be C-4.9.0 and Fortran-4.5.3 (according to nc- and nf-config). I have asked an FMS engineer to run unit tests on an internal system at GFDL with C-4.9.0 and Fortran-4.6.0. — Reply to this email directly, view it on GitHub <#395 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDS4FUERVV4QAKEAZBTP4TW6R77NANCNFSM6AAAAAAWJKYNJA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- George W Vandenberghe *Lynker Technologies at * NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 ***@***.*** 301-683-3769(work) 3017751547(cell)

[uramirez8707]

Maybe the variable is named "file:RESTART/20200602.060000.fv_core.res.ncvariable :xaxis_1"? Can you ask the programmers?

The filename is "RESTART/20200602.060000.fv_core.res.nc"
and the variable name is "xaxis_1"

Something like this should be the traceback
https://github.com/NOAA-GFDL/GFDL_atmos_cubed_sphere/blob/cfae631cb9c07b40863441c157a39787626ba1fc/tools/fv_io.F90#L131
https://github.com/NOAA-GFDL/FMS/blob/7188e3a2e634376da74c3e4247bc9b487ef52700/fms2_io/fms_netcdf_domain_io.F90#L548
https://github.com/NOAA-GFDL/FMS/blob/7188e3a2e634376da74c3e4247bc9b487ef52700/fms2_io/netcdf_io.F90#L956-L964

[GeorgeVandenberghe-NOAA]

Forwarding to GFDL . I will have to rebuild FMS to get that traceback (FMS/2022.04 is the version used) The failure is in the coupled UFS:MOM6 execution but I saw an ATM failure also so it isn't just MOM6 I am working other critical issues at NCEP

…

On Thu, Mar 30, 2023 at 10:09 AM uramirez8707 ***@***.***> wrote: Maybe the variable is named "file:RESTART/20200602.060000.fv_core.res.ncvariable *:xaxis_1*"? Can you ask the programmers? The filename is "RESTART/20200602.060000.fv_core.res.nc" and the variable name is "xaxis_1" Something like this should be the traceback https://github.com/NOAA-GFDL/GFDL_atmos_cubed_sphere/blob/cfae631cb9c07b40863441c157a39787626ba1fc/tools/fv_io.F90#L131 https://github.com/NOAA-GFDL/FMS/blob/7188e3a2e634376da74c3e4247bc9b487ef52700/fms2_io/fms_netcdf_domain_io.F90#L548 https://github.com/NOAA-GFDL/FMS/blob/7188e3a2e634376da74c3e4247bc9b487ef52700/fms2_io/netcdf_io.F90#L956-L964 — Reply to this email directly, view it on GitHub <#395 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDS4FUWWZ4H2OGLPHTDXLLW6WHYDANCNFSM6AAAAAAWJKYNJA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- George W Vandenberghe *Lynker Technologies at * NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 ***@***.*** 301-683-3769(work) 3017751547(cell)

[bensonr]

@GeorgeVandenberghe-NOAA - I'll ask again, what compiler (assuming Intel - ifx? ifort?) and version (19, OneAPI 2022, ??) are you using?

[GeorgeVandenberghe-NOAA]

My omission, sorry. Intel ifort, icc icpc.. whatever intel-classic loads on gaea c5. * No oneapi!! that's a nopitty nope for many reasons!* Hdf5/1.14.0, netcdf/4.7.4 netcdf-fortran/4.6.0, also replicated with hdf5/1.10.6 netcdf/4.9.1, netcdf-fortran/4.6.0

…

On Thu, Mar 30, 2023 at 7:09 PM Rusty Benson ***@***.***> wrote: @GeorgeVandenberghe-NOAA <https://github.com/GeorgeVandenberghe-NOAA> - I'll ask again, what compiler (assuming Intel - ifx? ifort?) and version (19, OneAPI 2022, ??) are you using? — Reply to this email directly, view it on GitHub <#395 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDS4FW6WITTYQZJPYSR6A3W6XK4XANCNFSM6AAAAAAWJKYNJA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- George W Vandenberghe *Lynker Technologies at * NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 ***@***.*** 301-683-3769(work) 3017751547(cell)

[GeorgeVandenberghe-NOAA]

The compilation module loaded is intel-classic/2022.0.2 on gaea C5 On Thu, Mar 30, 2023 at 7:18 PM George Vandenberghe - NOAA Affiliate < ***@***.***> wrote:

…

My omission, sorry. Intel ifort, icc icpc.. whatever intel-classic loads on gaea c5. * No oneapi!! that's a nopitty nope for many reasons!* Hdf5/1.14.0, netcdf/4.7.4 netcdf-fortran/4.6.0, also replicated with hdf5/1.10.6 netcdf/4.9.1, netcdf-fortran/4.6.0 On Thu, Mar 30, 2023 at 7:09 PM Rusty Benson ***@***.***> wrote: > @GeorgeVandenberghe-NOAA <https://github.com/GeorgeVandenberghe-NOAA> - > I'll ask again, what compiler (assuming Intel - ifx? ifort?) and version > (19, OneAPI 2022, ??) are you using? > > — > Reply to this email directly, view it on GitHub > <#395 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ANDS4FW6WITTYQZJPYSR6A3W6XK4XANCNFSM6AAAAAAWJKYNJA> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > -- George W Vandenberghe *Lynker Technologies at * NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 ***@***.*** 301-683-3769(work) 3017751547(cell)

-- George W Vandenberghe *Lynker Technologies at * NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 ***@***.*** 301-683-3769(work) 3017751547(cell)

[GeorgeVandenberghe-NOAA]

Also if I rebuild FMS with -traceback -g, where is that specified in the source tree directives files, i.e CMakeLists.txt and whatnot?

…

On Thu, Mar 30, 2023 at 7:09 PM Rusty Benson ***@***.***> wrote: @GeorgeVandenberghe-NOAA <https://github.com/GeorgeVandenberghe-NOAA> - I'll ask again, what compiler (assuming Intel - ifx? ifort?) and version (19, OneAPI 2022, ??) are you using? — Reply to this email directly, view it on GitHub <#395 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANDS4FW6WITTYQZJPYSR6A3W6XK4XANCNFSM6AAAAAAWJKYNJA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- George W Vandenberghe *Lynker Technologies at * NOAA/NWS/NCEP/EMC 5830 University Research Ct., Rm. 2141 College Park, MD 20740 ***@***.*** 301-683-3769(work) 3017751547(cell)