Add option to use netcdf global_berror files

[RussTreadon-NOAA]

Description
This PR adds the option for gsi.x to read netcdf format global_berror files.

Resolves #808

Depends on

• GSI-fix #25
• g-w issue #3128

Type of change

• New feature (non-breaking change which adds functionality)

How Has This Been Tested?

ctests run on WCOSS2 (Dogwood) with Passed result for all tests.

NetCDF format global_berror files tested for various resolutions on Dogwood. Runs using netcdf and binary global_berror files yield bitwise identical analysis increment files.

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• New and existing tests pass with my changes

[RussTreadon-NOAA]

This PR will changed to Ready for review once

• GSI-fix PR #25 is merged into develop
• g-w issue #3128 is closed
• the fix submodule hash in RussTreadon-NOAA:feature/netcdf_berror is updated
• GSI_BINARY_SOURCE_DIR is updated to point at the new directory for staged non-ASCII GSI fix files

[RussTreadon-NOAA]

Install RussTreadon-NOAA:feature/netcdf_berror at 0e07d71 and develop at 9ada88e on Dogwood, Hera, Hercules, and Orion. Run ctests with the following results

Dogwood (WCOSS2)

Test project /lfs/h2/emc/da/noscrub/russ.treadon/git/gsi/berror/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_rdasens
    Start 4: hafs_4denvar_glbens
    Start 5: hafs_3denvar_hybens
    Start 6: global_enkf
1/6 Test #3: rrfs_3denvar_rdasens .............   Passed  1207.91 sec
2/6 Test #6: global_enkf ......................   Passed  1331.62 sec
3/6 Test #5: hafs_3denvar_hybens ..............   Passed  1633.44 sec
4/6 Test #4: hafs_4denvar_glbens ..............   Passed  1813.80 sec
5/6 Test #1: global_4denvar ...................   Passed  2523.63 sec
6/6 Test #2: rtma .............................   Passed  3010.41 sec

100% tests passed, 0 tests failed out of 6

Total Test time (real) = 3010.53 sec

Orion

Test project /work2/noaa/da/rtreadon/git/gsi/pr810/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_rdasens
    Start 4: hafs_4denvar_glbens
    Start 5: hafs_3denvar_hybens
    Start 6: global_enkf
1/6 Test #3: rrfs_3denvar_rdasens .............   Passed  967.15 sec
2/6 Test #6: global_enkf ......................   Passed  1207.74 sec
3/6 Test #2: rtma .............................   Passed  1867.69 sec
4/6 Test #5: hafs_3denvar_hybens ..............   Passed  3085.11 sec
5/6 Test #4: hafs_4denvar_glbens ..............   Passed  3267.06 sec
6/6 Test #1: global_4denvar ...................***Failed  4202.53 sec

83% tests passed, 1 tests failed out of 6

Total Test time (real) = 4202.56 sec

The global_4denvar test failed due to

The runtime for global_4denvar_hiproc_updat is 881.487325 seconds.  This has exceeded maximum allowable threshold time of 815.596658 seconds, resulting in Failure of timethresh2 the regression test.

gsi.x wall times are known to be highly variable on Orion, especially when running in the /work/noaa/stmp fileset. This is not a fatal fail.

Hera

Test project /scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/pr810/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_rdasens
    Start 4: hafs_4denvar_glbens
    Start 5: hafs_3denvar_hybens
    Start 6: global_enkf
1/6 Test #3: rrfs_3denvar_rdasens .............   Passed  858.84 sec
2/6 Test #6: global_enkf ......................   Passed  2735.79 sec
3/6 Test #5: hafs_3denvar_hybens ..............   Passed  3085.84 sec
4/6 Test #4: hafs_4denvar_glbens ..............   Passed  3624.06 sec
5/6 Test #1: global_4denvar ...................   Passed  3727.23 sec
6/6 Test #2: rtma .............................   Passed  3850.52 sec

100% tests passed, 0 tests failed out of 6

Total Test time (real) = 3850.55 sec

Hercules
All GSI ctests are still pending on Hercules. The machine has 2803 jobs in the queue. 103 are running. 2700 are pending. Not sure when the Hercules ctests will complete.

[RussTreadon-NOAA]

Hercules

Test project /work/noaa/da/rtreadon/git/gsi/pr810/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_rdasens
    Start 4: hafs_4denvar_glbens
    Start 5: hafs_3denvar_hybens
    Start 6: global_enkf
1/6 Test #3: rrfs_3denvar_rdasens .............   Passed  27630.52 sec
2/6 Test #6: global_enkf ......................   Passed  29783.16 sec
3/6 Test #2: rtma .............................   Passed  31817.00 sec
4/6 Test #5: hafs_3denvar_hybens ..............   Passed  31827.25 sec
5/6 Test #1: global_4denvar ...................   Passed  31994.43 sec
6/6 Test #4: hafs_4denvar_glbens ..............***Failed  32437.47 sec

83% tests passed, 1 tests failed out of 6

Total Test time (real) = 32437.51 sec

The hafs_4denvar_glbens failure is due to

The runtime for hafs_4denvar_glbens_loproc_updat is 368.637720 seconds.  This has exceeded maximum allowable threshold time of 348.262471 seconds, resulting in Failure time-thresh of the regression test.

The runtime for hafs_4denvar_glbens_hiproc_updat is 264.768721 seconds.  This has exceeded maximum allowable threshold time of 253.254322 seconds, resulting in Failure of timethresh2 the regression test.

This is not a fatal failure. The /work/noaa/stmp fileset is know to impact executable wall time.

[RussTreadon-NOAA]

Additional tests on Dogwood

Run the following cases:

1. C24L31 with C12L31 ensemble
2. C96L127
3. C192L127
4. C384L127
5. C766L127 with C384L127 ensemble

Cases 2 through 4 were run twice: (a) develop gsi.x using the binary global_berror, (b) feature/netcdf_berror gsi.x using the netcdf global_berror. For each case the analysis results are identical between the executables and global_berror format.

The two jobs for Case 5 remain pending in the queue.

Case 1 is the ultra low-resolution case to be used in g-w CI. The feature/netcdf_berror gsi.x successfully ran to completion using the L31 netcdf global_berror file.

[RussTreadon-NOAA]

Case 5, C768L127 with C384L127 ensemble is done. feature/netcdf_berror gsi.x using the netcdf global_berror generates analysis increments that are bitwise identical to those generated by develop gsi.x using the binary global_berror

[RussTreadon-NOAA]

@danholdaway , @DavidNew-NOAA , and @CatherineThomas-NOAA

This PR adapts code from the JEDI GSIBEC to allow gsi.x to read netcdf format global_berror files. EIB added the netcdf global_berror files to their staged GSI directory. gsi.x determines the file format, netcdf or binary, upon reading local file berror_stats in the run directory.

@CoryMartin-NOAA created netcdf global_berror files for for C24L31 and C12L31. He also created a C24L31 deterministic background along with a 3 member C12L31 ensemble. These L31 netcdf global_berror files are in EIB's staged directory along with netcdf global_berror files for other resolutions. The updated fix hash in this PR brings in L31 ascii fix files. I successfully ran 3DEnVar gsi.x with a C24 deterministic background and a 3-member C12 ensemble.

GSI code review policy requires two peer reviews and approvals before PRs can be merged into develop. If any of you have time, a review would be helpful. Thanks.