CMORsnack: success across almost all, if not all, variables

[ilaflott]

i.e. there should be as few "special cases" as possible.

remaining cases from the bite:

• 4. atmos gr1 / Amon / ta
  • allow no-lon coordinate axis case
• CMORmeal: successfully process ocean variables on tripolar grid with cmor #298 for the next push: CMORmeal
  • [ ] nc_copy bug
  • [ ] rewrite ocean file with missing lat(yh, xh), lon(yh, xh) from statics if avail
  • find statics file in bronx pp directory
  • copy lat/lon info from statics file
  • compute _bnds for copied lat/lon on native grid
  • check that these are appended to the file correctly
  • spin off to sep issue
• 7. ocean gr / Omon / so (three dimensional ocean)
  • [ ] nc_copy bug
  • expecting olevel, has z_i, z_l instead.
• 8. atmos gn / Amon / ch4global
  • allow no-lon coordinate variable case
  • allow no-lat coordinate variable case
• 9. LUmip gr1 Emon / gppLut
  • [ ] come up with an edge-case work around for non-typical coordinate in place of a more typical vertical coordinate
  • rewriting example
  • rewrite the landfile to use a landuse coordinate variable to be string-valued
  • do it on the fly

[ilaflott]

related issues: #250, #241

[ilaflott]

case 4 works as of #243 / 5fb52ce

[ilaflott]

case 8 works as of #243 / 30adf0f

[ilaflott]

RE case 9, LUmip gr1 Emon / gppLut, coordinate variable landUse must be present and string-valued, instead of an integer flag e.g. landuse. rewriting example

RE cases 6 and 7, nc_copy bug now irrelevant because nc_copy is not needed for copying. It's not clear why it failed and it doesn't matter.

RE case 6, ocean gn / Omon / sos specifically: CF conventions and CMIP demand that for every yh, xh coordinate on a native ocean grid, there should be a corresponding lat(yh, xh) and lon(yh, xh) appended for easy reference. I thought there might be a way to do this conversion on-the-fly, but that was before i learned how tricky the tripolar grid is to describe analytically. 1) tripolar grids aren't so simple that one can just ask for a general implementation of projections back to spherical coordinates... and even if i found one that did what i wanted specifically, 2) this thing is likely embedded in regridding routines that do far more than what i need: a simple projection operation.

Somewhat anti-climactically, by convention, these numbers are usually already computed and present in a corresponding statics file provided by fre-bronx workflows. run_test_file_cases.py comments/notes updated accordingly. to do: proof-of-concept rewriting script ala land file.

RE case 7, ocean gr / Omon / so (three dimensional ocean), CF/CMIP are looking for olevel, which is particularly opaque. I've got some leads and am parsing the details now. This is on the spherical 1x1 degree lat/lon grid, so i won't run into the same problem as I have with case 6.

[ilaflott]

case 7 now works as of #243 / 920a361

computing z_l's bounds from z_i in the same file, and calling it depth_coord to cmor, is what was needed.

i suspect i can do the same thing with case 9 now.

[ilaflott]

case 9 now works as of #243 / 83eaaa5

my speculation from the previous comment was correct

[ilaflott]

RE case 6, though this would be easier... probably making a silly mistake somewhere. Haven't given up quite yet!

Regardless, for future development/reference's-sake, some sources i want to jot down here with how to address this case for CMIP:

• CF convention on horizontal lat/lon coordinates on 4-sided cells on a non-rectangular coordinate system
• ProjectPythia doc with an example showing how one writes a lat(y,x) and lon(y,x) data
  • note that in this example, the lat and lo values are generaated explicitly. this is not what we are aiming to do, but the way the data is being written to the file after it's generated, THAT i believe is what we need

[ilaflott]

RE case 6, this IS going to be a little tricky.

• latitude needs bounds, longitude will obviously too
• to make bounds, peek at the CF convention link i linked in my prev comment... wait, i need more than the h point nominal lat/lon!
• ... whew, ncdump -h /path/to/ocean_statics | egrep geol shows we have the required info (some output removed for space):

	float geolat(yh, xh) ;
		geolat:long_name = "Latitude of tracer (T) points" ;
	float geolat_c(yq, xq) ;
		geolat_c:long_name = "Latitude of corner (Bu) points" ;
	float geolat_u(yh, xq) ;
		geolat_u:long_name = "Latitude of zonal velocity (Cu) points" ;
	float geolat_v(yq, xh) ;
		geolat_v:long_name = "Latitude of meridional velocity (Cv) points" ;

	float geolon(yh, xh) ;
		geolon:long_name = "Longitude of tracer (T) points" ;
	float geolon_c(yq, xq) ;
		geolon_c:long_name = "Longitude of corner (Bu) points" ;
	float geolon_u(yh, xq) ;
		geolon_u:long_name = "Longitude of zonal velocity (Cu) points" ;
	float geolon_v(yq, xh) ;
		geolon_v:long_name = "Longitude of meridional velocity (Cv) points" ;

• gotta get this vertex notation right: see MOM6 doc here, and for background, read about the Arakawa grid... uh oh, what scheme are we using??
• according to the MOM6 repo github.io, C grid, probably

[ilaflott]

RE case 6

• geolon_q and geolat_q would be the numbers i need for the bounds.
• the bounds happen to already be computed + stored at https://gitlab.gfdl.noaa.gov/pcmdi/cmip6_utils

[ilaflott]

RE case 6

• the bounds at the above link have a differing resolution than the data i am working with. the data i am working with has 1161 yh coordinate points, yet the files the link points to have 1080...
  • 🎵 🎶 "... you can't always get, what you want..." 🎶 🎵
• the resolution of the statics file in the assoc FRE bronx directory IS the correct # of data points, but has no bounds.
  • 🎶 🎵 "... you can't always get, what you want..." 🎵 🎶
• peek at history attribute of the files in cmip6_utils linked above... being pointed to a GOLD directory. I peruse the directory and find a notebook called "Generate lat lon bounds.ipynb"
  • 🎶 🎵 "... but if you try sometimes... " 🎵 🎶
• see an algorithm that should generalize to my exact scenario...
  • 🎵 🎶 "... you can get what you nee-EED" 🎶 🎵

>>> epmt notebook -- --no-browser --ip=$(hostname -i) --port=$(jhp epmt) --notebook-dir=/archive/gold/datasets/OM4_025/mosaic.v20170622.unpacked
import numpy
import netCDF4
new_file.createDimension('xh', orig_static.dimensions['xh'].size)
new_file.createDimension('yh', orig_static.dimensions['yh'].size)
new_file.createDimension('vertex', 4);
# Variable definitions
lon_bnds = new_file.createVariable('lon_bnds', numpy.float32, ('yh','xh','vertex',))
lat_bnds = new_file.createVariable('lat_bnds', numpy.float32, ('yh','xh','vertex',))
vals = orig_static.variables['geolon_c'][:] # Read all coordinate values

# Write lon_bnds data
data = numpy.zeros(lon_bnds.shape) # Array to populate and write
data[:,:,0] = vals[1:,1:] # NE corner
data[:,:,1] = vals[1:,:-1] # NW corner
data[:,:,2] = vals[:-1,:-1] # SW corner
data[:,:,3] = vals[:-1,1:] # SE corner
lon_bnds[:] = data[:] # Write to file

# Write lat_bnds data
vals = orig_static.variables['geolat_c'][:] # Read all coordinate values
data = numpy.zeros(lat_bnds.shape) # Array to populate and write
data[:,:,0] = vals[1:,1:] # NE corner
data[:,:,1] = vals[1:,:-1] # NW corner
data[:,:,2] = vals[:-1,:-1] # SW corner
data[:,:,3] = vals[:-1,1:] # SE corner
lat_bnds[:] = data[:] # Write to file

[ilaflott]

RE case 6:

• GAH!!! the missing sauce is here
• 
  • EDIT: not exactly, but this is why/where projected_{x,y}_coordinate is accepted

[ilaflott]

obvious next question... how to use the above linked info and, where's the doc???

• the info from CMIP6_grids.json in a sample file i downloaded is these lines
• none of CMOR's python examples go over this
• the api doc section covers some info, but i'm finding it's explanation(s) lacking.
• the getting started section had the most, search for grid and come to these bits:
  • "CMIP6_grids.json supplements the Coordinates Table with axis information that is sometimes needed in the treatment of unstructured grids."
  • ... "In the case of non-Cartesian longitude-latitude grids or for “station data”, define the grid and its mapping parameters (cmor_grid and cmor_set_grid_mapping)" ...

[ilaflott]

• PCMDI/cmor issue about tripolar grid
  • not clear if original user, @jxmaster found the ensuing back/forth helpful
• seems like i need to do a dummy-index thing. OK, can try that.

[ilaflott]

PCMDI/cmor#766 conversation now also spanning PCMDI/cmor#767

[ilaflott]

Case 6 will be followed up in #298. for focus, and to get what is currently effectively done merged in for the release. 8/9 cases isn't too bad for initial workflow usage.