Support single griddap dataset with 2d and 1d variables.

[ChrisJohnNOAA]

This needs some investigation on if and how to implement the feature. Original message below:

Pinging @ChrisJohnNOAA is this capacity that @jklymak describes something ERDDAP can support/could support in future?

As I understand, the desired behavior is to have a single griddap dataset which serves 2-D gridded variables like temperature(profile_num, depth_bin) as well as 1-D variables like lat(profile_num), without broadcasting these 1-D variables to 2-D.

At the moment, from reading the docs @rmendels highlighted ( In EDDGrid datasets, all data variables MUST use (share) all of the axis variables.), it seems this is not currently supported. So you would have to create two different datasets on your ERDDAP server to achieve this.

Originally posted by @callumrollo in #177 (comment)

[jklymak]

Thanks for considering @ChrisJohnNOAA

As noted elsewhere in the discussion, NCEI standard has this format as well:
https://www.ncei.noaa.gov/data/oceans/ncei/formats/netcdf/v2.0/index.html, eg https://www.ncei.noaa.gov/thredds-ocean/catalog/example/v2.0/catalog.html?dataset=example/v2.0/NCEI_trajectoryProfile_template_v2.0_2016-09-22_181838.014029.nc has a structure like:

Dimensions:      (trajectory: 1, obs: 10, z: 4)
Coordinates:
  * trajectory   (trajectory) int32 -2147483647
    time         (trajectory, obs) object ...
    lat          (trajectory, obs) float64 ...
    lon          (trajectory, obs) float64 ...
  * z            (z) float64 1.0 2.0 3.0 4.0
Dimensions without coordinates: obs
Data variables:
    sal          (trajectory, obs, z) float64 ...
    temp         (trajectory, obs, z) float64 ...

Also H.6.2 at https://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/build/aphs06.html

This is a super useful way to organize data sets so hopefully it's not too hard to implement.

[jenseva]

Thanks for considering this!

Copying over from the discussion here:

I believe Callum is correct, this is a present limitation in ERDDAP.

Here is an except Bob in 2017:
_If a variable in the source file is e.g., lat and lon values that use different dimensions than the main data variables and that convert the projection x,y locations into lat and lon, then in ERDDAP they need to be in a separate dataset.

See https://coastwatch.pfeg.noaa.gov/erddap/download/setupDatasetsXml.html#dataStructures
and the subsequent few paragraphs.

I know this sounds goofy and severely limiting. It is the one major situation where using ERDDAP isn't the best choice. But it only affects a small percentage of the total data files in NOAA (that is small consolation for you where it affects perhaps 100% of your data files for polarwatch). There is a solution -- a modification to ERDDAP that would support this but doing it would be a massive effort on my part (a couple of months with no distractions) so I haven't had time to do it.

There was a reason for doing it this way: it is this slightly-simpler-than-netcdf data model that allows ERDDAP to read data from many file types and write data to many file types. So there is great benefit, but it comes at a cost. Few people/groups/datasets pay the cost, but you are. Sorry._

The solution we had to implement at PolarWatch meant there with two datasets which was a bit of a hack and difficult for users.

It would be great to see this feature added to ERDDAP! I agree there is value in having this type of synthesized glider data accessible via griddap over tabledap. The list of benefits is quite long.

• Personally, at Scripps IDG we have similar glider products that would benefit from this feature.
• Please also check in with Dale and Sunny on the PolarWatch use case with gridded projected datasets.
• This feature may also accomodate satellite swath data and be of interest to CoastWatch/NASA.

Best,
Jenn