Multi-file datasets with different calendars between files (non-monotonic global indexes)

[tomvothecoder]

Original comment: #343 (comment)

Thanks for pointing the possibility of datasets using different calendars (which I think is not correct or good practice?).

I ran some experiments to try to open up those files in native xarray with different settings.

1. Keep time coordinates as integers (decode_times=False)

• Code:

   import xarray as xr
   
   dpath = '/p/css03/esgf_publish/CMIP6/PMIP/MIROC/MIROC-ES2L/past1000/r1i1p1f2/Amon/tas/gn/v20200318/*nc'
   
   ds= xr.open_mfdataset(dpath, decode_times=False)

• Produces error: ValueError: Resulting object does not have monotonic global indexes along dimension time

• Why is this happening?

  This isn't a bug, it's throwing the error it should be throwing given your input. However I can see how the documentation doesn't make it very clear as to why this is happening!
  ...

  That error is an explicit rejection of the input you gave it: "you gave me overlapping coordinates, I don't know how to concatenate those, so I'll reject them." Normally this makes sense - when the overlapping coordinates aren't NaNs then it's ambiguous as to which values to choose.
  ...

  Instead, what you need to do is trim off the overlap first. That shouldn't be hard - presumably you know (or can determine) how big your overlap is, and all your NaNs are on one dataset. You just need to use the .isel() method with a slice. Once you've got rid of the overlapping NaNs then you should be able to combine it fine (and you shouldn't need to specify compat either). If you're using combine_by_coords as part of opening many files with open_mfdataset then it might be easier to write a trimming function which you apply first using the preprocess argument to open_mfdataset.
  -- https://stackoverflow.com/a/62860066

Based on the above, it seems like handling this edge case using preprocess might be beyond the scope of xcdat (data quality, possibly complex workaround).

2. Try to decode time coordinates as datetimes (decode_times=True)

• Code:

   import xarray as xr
   
   dpath = '/p/css03/esgf_publish/CMIP6/PMIP/MIROC/MIROC-ES2L/past1000/r1i1p1f2/Amon/tas/gn/v20200318/*nc'
   
   ds = xr.open_mfdataset(dpath, decode_times=True)

• Produces warning

       /home/vo13/miniconda3/envs/xcdat_dev/lib/python3.9/site-packages/xarray/coding/times.py:150: SerializationWarning: Ambiguous reference date string: 850-1-1. The first value is assumed to be the year hence will be padded with zeros to remove the ambiguity (the padded reference date string is: 0850-1-1). To remove this message, remove the ambiguity by padding your reference date strings with zeros.
       warnings.warn(warning_msg, SerializationWarning)
       /home/vo13/miniconda3/envs/xcdat_dev/lib/python3.9/site-packages/xarray/coding/times.py:682: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
       dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime)
       /home/vo13/miniconda3/envs/xcdat_dev/lib/python3.9/site-packages/xarray/core/indexing.py:524: SerializationWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using cftime.datetime objects instead, reason: dates out of range
       return np.asarray(array[self.key], dtype=None)

• Produces error: TypeError: Cannot combine along dimension 'time' with mixed types. Found: Datetime360Day, DatetimeProlepticGregorian. If importing data directly from a file then setting 'use_cftime=True' may fix this issue.

# 3. Try to decode time coordinates as datetimes (decode_times=True)

• Code:

   import xarray as xr
   
   dpath = '/p/css03/esgf_publish/CMIP6/PMIP/MIROC/MIROC-ES2L/past1000/r1i1p1f2/Amon/tas/gn/v20200318/*nc'
   
   ds = xr.open_mfdataset(dpath, decode_times=True, use_cftime=True)

• Produces error: TypeError: Cannot combine along dimension 'time' with mixed types. Found: Datetime360Day, DatetimeProlepticGregorian. If importing data directly from a file then setting 'use_cftime=True' may fix this issue.

Originally posted by @tomvothecoder in #343 (comment)