Intake conversion NearestNeighbourDistance

[rbeucher]

Following the discussion in issue #313, we propose converting the recipes to use Intake, given that the Cookbook is no longer supported and the ACCESS-NRI Intake catalog is now available.

A few months ago, @max-anu began working on this transition. This pull request contains the changes @max-anu made to the notebook specified in the title.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[navidcy]

This is ready to be reviewed. Any volunteers @COSIMA/cosima-contrib?

[navidcy]

Ready to review!

[julia-neme]

Line #3.    import pandas as pd

Remove as not used

Reply via ReviewNB

[julia-neme]

Line #2.    client = Client("tcp://10.6.48.33:8786")

Curious as to what this means/why we need it? Is it better than dong Client()? I get this error when running it:

OSError: Timed out trying to connect to tcp://10.6.48.33:8786 after 30 s

Reply via ReviewNB

[julia-neme]

Running this with a "small" ARE session crushes the kernel. I think this is because it is not efficient to try to open and then subset the times...

Reply via ReviewNB

[julia-neme]

Line #2.    cat_subset = catalog['01deg_jra55v140_iaf_cycle4']

Already run in the cell above.

Reply via ReviewNB

[julia-neme]

Line #4.    var_search = var_search.search(path=var_search.df['path'][0])

What does this do/why we need it?

Reply via ReviewNB

[anton-seaice]

This is hack to only load this data from one file (it doesn't change in time, but is saved multiple time in the model output)

Its equivalent to the n=1 argument supplied to getvar in the cookbook version

[julia-neme]

I have only commented on the intake part since this is what this Issue is about, but I'm wondering whether we want to convert this to model agnostic/cfxarray sort of example.

But my biggest concern is that I tried to run it using a small ARE session, and my kernel crashed trying to open the data (3rd cell). It is perfectly fine to open it with the cosima_cookbook. Related to my test here.

[navidcy]

But my biggest concern is that I tried to run it using a small ARE session, and my kernel crashed trying to open the data (3rd cell). It is perfectly fine to open it with the cosima_cookbook. Related to my test here.

I'm cc'ing here @rbeucher, @anton-seaice, @dougiesquire, @AndyHoggANU in case they want to comment on this?

[julia-neme]

See comments in ReviewNB

[dougiesquire]

I'm cc'ing here @rbeucher, @anton-seaice, @dougiesquire, @AndyHoggANU in case they want to comment on this?

I think this is due to the way the CICE data are structured (with the 2D coordinates saved in every file). The issue has been brought up in the past on the forum in the context of the cookbook - see here. There are two solutions described in that topic:

• open with decode_coords = False which will drop the 2D coords - this is what was being done with the cookbook before this PR (hence why it worked @julia-neme), but the kwarg was dropped in this PR (it's still actually mentioned in a comment)
• open with compat="override", data_vars="minimal", coords="minimal" to skip coordinate verification - this is the better approach as you'll keep the 2D coords in your returned Dataset.

So, pass:

xarray_combine_by_coords_kwargs=dict(
    compat="override",
    data_vars="minimal",
    coords="minimal"
)

to your to_dask() calls. I've tried this and the nodebook runs on a small ARE instance.

[anton-seaice]

As dougie noted, adding the xarray_combine_by_coords_kwargs would be better than using decode_coords as referred to in this note. This note needs updating along with the cell below.

Reply via ReviewNB

[anton-seaice]

I suggest getting rid of the cat subset line and merging into the var_search line.

and changing darray to ds because its a dataset not a data array

This would be similar to cell 4 in https://github.com/COSIMA/cosima-recipes/blob/4fd51150331ebb1669fdb8bc58ff4ac16b313b0e/Examples/Bathymetry.ipynb

Reply via ReviewNB

[anton-seaice]

We can make the var_ice chunks larger to get rid of these warnings I think?

Reply via ReviewNB