python-data/notebooks/intro_to_ncas_cf_data_tools.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0cd6019e-c5b3-4dc6-bf86-6fcf10b06851",
   "metadata": {},
   "source": [
    "# Introduction to the NCAS CF Data Tools, cf-python and cf-plot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4378ad0-8448-44ef-80e2-cd6e67346a1b",
   "metadata": {},
   "source": [
    "## Context and learning objectives\n",
    "\n",
    "### What are the NCAS CF Data Tools and why do they all have 'cf' in the name?\n",
    "\n",
    "The _NCAS CF Data Tools_ are a suite of complementary Python libraries which are designed to facilitate working with data for research in the earth sciences and aligned domains. The two that are of most relevance to the average user, and those wanting to process, analyse and visualise atmospheric data, are *cf-python* (https://ncas-cms.github.io/cf-python/) and *cf-plot* (https://ncas-cms.github.io/cf-plot/build/). We will be focusing on use of cf-python and cf-plot today.\n",
    "\n",
    "The 'cf' in the names of the NCAS CF Data Tools corresponds to the _CF Conventions_, a metadata standard, because they are built around this standard in the form of using the CF Data Model, which as well as performance is considered a 'unique selling point' of the tools.\n",
    "\n",
    "\n",
    "### What are the CF Conventions?\n",
    "\n",
    "The _CF Conventions_, usually referred to in this way but also know by the full name of the **C**limate and **F**orecast (CF) metadata conventions, are a metadata standard which is becoming the de-facto convention to cover the description of geoscientific data so that sharing and intercomparison is simpler. See https://cfconventions.org/ for more information.\n",
    "\n",
    "\n",
    "### What are we going to learn in this session?\n",
    "\n",
    "Our **learning aim** is to be able to use the NCAS CF Data Tools Python libraries, namely cf-python and cf-plot to process, analyse and visualise netCDF and PP datasets, whilst appreciating the context and 'unique selling point' of the libraries as being built to use the CF Conventions, a metadata standard for earth science data, to make it simpler to do what you want to do with the datasets, by working on top of a Data Model for CF.\n",
    "\n",
    "We have **six distinct objectives**, matching the sections in this notebook and in the practical notebook you will work through. By the end of this lesson you should be familiar and have practiced using cf-python and cf-plot to:\n",
    "\n",
    "1. read dataset(s) and view the (meta)data at different detail levels;\n",
    "2. edit the (meta)data and write out the edited version to file;\n",
    "3. reduce datasets by subspacing and collapsing;\n",
    "4. visualise datasets as contour and vector plots;\n",
    "5. analyse data: applying mathematical and statistical operations and plotting trends;\n",
    "6. change the underlying grid of data through regridding."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9d89bbc-e583-45cc-a79f-cf5fdb783050",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<i>Note:</i> much of what you can do with cf-python you can do with the xarray library. Use whichever approach, the cf-python/cf-plot way, or the xarray way, works best for you! However, we want to emphasise that the NCAS CF Data Tools are built around the CF Conventions whereas xarray is not, so cf-python and cf-plot offer better metadata awareness to xarray, which could be a core advantage to our approach for users in/from geoscience. (If you have suggestions for how we can improve cf-python and/or cf-plot for you or your work, please let us know through the Issue Trackers linked at the end of this Notebook.)\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c3beb87-7b0d-44ad-9da3-fa32494b795c",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "196e9e0a-83d1-464c-b53c-a6fc2e0c19ff",
   "metadata": {},
   "source": [
    "## Setting up\n",
    "\n",
    "**In this section we set up this Notebook, import the libraries and check the data we will work with, ready to use the libraries within this notebook.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d073760a-6734-4a1d-9832-a70a8e0dd089",
   "metadata": {},
   "source": [
    "Import cf-python and cf-plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9640ef13-50d2-43fe-9a84-5ce701bdfcbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "import cfplot as cfp\n",
    "import cf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab792505-5452-4f44-b194-6181e977385a",
   "metadata": {},
   "source": [
    "Run some set up for nice outputs in this Jupyter Notebook (not required in interactive Python or a script):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9bfec50d-94c0-442a-a74d-0a4deb5c4bb0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b00d47b-4497-4540-b1bd-6fa8fa8e3226",
   "metadata": {},
   "source": [
    "Inspect the versions of cf-python and cf-plot and the version of the CF Conventions those are matched to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea75b1b6-ba78-428d-b9b2-ca29a8cf6f0d",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"cf-python version is:\", cf.__version__)\n",
    "print(\"cf-plot version is:\", cfp.__version__)\n",
    "print(\"CF Conventions version is:\", cf.CF())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "628af6cc-24f1-4e85-bf8b-654268e72bd3",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<i>Note:</i> you can work with data compliant by any other version of the CF Conventions, or without (much) compliance, but the CF Conventions version gives the maximum version that these versions of the tools understand the features of.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8135f5c6-3d3e-45b6-a84a-3dcea754372f",
   "metadata": {},
   "source": [
    "Finally, see what datasets we have to explore:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfab63fe-088d-4ca8-ab95-40f61f0c2fd3",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<i>Note:</i> in a Jupyter Notebook, '!' preceeeds a shell command, so this is a terminal command and not Python\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a45afce7-abd8-4edd-b14c-eb033566bd52",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!ls ../data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7640cdb-0d6f-4933-86d7-e86e7b9acbe4",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b7a3c0a-262d-4ac4-8c10-6fa3a42ae73a",
   "metadata": {},
   "source": [
    "## 1. Reading dataset(s) and viewing the (meta)data at different detail levels\n",
    "\n",
    "**In this section we look at a basic use of cf-python, reading in one or more datasets from file and inspecting the data and the metadata at different levels of detail to suit the amount of information you want to see.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15de8a85-99ae-40a8-a429-34c30a3e1c76",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<i>Note:</i> In cf-python and when discussing related code and datasets, we use terminology from the CF Data Model (for more detail see: <a href=\"url\">https://ncas-cms.github.io/cf-python/cf_data_model.html</a>). For example cf-python methods are named in relation to concepts from this data model. We don't have time to cover this in detail but for this session it is useful to know the following terms:\n",
    "\n",
    "<ul>\n",
    "    <li><b>field</b>: a self-contained cf-python object corresponding to a netCDF data variable with all of its (CF) metadata attached;</li>\n",
    "    <li><b>field list</b>: a list of lields (see above), stored as its own cf-python object 'FieldList' which is similar to a Python list;</li>\n",
    "    <li><b>coordinate</b>: a (CF) metadata concept which corresponds to netCDF coordinate variables. One or more coordinates are defined on every field as either 'dimension' or 'auxiliary' coordinate objects in cf-python.</li>\n",
    "</ul>\n",
    "</div>\n",
    "\n",
    "The examples from this section should help you to familiarise yourself with these terms and their practical usage."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9da08bf9-a15b-4d65-8c8e-632897daec0b",
   "metadata": {},
   "source": [
    "### a) Reading in data and extracting the _field_ of interest"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60d6c6a4-a3cd-4515-9d7c-e0e5b0fdc1f3",
   "metadata": {},
   "source": [
    "Read a chosen data file. Sometimes datasets have descriptive names but this one doesn't, so let's find out what it is!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67b1cd25-dedb-409b-8a57-3fff5de333dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "fieldlist = cf.read(\"../data/data1.nc\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e49e1f9-3246-4e67-ac94-a322937c2c42",
   "metadata": {},
   "source": [
    "See the 'fieldlist' that cf-python interprets from the data read in:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70dfc26a-3163-4b56-948d-98a27472dce4",
   "metadata": {},
   "outputs": [],
   "source": [
    "fieldlist"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b99aa858-bdd3-48d4-b91b-b6fd122fdcb4",
   "metadata": {},
   "source": [
    "Select a particular field from the fieldlist of interest:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "269896a7-4f55-4792-b9be-fcbc7d931edc",
   "metadata": {},
   "outputs": [],
   "source": [
    "field = fieldlist[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4ed8112-fc35-496c-84ae-e8a05334c33c",
   "metadata": {},
   "source": [
    "### b) Inspecting the _field_ of interest with different amounts of detail"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b78cc22c-c52b-4d8d-bcab-15da6752b592",
   "metadata": {},
   "source": [
    "View the field with **minimal detail**, i.e. a one-line summary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84ab31d2-814d-41a7-8da6-50ab6f9a0969",
   "metadata": {},
   "outputs": [],
   "source": [
    "field"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a9af4ea-270a-49e9-bc49-54c36158bf09",
   "metadata": {},
   "source": [
    "Or you can view it with a **medium level of detail** with the Python built-in `print` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01618a8c-928a-4250-ba3c-4fef28bfa19e",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(field)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8c2af1e-fd8d-415f-b430-d521e68b98ea",
   "metadata": {},
   "source": [
    "A final option is to view it with **maximal detail** using the `dump()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb1170b2-dd14-4ebf-8bc4-7e0559c8a4a0",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "field.dump()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0af01357-b0b2-4581-a830-437337565d0d",
   "metadata": {},
   "source": [
    "### c) Inspecting a metadata _construct_ e.g. _coordinate_ from the _field_ of interest"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f348261-09fe-4193-ab6f-3203cbd36e40",
   "metadata": {},
   "source": [
    "Use the same approach to view a particular metadata aspect, for example the latitude coordinate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d907740-67af-4c28-a13d-6a2a707cc531",
   "metadata": {},
   "outputs": [],
   "source": [
    "lat = field.coordinate(\"latitude\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec72414b-cebd-4bb6-bf8b-b7f13ebf394d",
   "metadata": {},
   "outputs": [],
   "source": [
    "lat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "307259f7-8528-4fe9-8587-449a70836461",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(lat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e342afb-519f-42c6-889a-aecfb1e1ced6",
   "metadata": {},
   "outputs": [],
   "source": [
    "lat.dump()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd7fd831-8362-49eb-a03a-281d75596309",
   "metadata": {},
   "source": [
    "### d) Inspecting a data array of interest"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb66c5c4-c535-4710-bfdc-bfb6d0fde827",
   "metadata": {},
   "source": [
    "Likewise, the same approach works to view the data itself in the field (i.e. the underlying arrays). First, grab the data from the field with the `data` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "562b2a02-e61a-472e-85e2-7020f704e0cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = field.data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b385616e-a130-4bf1-a197-f0ae8e193948",
   "metadata": {},
   "source": [
    "Then view it in a chosen level of detail as with the above objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e650fb5e-319a-41a1-bae9-b9d7e99a43c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e07f7420-37d1-4ba4-8cd8-f068425301bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0147639e-56a4-4a50-9580-02ba0b557493",
   "metadata": {},
   "outputs": [],
   "source": [
    "data.dump()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4aa94e2b-7226-4846-be7d-696e3996eb8f",
   "metadata": {},
   "source": [
    "If you want to see more of the data array itself, you can access it with the `array` attribute. Beware, for real-life datasets:\n",
    "\n",
    "* this will be large and Python will likely truncate it so your screen isn't spammed with sub-arrays of values!\n",
    "* it is computationally intensive to access the underlying data array if it is large, especially if it is multi-dimensional, so your computer will often have to work hard to get the array, so use the `array` method sparingly (only when needed)!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89d5c9f7-423f-4e03-9517-be40143e5aa4",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data.array"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09d04ff2-8998-48fb-a0a6-bc5deafe8fc6",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "932d2d38-b8d7-46cc-848f-b6f50b877310",
   "metadata": {},
   "source": [
    "## 2. Editing the (meta)data and writing out the edited version to file\n",
    "\n",
    "**In this section we demonstrate how to change the data that has been read-in from file, both in terms of the data arrays and the metadata that describes it, and then how to write data back out to file with a chosen name, so that you can see how cf-python can be used to edit data or to make new data.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf3c9786-528f-49c8-8388-574062112206",
   "metadata": {},
   "source": [
    "Using the same data file from te previous section, let's say we want to change the data and metadata of this. As-is the field and its data are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7af83b82-0258-4ba2-bb3e-7c0c0ae6e929",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Field is:\", field, \"Data is:\", data, sep=\"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc7c654e-38e5-4b06-a7c0-01828701d92f",
   "metadata": {},
   "source": [
    "### a) Changing the underlying data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9acf4da4-881b-46c7-88f3-8fe72972cf1b",
   "metadata": {},
   "source": [
    "To change the data, use assignment to the relavant index or indices. For example, to change all values we can use the special index of an ellipsis like so, in this case changing them all to an identical scalar value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3983e97-ffc8-4799-9a29-fa88e036a169",
   "metadata": {},
   "outputs": [],
   "source": [
    "data[...] = 10.0\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc968051-56f8-4814-a79b-be3402e0b285",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06e5ef92-f3ab-4b5f-b004-4e2fa41429e5",
   "metadata": {},
   "source": [
    "Or could change more specifically just one sub-array of these to a different value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b168b278-0418-4ed0-85cf-de6d0e795638",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data[0, 0, 0] = 3.0\n",
    "data.array"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba4ebc7f-45c4-4c9c-a7e7-4b7cafe2d305",
   "metadata": {},
   "source": [
    "Instead of setting the whole sub-array to one value, you can set the whole array to your precise specification, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bbe84007-811a-4290-bfbe-9e222bded840",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data[0, 0, 0] = range(320)\n",
    "data.array"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f2d2721-58c2-44d9-b422-3dd81ee8b333",
   "metadata": {},
   "source": [
    "### b) Changing some metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3286f679-2c8a-40a8-b7d4-b67f34122e7d",
   "metadata": {},
   "source": [
    "To change metadata, first get the metadata you want to change as an object. One of the most flexible ways to do so is to use the `construct` method and as an argument specify the name of the coordinate you are interested in:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f81cbe5-5d1d-4d30-9388-2750cc81d613",
   "metadata": {},
   "outputs": [],
   "source": [
    "pressure = field.construct(\"pressure\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2fbbdb05-94e4-432b-b54e-2675901e6d75",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(pressure)\n",
    "print(pressure.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4987c89e-e4c0-4c91-aa2e-fa78ff9f617e",
   "metadata": {},
   "source": [
    "You can inspect the units specifically using the `units` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2e55b76-f9e7-41cc-b11f-995c0bc0f2eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(pressure.units)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3051327c-e237-4cdf-adb3-74d5c65d28b4",
   "metadata": {},
   "source": [
    "Let's change the units to an equivalent but different unit, the `bar` (out by a factor of 1000), as an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0f6b8a5-ab63-42b9-8b59-1dfe68d48294",
   "metadata": {},
   "outputs": [],
   "source": [
    "pressure.units = \"bar\"\n",
    "print(pressure.units)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c15baee-fdaf-4015-b265-41920e6c6d5e",
   "metadata": {},
   "source": [
    "Notice how the data has been converted to account for the new units - cf-python's metadata awareness makes contextual changes like this so we don't have to do it manually!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35d51c76-7c36-4b7b-98c6-85df363259da",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(pressure.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3e11a1b-9841-49ff-a74f-5e27e5b08629",
   "metadata": {},
   "source": [
    "Note how the pressure units are changed in the field too, since we edited the same object in a Pythonic sense:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2fa9715-2ac2-4d22-aaec-23edb764708c",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(field)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf7cf826-2d23-40cd-bfca-648de59fd02c",
   "metadata": {},
   "source": [
    "### c) Writing a (list of) fields out to a file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38de7492-13b0-4983-b943-16e23e7851ed",
   "metadata": {},
   "source": [
    "We changed some metadata (units) and the data itself from our dataset read-in from file. Let's write the new data out\n",
    "as a new file and read it back in to show that it has been changed relative to the original. You write files out to disk using the `write` function with an argument giving the path, including the name (it can _just_ be the name to write a file to the current working directory), you want to create the file to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e697f71b-3f6c-45ba-b4bc-547d26e1e8e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "cf.write(field, \"../data/data1-updated.nc\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86c8650c-23a4-40f7-9a26-a9aba1b8ee6a",
   "metadata": {},
   "source": [
    "See that it was written out to the directory we specified:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ce47770-6ee4-482c-8774-16136e235b50",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<i>Note:</i> in a Jupyter Notebook, '!' preceeeds a shell command, so this is a terminal command and not Python\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ef24395-3a11-45d5-b91e-a742e82d80fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls ../data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a0f2ed2-51b2-4cf2-9529-4acf7cf1ff15",
   "metadata": {},
   "source": [
    "To check it wrote out the edited version from this Notebook, we can read the file back in and inspect it again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac357e9f-85e1-4139-9a22-30c5cc0427da",
   "metadata": {},
   "outputs": [],
   "source": [
    "updated_fieldlist = cf.read(\"../data/data1-updated.nc\")\n",
    "reread_field = updated_fieldlist[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae8fd9e7-c789-4e7d-8773-074bdbb8c2ce",
   "metadata": {},
   "source": [
    "See what `g` is by medium detail inspection:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d08cff69-4bc8-4862-bfc2-06acce62ac19",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(reread_field)\n",
    "print(reread_field.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a4f1a78-6dcc-4a65-b927-54b9c044a55d",
   "metadata": {},
   "source": [
    "Notice the pressure coordinate units are 'bar' as per our change and the first data array item starts with `0.0` and the final one ends with `10.0` as per our change."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dd6be4b-60d3-4ca8-8108-fcda62bb7d95",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ccfd6a0-567d-43f3-9f06-15bb21afae6a",
   "metadata": {},
   "source": [
    "## 3. Reducing datasets by subspacing and collapsing\n",
    "\n",
    "**In this section we show how multi-dimensional data can be tamed using cf-python so that you can get a reduced form that can be analysed or plotted, by reducing the dimensions by selecting a subset of point(s) along the axes or collapsing down according to some statistic such as the mean or an extrema.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6899662-eaaa-480c-8f0a-4a6f069292a8",
   "metadata": {},
   "source": [
    "Often datasets represent highly multi-dimensional data, for example 4D or higher. Usually we want to find a either a sub-space, or a statistical representation (such as an average or extrema), of the full data array in less dimensions, such as in 3D or 2D or even in the form of a 1D time series or 0D statistic.\n",
    "\n",
    "We'll demonstrate this with another dataset and field selected from it. This serves as a reminder on concepts from the first section of the Notebook:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e635599e-aa0b-4f48-a49e-28ec8aab9ef7",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<i>Note:</i> here we are reading in a file in the 'PP' format, which is a file format originating from the Met Office that can often be encountered in geoscience, like netCDF, hence the file extension '.pp'. You can read and process PP files using cf-python exactly the same way you do for netCDF files, so you do not need to concern yourself with the difference in file format in your code.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57d21990-e813-41c5-b686-9c90deba307e",
   "metadata": {},
   "outputs": [],
   "source": [
    "field2 = cf.read(\"../data/aaaaoa.pmh8dec.pp\")[2]\n",
    "print(field2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a019f77b-1824-44f8-a534-06445f1ec48f",
   "metadata": {},
   "source": [
    "In this case, see the numbers representing axes sizes on the 'Data' line: we have a 3D field where the axes sizes are 17 for air pressure, 30 for grid latitude and 24 for grid latitude. To reduce this to a 2D form, we need to take one of the non-zero axes and convert it to size 1. This can be done either by **subspacing** or by **statistically collapsing** it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "410ac6d4-cd8f-43ed-a3f5-eaf8d1d8755a",
   "metadata": {},
   "source": [
    "### a) Subspacing using metadata conditions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ed6a2a2-3b35-40b7-9275-eb339eeae23e",
   "metadata": {},
   "source": [
    "Use the `subspace` method to find a subspace of a field, the output of which is another field reduced down in the way specified by the method arguments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4b98910-09ef-4948-93c7-f85d112ce33b",
   "metadata": {},
   "outputs": [],
   "source": [
    "field2_subspace1 = field2.subspace(air_pressure=1000.0000610351562)  # taking first value\n",
    "print(field2_subspace1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19ac33bb-384c-413b-965f-992ba0934a3f",
   "metadata": {},
   "source": [
    "Let's do the same but to subspace to reduce the data along a different axis, this time `grid_latitude`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e71740d-1e23-4295-adf3-15dcdae740e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "field2_subspace2 = field2.subspace(grid_latitude=-5.279999852180481)  # taking the last value\n",
    "print(field2_subspace2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb1db98b-27b3-4c8a-bf63-f67f83dd2bb8",
   "metadata": {},
   "source": [
    "### b) Subspacing using indexing, including equivalency to the above"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "918b145a-0fb1-446a-a003-6e7bb75601d1",
   "metadata": {},
   "source": [
    "We can also use indexing to do a subspace. So, instead of picking out a given value from the printed information,\n",
    "we can use a specific Python index to pick it out.\n",
    "\n",
    "The `air_pressure` coordinate is listed as first in axes order, so you must specify the index at the first position of the three i.e. `[<index>, :, :]` where `:` means to not take a subspace and leave the whole axis as it was. For example, the following takes the first (position 0 in Python indexing) value of the `air_pressure`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abdd7e85-9ae2-4d63-b69b-56a50ca7775b",
   "metadata": {},
   "outputs": [],
   "source": [
    "field2_subspace1_by_index = field2[0, :, :]  # taking first value from first coordinate\n",
    "print(field2_subspace1_by_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8571fdd2-4262-4979-8430-e3738327faa6",
   "metadata": {},
   "source": [
    "To prove that this is the same field as we got using the direct `subspace` method from the sub-section above, namely `h.subspace(air_pressure=1000.0000610351562)`, we can use the `equals` method, which states whether one field is identical to another:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37028743-07e5-454c-becb-bbb1f9e5c71f",
   "metadata": {},
   "outputs": [],
   "source": [
    "field2_subspace1_by_index.equals(field2_subspace1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85f044f1-f4e6-4110-a50c-acd58838a3c3",
   "metadata": {},
   "source": [
    "Similarly, the `grid_latitude` coordinate is listed as second in axes order, so you must specify the index at the second position of the three i.e. `[:, <index>, :]`. Let's take the last (position -1 in Python indexing) value from this, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d42f36cf-4772-4081-94d9-b03a571fdb00",
   "metadata": {},
   "outputs": [],
   "source": [
    "field2_subspace2_by_index = field2[:, -1, :]  # taking last value from second coordinate\n",
    "print(field2_subspace2_by_index)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40204dfd-a51a-4597-876b-70a0127fdd09",
   "metadata": {},
   "source": [
    "Again, to prove that this is the same field as we got as using the direct `subspace` approach previously with `h.subspace(grid_latitude=-5.279999852180481)`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c30fcf5e-fc93-49bc-9ab1-41406fa76954",
   "metadata": {},
   "outputs": [],
   "source": [
    "field2_subspace2_by_index.equals(field2_subspace2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1883b43f-521e-4653-ac36-1069e62fe25d",
   "metadata": {},
   "source": [
    "You can do multiple subspaces at once via either of the methods above, so for example you can combine the two separate subspaces into one call:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f44273de-e39b-406a-bdbc-02d6e23a1412",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "field2_subspace3 = field2.subspace(air_pressure=1000.0000610351562, grid_latitude=-5.279999852180481)\n",
    "field2_subspace3_by_index = field2[0, -1, :]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36b7314b-cabf-4101-8cdc-3f2ec9c468ff",
   "metadata": {},
   "source": [
    "This results in (in both cases):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89485fca-8101-4e16-a0c4-13f4f7e38da9",
   "metadata": {},
   "outputs": [],
   "source": [
    "field2_subspace3.equals(field2_subspace3_by_index)\n",
    "print(field2_subspace3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b870563e-6f45-46ca-9dad-6d8a5cb69ec5",
   "metadata": {},
   "source": [
    "### c) Statistical collapses"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "502933e9-f4d9-42d9-bc40-a2b2ffa75958",
   "metadata": {},
   "source": [
    "Instead of extracting the data at a particular value from the `air_pressure` dimension coordinate, we might want to collapse the data down according to a representative statistic covering all of those values, such as an average or extrema value. We do this with the `collapse` method.\n",
    "\n",
    "Say we want to get the _mean_ of all of the air pressure values to reduce that coordinate from having the 17 values to just one mean representation, we would do this as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6745c82-b11d-4cca-a0f7-4dd825f446cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "field2_collapse1 = field2.collapse(\"air_pressure: mean\")  # taking mean of the 17 values for air pressure\n",
    "print(field2_collapse1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32834e16-5922-4a3d-a466-630ae3791de0",
   "metadata": {},
   "source": [
    "Another example is taking the _minimum_ of all of the values across the grid latitudes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e650cc2-455b-462b-b078-ab8c99b0413e",
   "metadata": {},
   "outputs": [],
   "source": [
    "field2_collapse2 = field2.collapse(\"grid_latitude: minimum\")  # taking minimum of the 30 values for the grid latitude\n",
    "print(field2_collapse2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60b3f3a1-3a17-4a6d-bdaa-a225e7feecb3",
   "metadata": {},
   "source": [
    "Equivalently, you can specify the axes via an `axes` argument when the coordinate to collapse along is one of `X`, `Y`, `Z` or `T`. So this `collapse` call will give the same result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84caa2f2-1e13-493e-aeb8-5a3c7bd7c763",
   "metadata": {},
   "outputs": [],
   "source": [
    "field2_collapse3 = field2.collapse(\"minimum\", axes=\"Y\")  # taking minimum of the 30 values for the grid latitude\n",
    "print(field2_collapse3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08377cd7-54d1-4de6-8a9f-82aff981fccf",
   "metadata": {},
   "source": [
    "Proving that this gives the same result as with the `\"grid_latitude: minimum\"` argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff981201-b983-4f18-a230-3b9c7a2ebf07",
   "metadata": {},
   "outputs": [],
   "source": [
    "field2_collapse3.equals(field2_collapse2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59b69a9d-a94c-464c-aa3c-482c699bc404",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80edfa76-7f87-42b9-8cef-058304ca902a",
   "metadata": {},
   "source": [
    "## 4. Visualising datasets as contour and vector plots\n",
    "\n",
    "**In this section we demonstrate how to plot using cf-plot the data we have read and then processed and/or analysed using cf-python, notably showing how to create contour plots and vector plots as examples of some of the available plot types.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "079820ff-aa40-48c6-8775-46d6b12aec69",
   "metadata": {},
   "source": [
    "For this section let's use other fields within the fieldlist we read in for section one, where we investigated the first of the four fields (index 0) in that section:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a71db8e6-d3a7-4bef-b580-143fff2ffe88",
   "metadata": {},
   "outputs": [],
   "source": [
    "fieldlist"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e227e533-3651-4d92-8c99-c89a8258c310",
   "metadata": {},
   "source": [
    "This time let's investigate the other (final) three fields. First we unpack these to variables to save us applying the indexing multiple times (note we assign variable names `fieldN` to a number `N` incremented with fields read and assigned to variable names from the start of the Notebook, not with numbers corresponding to the index position):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "481c51b9-40cc-4949-bfa6-e77d2a7962c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "temp_field = fieldlist[1]\n",
    "wind_u_field = fieldlist[2]\n",
    "wind_v_field = fieldlist[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e400727a-e49e-4d51-84b5-c263594b41f9",
   "metadata": {},
   "source": [
    "Now let's see what the fields are like, with medium detail, using a new-line separated print to break up the outputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60cdd349-a7df-4788-8332-5fee2cbe3598",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"'temp_field' is:\", temp_field, \"\\n\")\n",
    "print(\"'wind_u_field' is:\", wind_u_field, \"\\n\")\n",
    "print(\"'wind_v_field' is:\", wind_v_field, \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "794c4940-43fd-45a3-ba9a-69084c0e5ccd",
   "metadata": {},
   "source": [
    "### a) Making a contour plot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3389b8d4-a3ff-415f-9507-cf3baa82c5a1",
   "metadata": {},
   "source": [
    "All of these fields, as we can see from the latitude and longitude value ranges, span the whole globe in extent. Another quick way to gauge this, and plenty of other information about a field notably the data itself and its form and pattern(s), is to plot it.\n",
    "\n",
    "To make a contour (level) plot of a field, use the `con` function of cf-plot. It requires as its one positional argument a field with only two axes which are greater than size one in order to have an effective 2D field, often a 'slice' i.e. subspace or collapse of a higher-dimensional field, it can plot.\n",
    "\n",
    "For example, for our fields above we need to reduce one axis down to achieve this, so for example we can reduce the `pressure` coordinate. Let's take a subspace at a given pressure (we take 1000 mbar, the highest in the field), and plot it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e9b510f-0acc-4e48-ad2b-cc8b4e56fa13",
   "metadata": {},
   "outputs": [],
   "source": [
    "cfp.con(temp_field.subspace(pressure=1000.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e2e7348-0df5-4c97-aab5-73e73a3bca0f",
   "metadata": {},
   "source": [
    "Let's see what the equivalent plot would be for the subspace over a different pressure, this time the lowest 1 mbar:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18589adf-83eb-4ffb-a43c-ca4a19c1a6d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "cfp.con(temp_field.subspace(pressure=1.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b19186dc-3480-47d1-a859-79008841bd58",
   "metadata": {},
   "source": [
    "### b) Customising the (contour) plot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f9f60bb-a492-447c-b5ba-d248eec9fe38",
   "metadata": {},
   "source": [
    "cf-plot provides numerous ways to customise a specific plot such as those above. We demonstrate just a small number here.\n",
    "\n",
    "The second plot from the previous sub-section, in particular, has contour lines which are so dense they obscure the world map and data pattern so let's hide those which you do by specifying `lines=False` to the `con` call:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ece996ff-6da0-42be-8d45-02de998756b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "cfp.con(temp_field.subspace(pressure=1.0), lines=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6285497e-e579-48cf-a10a-032ddf4d944f",
   "metadata": {},
   "source": [
    "Let's also do some further customisation of the plot above, as follows:\n",
    "\n",
    "- use a different map projection;\n",
    "- set a different colour map for the data;\n",
    "- move the colour bar to be vertical not horizontal;\n",
    "- add a title:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fba81794-1652-44cb-bed1-f152cc068a84",
   "metadata": {},
   "outputs": [],
   "source": [
    "cfp.mapset(proj=\"robin\") # changes the projection\n",
    "cfp.cscale(\"plasma\")  # changes the colour scale/map\n",
    "cfp.con(\n",
    "    temp_field.subspace(pressure=1.0),\n",
    "    lines=False,\n",
    "    colorbar_orientation=\"vertical\",  # tells cf-plot to make a vertical colour bar (horizontal is the default)\n",
    "    title=\"Air temperaure field\"  # adds the specified title\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "306d0813-dc38-44ff-8b38-705245be209c",
   "metadata": {},
   "source": [
    "We might want to view a smaller section of the data, say over Australia only. We can do this in two ways:\n",
    "\n",
    "1. by reducing the data further; or\n",
    "2. by specifying to cf-plot to only show a smaller window on the data.\n",
    "\n",
    "For (1), instead of a subspace on just pressure, we also suspace on X and Y i.e. longitude and latitude, to reduce those axes. We can use `cf.wi` which specifies a 'within' condition. First, define the extrema of both of these we want to view, for example these are those longitudes and latitudes which provide a view centered around Australia:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9e08f06-969d-4019-a82e-210dabe28470",
   "metadata": {},
   "outputs": [],
   "source": [
    "cfp.mapset() # resets the projection\n",
    "lonmin = 110\n",
    "lonmax = 155\n",
    "latmin = -45\n",
    "latmax = -5"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46721929-6a3a-4ad5-8388-fc183d1bd7d1",
   "metadata": {},
   "source": [
    "Now do method (1) from above, via a subspace on three axes, pressure (as before) as well as longitude and latitude (new). Note we could do each subspace via a separate call, but they can also be done together in one call like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fab363a9-53b5-4338-8572-2fea2fd01029",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Do further subspacing to reduce the latitude and longitude with cf-python\n",
    "fsub_all = temp_field.subspace(\n",
    "    pressure=1.0,\n",
    "    longitude=cf.wi(lonmin, lonmax),\n",
    "    latitude=cf.wi(latmin, latmax)\n",
    ")\n",
    "\n",
    "# Plot the above\n",
    "cfp.con(fsub_all, lines=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2620c019-4e45-4d37-a788-cf5eb76c1915",
   "metadata": {},
   "source": [
    "Method (2) can be achieved by setting the keyword arguments to the cf-plot `mapset` method like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffe229db-b678-465a-9f13-f06e0d00b120",
   "metadata": {},
   "outputs": [],
   "source": [
    "cfp.mapset(lonmin=lonmin, lonmax=lonmax, latmin=latmin, latmax=latmax)\n",
    "cfp.con(temp_field.subspace(pressure=1.0), lines=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "290bdc79-069f-49e1-a997-81413f35c398",
   "metadata": {},
   "source": [
    "Note the differences in colour shades between the two plots from the different approaches are because in approach (2) cf-plot normalises the colour map extent of the data to only account for the area shown."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6964260d-84fb-41d9-a1b9-a74e85a79c20",
   "metadata": {},
   "source": [
    "### c) Making a vector plot with basic customisation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8688ff02-0aa0-42ba-a38c-abcf6e3494ea",
   "metadata": {},
   "source": [
    "Finally let's make some vector plots. Notice our final two fields from the read-in fieldlist are in eastward and northward components, therefore can be combined into vectors from these vector components:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d493bb8a-f031-4231-90bd-fa2a08d0400a",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(wind_u_field)\n",
    "print(wind_v_field)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e58a7017-a0d3-4191-b85d-d4e72c251298",
   "metadata": {},
   "source": [
    "Again we need to subspace in pressure so that we have an effective 2D not 4D space that can be visualised as a contour plot.\n",
    "This time let's take our subspace at 500 mbar pressure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0cce64a8-2b29-4ac1-9034-f9d82e501827",
   "metadata": {},
   "outputs": [],
   "source": [
    "u = wind_u_field.subspace(pressure=500.0)\n",
    "v = wind_v_field.subspace(pressure=500.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f7ebfb0-73a9-492b-9e5e-614e26333ac6",
   "metadata": {},
   "source": [
    "To make a vector plot using cf-plot, use the `vect` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6639ea78-4d27-4c82-81c6-cfee0587dc04",
   "metadata": {},
   "outputs": [],
   "source": [
    "cfp.mapset()  # reset the mapping settings, including map projection\n",
    "cfp.vect(u=u, v=v)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd721190-98da-4770-9842-fcd006bbe3b0",
   "metadata": {},
   "source": [
    "We can mostly just see black here! This is expected and because we need to customise the vector spacing. There are clearly too many vectors shown they overwhelm the plot area, so use the `scale` and `stride` keyword arguments to `vect` to edit the size and spacing of the vector arrows respectively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72527e5d-cf24-4beb-9e19-eb90995af0c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "cfp.vect(u=u, v=v, scale=100, stride=4)  # much better - found these values after playing around with values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3df50206-c6df-4ab0-bbd0-4c09976994b8",
   "metadata": {},
   "source": [
    "Like for contour plotting, we can focus in on specific areas of the data in space via the two methods covered there. In the vector plot case, note we may need to adjust the vector spacing and size again for optimal viewing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f6b8d5c-5c14-4e08-b539-b256a65046e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "cfp.mapset(lonmin=lonmin, lonmax=lonmax, latmin=latmin, latmax=latmax) # also resets the projection to the default 'cyl'\n",
    "cfp.vect(u=u, v=v)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "330207a2-dd98-480d-b828-7ed34b73f39d",
   "metadata": {},
   "source": [
    "Trial and error for vector size and spacing gives a nicer result for the above plot with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5575b4c9-6b78-4c1f-a157-40df44f028bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "cfp.vect(u=u, v=v, scale=50, stride=1)  # better than the above!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b191c8b2-a038-4601-b046-997a4566c2f6",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72524bb7-a643-4753-ad8b-f419656883fa",
   "metadata": {},
   "source": [
    "## 5. Analysing data: applying mathematical and statistical operations and plotting trends\n",
    "\n",
    "**In this section we demonstrate how to do some data analysis including performing arithmetic and statistical calculations on the data, showing how cf-python's CF Conventions metadata awareness means that the metadata is automatically updated to account for the operations that are performed.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7378c40-8398-4e3c-9906-d37f7f036ca7",
   "metadata": {},
   "source": [
    "As well as the statistics you can calculate and explore from collapsing fields in section (3c), you can ..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59637071-edcb-40fe-970a-1d89e0209aa7",
   "metadata": {},
   "source": [
    "### a) Applying mathematics e.g. arithmetic and trigonometry on fields"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1eb2eac-2a20-4eae-ba82-1cb3504d497a",
   "metadata": {},
   "source": [
    "We will use another dataset to demonstrate this, to remind of the `read` function and fieldlist unpacking from section one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53ecf0e8-a0a5-46a2-9b50-df6ea0f2a346",
   "metadata": {},
   "outputs": [],
   "source": [
    "monthly_field = cf.read(\"../data/IPSL-CM5A-LR_r1i1p1_tas_n96_rcp45_mnth.nc\")[0]\n",
    "print(monthly_field)\n",
    "print(monthly_field.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7139b32e-c39f-4387-8bec-76e290d70f1c",
   "metadata": {},
   "source": [
    "You can perform arithmetical operations on fields using the usual operators, e.g. let's multiply the field's underlying data by 2 and subtract 10 from it to illustrate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ccd2db42-6dae-48f6-b786-2deb15ec064b",
   "metadata": {},
   "outputs": [],
   "source": [
    "double_minus_ten_monthly_field = 2 * monthly_field - 10\n",
    "print(double_minus_ten_monthly_field)\n",
    "print(double_minus_ten_monthly_field.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77db331e-2d98-4094-ab2e-646ccdfb9b3d",
   "metadata": {},
   "source": [
    "You can apply mathematical operations too via various available methods. Let's try some rounding and trigonometry as an illustration:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75c53609-a668-4f14-91f6-5b033e5af493",
   "metadata": {},
   "outputs": [],
   "source": [
    "round_monthly_field = monthly_field.round()\n",
    "print(round_monthly_field.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab014aaa-d778-4c15-abea-fcbe3d1f141c",
   "metadata": {},
   "outputs": [],
   "source": [
    "cosine_monthly_field = monthly_field.cos()\n",
    "print(cosine_monthly_field)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1dddf74-b55a-46d5-bb75-17bac0a29747",
   "metadata": {},
   "source": [
    "Note the units change appropriately with some operations, again showcasing cf-python's metadata awareness."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "561caf84-0824-4b69-8ad9-82ce28d5bfd1",
   "metadata": {},
   "source": [
    "### b) Line plotting"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e81d7d20-1e49-4183-838c-5e03993e19bc",
   "metadata": {},
   "source": [
    "Often we want to pick out statistical trends from a subspace of the data. cf-python has season-selecting and season-collapsing methods to help you determine information on a per-season basis (as well as month-selecting methods to do the same for specific months, though we don't try any of those here).\n",
    "\n",
    "Firstly, let's show a line plot of our data averaged across the whole spatial area, so we can see the overall pattern. You can produce a line plot with cf-plot using the `lineplot` function, where the argument should be a field with a 1D series (ignoring any 1D axes such as latitude and longitude in our case which after the collapse become 1D):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c15c3068-d84f-4719-9529-d856024267ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "spatial_mean_monthly_field = monthly_field.collapse(\"area: mean\")\n",
    "print(spatial_mean_monthly_field)\n",
    "\n",
    "cfp.lineplot(spatial_mean_monthly_field)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "682335b3-5eba-416c-bbd2-182f5915df7d",
   "metadata": {},
   "source": [
    "### c) Calculating seasonal trends"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7a55b54-c513-4b5a-8e9c-677c051fcf04",
   "metadata": {},
   "source": [
    "This data looks like it has trends on different time scales e.g. due to seasons. Let's pick some of those out using cf-python. We select two of the four seasons, using the `djf` and `jja` methods like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eca6c20d-c39d-42db-bc3b-8b676232623a",
   "metadata": {},
   "outputs": [],
   "source": [
    "get_djf_season = cf.djf()  # specific collapse type for the months of December, January, February\n",
    "get_jja_season = cf.jja()  # specific collapse type for the months of June, July, August"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a7a2bab-d6a6-4a3d-8793-aee95bfea6f9",
   "metadata": {},
   "source": [
    "Then we collapse on these only using the `group` keyword argument to the `collapse` method, which we call a 'grouped collapse':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7fd95f11-90d9-42f8-b459-1d672d91be81",
   "metadata": {},
   "outputs": [],
   "source": [
    "djf_season_mean = spatial_mean_monthly_field.collapse(\"T: mean\", group=get_djf_season)  # mean across DJF season\n",
    "cfp.lineplot(djf_season_mean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "098f995c-ef89-4b00-87da-8844036e6d06",
   "metadata": {},
   "outputs": [],
   "source": [
    "jja_season_mean = spatial_mean_monthly_field.collapse(\"T: mean\", group=get_jja_season)  # mean across JJA season\n",
    "cfp.lineplot(jja_season_mean)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b094573f-f690-48da-a8e9-93481ead73fb",
   "metadata": {},
   "source": [
    "### d) Plotting the seasonal trends on one (line)plot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b183d71f-437f-405e-b43d-5680ae0a18f3",
   "metadata": {},
   "source": [
    "To put those seasonal averages into context from the original data, it would be nice to plot them on top of the original line plot. We can do that using cf-plot, as an example of some more advanced cf-plot plotting capability. If you want to plot multiple aspects on one plot in this way, wrap the calls to the functions to plot such as `con` or `vect` or `lineplot` with the opening and closing functions `gopen` and `gclose`, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f834d38e-bb8d-43c3-a92b-0c68fb573b08",
   "metadata": {},
   "outputs": [],
   "source": [
    "cfp.gopen()\n",
    "# By adding the 'label' argument, we allow labels for the corresponding line on the plot legend\n",
    "cfp.lineplot(spatial_mean_monthly_field, label=\"Original monthly data (spatial mean)\")\n",
    "cfp.lineplot(djf_season_mean, label=\"Mean over the DJF months of the original spatial mean\")\n",
    "cfp.lineplot(jja_season_mean, label=\"Mean over the JJA months of the original spatial mean\")\n",
    "cfp.gclose()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b746f6b-ef82-4aa9-8d39-ca179f15db62",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e4c72ef-8631-4749-b4fd-6c79c8d6e455",
   "metadata": {},
   "source": [
    "## 6. Changing the underlying grid of data through regridding\n",
    "\n",
    "**In this section we demonstrate how to change the underlying grid of the data to another grid which could be a higher- or lower- resolution one, or a completely different grid, which is called regridding or interpolation, and indicate various options cf-python supports for doing this.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "854d7fd4-6d65-4827-b77c-4a89d42da6b0",
   "metadata": {},
   "source": [
    "### a) Getting a _source_ field ready to regrid"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b387753-9fee-4105-8758-b79f63338423",
   "metadata": {},
   "source": [
    "We read in a precipitation field and inspect it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f4bb94e-4531-4ef6-a7e7-fd46ea06b7fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "higher_res_field = cf.read(\"../data/precip_2010.nc\")[0]\n",
    "print(higher_res_field)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "387322a6-ba11-4a31-8468-c277b6ab4bf5",
   "metadata": {},
   "source": [
    "To get a feel for the data we have, let's view a basic subspace as a contour plot. Note we use the `blockfill` argument set to `True` which changes the contour plot to plot the data on a cellular basis rather than filled as smoothed contours (whether or not the contour lines are set to be shown from the `lines` argument, in this case we turn those off too):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67e544cd-44e0-48e8-8484-3fc9a6348801",
   "metadata": {},
   "outputs": [],
   "source": [
    "cfp.mapset()  # reset\n",
    "cfp.cscale(\"precip_11lev\")  # using a colour scale good for precipitation data\n",
    "cfp.con(higher_res_field[0], blockfill=True, lines=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "628e91df-76f0-485e-8288-19ee932a7fb6",
   "metadata": {},
   "source": [
    "### b) Getting the _destination_ field: another field in order to regrid the previous _onto its grid_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f5392a3-096a-4d0b-afe6-f4dcdbe29c22",
   "metadata": {},
   "source": [
    "Now we read in another precipitation field and inspect it. The key thing to note is that it is lower resolution than the previous one we read:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0dd58b04-f3e6-4cfe-890b-204bd67a3878",
   "metadata": {},
   "outputs": [],
   "source": [
    "lower_res_field = cf.read(\"../data/model_precip_DJF_means_low_res.nc\")[0]\n",
    "print(lower_res_field)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b398db6d-17c0-4940-ab85-2638b2648056",
   "metadata": {},
   "source": [
    "Again, get a feel for the data we have from a view of a basic subspace, to confirm that it is lower resolution (notice the larger cell blocks than before):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f27a6f45-6212-4547-9a92-0337fdd0a6a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "cfp.cscale(\"precip_11lev\")\n",
    "cfp.con(lower_res_field[0], blockfill=True, lines=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dde6c34c-37a9-48c9-8e02-a70913411d7a",
   "metadata": {},
   "source": [
    "### c) Performing the regrid operation from the source to the destination fields"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f799ab3-462f-4fa7-83ae-41649d65a3aa",
   "metadata": {},
   "source": [
    "Now the key step after the previous setup: regridding the first (*source*) field to the grid of the second (*destination*) field. We use the `regrids` method of cf-python to do this, where the `s` in the name stands for spherical for spherical regridding, as opposed to Cartesian regridding, also possible with cf-python.\n",
    "\n",
    "We are going to do two calls, demonstrating two different methods of interpolation, namely the `patch` and `conservative` methods. Assign these to variables so we can compare them next:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1847882c-0c17-4857-8f92-2da66c11e945",
   "metadata": {},
   "outputs": [],
   "source": [
    "patch_regridded_field = higher_res_field.regrids(lower_res_field, method=\"patch\")\n",
    "conservative_regridded_field = higher_res_field.regrids(lower_res_field, method=\"conservative\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6828e28-d06c-47cf-8bd9-8c7c868e86bf",
   "metadata": {},
   "source": [
    "These two methods do not give the same results!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bbb865cc-415c-4e8f-80e9-d2f50e21beaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "patch_regridded_field.equals(conservative_regridded_field)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72802b45-73bc-4b3f-adc8-7eda851f01c1",
   "metadata": {},
   "source": [
    "Recall how to do field arithmetic. We use subtraction to indicate the difference between the two regridded outcomes which differ only by the interpolation method used to regrid:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1db1455e-b266-4620-a1c4-ec746e03dbec",
   "metadata": {},
   "outputs": [],
   "source": [
    "difference_field = patch_regridded_field - conservative_regridded_field"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1286c96e-02f2-406e-a70d-914ae00389b5",
   "metadata": {},
   "source": [
    "### d) Finally, some more advanced cf-plot plotting to compare the source, destination, and regridded results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "182b8f55-78ad-41c2-ac59-a775f42750ed",
   "metadata": {},
   "source": [
    "As with the previous section, let's end by doing more advanced cf-plot plotting to show all of the relevant fields to help us to understand what the regridding did.\n",
    "\n",
    "Again we are using `gopen` and `gclose` wrapped around our calls to plot. This time we use `gpos` function calls which tell cf-plot exactly where we want to place each plot on the overall canvas. In this case, from our `x` and `y` extrema inputs `xmin`, `xmax`, `ymin` and `ymax`, we are building an effective 4 x 4 grid to place the four contour plots we will show.\n",
    "\n",
    "There is a lot to take in here, but the main thing is to understand that we use `con` to generate four contour plots corresponding to different stages of our regridding, telling cf-plot to place them at one of the four places in a 4 x 4 grid, with some customisation of colour scales and colour bar levels.\n",
    "\n",
    "Note here we are taking the [0] index subspace of all of the fields to plot, but we can use any other subspace of the data to view instead:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2bc9fa76-96e6-4ae3-ac1d-95db3d76c95f",
   "metadata": {},
   "outputs": [],
   "source": [
    "cfp.cscale(\"viridis\")  # reset cmap to perceptually uniform scale\n",
    "cfp.gopen(rows=2, columns=2, figsize=[7, 15])  #, bottom=0.1, top=0.85)\n",
    "\n",
    "# Configure first three plots\n",
    "cfp.mapset()  # reset from previous plots in the notebook - standalone don't need this\n",
    "cfp.levs(min=0, max=500, step=50)\n",
    "\n",
    "# First three plots\n",
    "cfp.gpos(xmin=0.1, xmax=0.5, ymin=0.55, ymax=1.0)\n",
    "cfp.cscale(\"precip_11lev\")\n",
    "cfp.con(\n",
    "    higher_res_field[0], blockfill=True, lines=False,\n",
    "    title=\"Precipitation field\\nbefore regridding\",\n",
    ")\n",
    "cfp.gpos(xmin=0.55, xmax=0.95, ymin=0.55, ymax=1.0)\n",
    "cfp.con(\n",
    "    patch_regridded_field[0], blockfill=True, lines=False,\n",
    "    title=\"...and after regridding with\\nthe patch recovery method\",\n",
    ")\n",
    "cfp.gpos(xmin=0.1, xmax=0.5, ymin=0.1, ymax=0.55)\n",
    "cfp.con(\n",
    "    conservative_regridded_field[0], blockfill=True, lines=False,\n",
    "    title=\"...and after regridding with\\nthe conservative method\",\n",
    ")\n",
    "cfp.gpos(xmin=0.55, xmax=0.95, ymin=0.1, ymax=0.55)\n",
    "\n",
    "# This final plot is showing the diff so is a bit different, apply some cf-plot plotting configuration\n",
    "cfp.levs()\n",
    "cfp.cscale(\"BlueDarkRed18\")  # is a diff so use a diverging colour map\n",
    "cfp.con(\n",
    "    difference_field[0], blockfill=True, lines=False,\n",
    "    title=\"Difference between the two\\nregridding methods is:\",\n",
    ")\n",
    "cfp.gclose()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f674f64e-b20e-4d7e-96b4-fac2c719a9ad",
   "metadata": {},
   "source": [
    "More generally, using regridding with cf-python you can convert from one grid to another very different grid, e.g. from a tripolar to a regular or rotated latitude-longitude grid or vice-versa, including to/from unstructured grids (UGRID grids)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e48ef2e-68bd-434d-99cb-a6c972f24132",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07c4f4f5-296c-42f1-9885-214577dccab8",
   "metadata": {},
   "source": [
    "## Conclusion and recap of learning objectives\n",
    "\n",
    "The NCAS CF Data Tools are a suite of Python libraries which are designed to facilitate working with data for research in the earth sciences and aligned domains. We learnt today about the cf-python (https://ncas-cms.github.io/cf-python/) and cf-plot (https://ncas-cms.github.io/cf-plot/build/). The 'cf' in the names of the NCAS CF Data Tools corresponds to the CF Conventions, a metadata standard becoming the de-facto convention across geoscience to cover the description of data so that sharing and intercomparison is simpler.\n",
    "\n",
    "Our **learning aim** was to be able to use the NCAS CF Data Tools Python libraries, namely cf-python and cf-plot to process, analyse and visualise netCDF and PP datasets, whilst appreciating the context and 'unique selling point' of the libraries as being built to use the CF Conventions, a metadata standard for earth science data, to make it simpler to do what you want to do with the datasets, by working on top of a Data Model for CF.\n",
    "\n",
    "For our **learning objectives**, we practiced using cf-python and cf-plot to:\n",
    "\n",
    "* read dataset(s) and view the (meta)data at different detail levels;\n",
    "* edit the (meta)data and write out the edited version to file;\n",
    "* reduce datasets by subspacing and collapsing;\n",
    "* visualise datasets as contour and vector plots;\n",
    "* analyse data: applying mathematical and statistical operations and plotting trends;\n",
    "* change the underlying grid of data through regridding."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bcb9631-0732-43dd-9fe4-23f891f0d10c",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2e0cfb4-8203-48d7-8029-7bfda9ce5be9",
   "metadata": {},
   "source": [
    "## Where to find more information and resources on the NCAS CF Data Tools\n",
    "\n",
    "Here are some links relating to the NCAS CF Data Tools and this training.\n",
    "\n",
    "* This training, with further material, is hosted online and there are instructions for setting up the environment so you can work through it in your own time: https://github.com/NCAS-CMS/cf-tools-training.\n",
    "* The cf-python documentation lives at https://ncas-cms.github.io/cf-python/.\n",
    "* The cf-python code lives on GitHub at https://github.com/NCAS-CMS/cf-python. There is an Issue Tracker to report queries or questions at https://github.com/NCAS-CMS/cf-python/issues.\n",
    "* The cf-plot documentation lives at https://ncas-cms.github.io/cf-plot/build/.\n",
    "* The cf-plot code lives on GitHub at https://github.com/NCAS-CMS/cf-plot. There is an Issue Tracker to report queries or questions at https://github.com/NCAS-CMS/cf-plot/issues.\n",
    "* There is a technical presentation about the NCAS CF Data Tools avaialble from https://hps.vi4io.org/_media/events/2020/summer-school-cfnetcdf.pdf.\n",
    "* The website of the CF Conventions can be found at https://cfconventions.org/.\n",
    "* The landing page for training into the CF Conventions is found here within the website above: https://cfconventions.org/Training/.\n",
    "\n",
    "If you have any queries after this course, please either use the Issue Trackers linked above or you can email me at: sadie.bartholomew@ncas.ac.uk."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08fa89e2-6c21-4118-9cf9-21958bdfa00e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "***"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 + Jaspy",
   "language": "python",
   "name": "jaspy"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}