PythonNotebooks/indexFileNavigation/index_file_navigation_fileUpdate.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> ABSTRACT </h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All CMEMS in situ data products can be found and downloaded after [registration](http://marine.copernicus.eu/services-portfolio/register-now/) via [CMEMS catalogue](http://marine.copernicus.eu/services-portfolio/access-to-products/).\n",
    "\n",
    "Such channel is advisable just for sporadic netCDF donwloading because when operational, interaction with the web user interface is not practical. In this context though, the use of scripts for ftp file transference is is a much more advisable approach.\n",
    "\n",
    "As long as every line of such files contains information about the netCDFs contained within the different directories [see at tips why](https://github.com/CopernicusMarineInsitu/INSTACTraining/blob/master/tips/README.md), it is posible for users to loop over its lines to download only those that matches a number of specifications such as spatial coverage, time coverage, provider, data_mode, parameters or file_name related (region, data type, TS or PF, platform code, or/and platform category, timestamp)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>PREREQUISITES</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [credentias](http://marine.copernicus.eu/services-portfolio/register-now/)\n",
    "- aimed [in situ product name](http://cmems-resources.cls.fr/documents/PUM/CMEMS-INS-PUM-013.pdf)\n",
    "- aimed [hosting distribution unit](https://github.com/CopernicusMarineInsitu/INSTACTraining/blob/master/tips/README.md)\n",
    "- aimed [index file](https://github.com/CopernicusMarineInsitu/INSTACTraining/blob/master/tips/README.md)\n",
    "\n",
    "i.e:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "user = ''  #type CMEMS user name within colons\n",
    "password = ''#type CMEMS password within colons\n",
    "product_name = 'INSITU_BAL_NRT_OBSERVATIONS_013_032' #type aimed CMEMS in situ product \n",
    "distribution_unit = 'cmems.smhi.se' #type aimed hosting institution\n",
    "index_file = 'index_latest.txt' #type aimed index file name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>DOWNLOAD</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Index file download"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import ftplib "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'221 Goodbye.'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ftp=ftplib.FTP(distribution_unit,user,password) \n",
    "ftp.cwd(\"Core\")\n",
    "ftp.cwd(product_name) \n",
    "remote_filename= index_file\n",
    "local_filename = remote_filename\n",
    "local_file = open(local_filename, 'wb')\n",
    "ftp.retrbinary('RETR ' + remote_filename, local_file.write)\n",
    "local_file.close()\n",
    "ftp.quit()\n",
    "#ready when 221 Goodbye.!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>QUICK VIEW</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reading a random line of the index file to know more about the information it contains."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from random import randint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "index = np.genfromtxt(index_file, skip_header=6, unpack=False, delimiter=',', dtype=None,\n",
    "           names=['catalog_id', 'file_name', 'geospatial_lat_min', 'geospatial_lat_max',\n",
    "                     'geospatial_lon_min', 'geospatial_lon_max',\n",
    "                     'time_coverage_start', 'time_coverage_end', \n",
    "                     'provider', 'date_update', 'data_mode', 'parameters'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dataset = randint(0,len(index)) #ramdom line of the index file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <style  type=\"text/css\" >\n",
       "        \n",
       "        \n",
       "        </style>\n",
       "\n",
       "        <table id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4b\" None>\n",
       "        \n",
       "\n",
       "        <thead>\n",
       "            \n",
       "            <tr>\n",
       "                \n",
       "                \n",
       "                <th class=\"blank level0\" >\n",
       "                  \n",
       "                \n",
       "                \n",
       "                \n",
       "                <th class=\"col_heading level0 col0\" colspan=1>\n",
       "                  939\n",
       "                \n",
       "                \n",
       "            </tr>\n",
       "            \n",
       "        </thead>\n",
       "        <tbody>\n",
       "            \n",
       "            <tr>\n",
       "                \n",
       "                \n",
       "                <th id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4b\"\n",
       "                 class=\"row_heading level0 row0\" rowspan=1>\n",
       "                    catalog_id\n",
       "                \n",
       "                \n",
       "                \n",
       "                <td id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4brow0_col0\"\n",
       "                 class=\"data row0 col0\" >\n",
       "                    COP-BO-01\n",
       "                \n",
       "                \n",
       "            </tr>\n",
       "            \n",
       "            <tr>\n",
       "                \n",
       "                \n",
       "                <th id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4b\"\n",
       "                 class=\"row_heading level0 row1\" rowspan=1>\n",
       "                    file_name\n",
       "                \n",
       "                \n",
       "                \n",
       "                <td id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4brow1_col0\"\n",
       "                 class=\"data row1 col0\" >\n",
       "                    <a href=ftp://cmems.smhi.se/Core/INSITU_BAL_NRT_OBSERVATIONS_013_032/latest/20170915/BO_LATEST_TS_FB_StenaHollandica_20170915.nc>ftp://cmems.smhi.se/Core/INSITU_BAL_NRT_OBSERVATIONS_013_032/latest/20170915/BO_LATEST_TS_FB_StenaHollandica_20170915.nc</a>\n",
       "                \n",
       "                \n",
       "            </tr>\n",
       "            \n",
       "            <tr>\n",
       "                \n",
       "                \n",
       "                <th id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4b\"\n",
       "                 class=\"row_heading level0 row2\" rowspan=1>\n",
       "                    geospatial_lat_min\n",
       "                \n",
       "                \n",
       "                \n",
       "                <td id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4brow2_col0\"\n",
       "                 class=\"data row2 col0\" >\n",
       "                    51.8977\n",
       "                \n",
       "                \n",
       "            </tr>\n",
       "            \n",
       "            <tr>\n",
       "                \n",
       "                \n",
       "                <th id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4b\"\n",
       "                 class=\"row_heading level0 row3\" rowspan=1>\n",
       "                    geospatial_lat_max\n",
       "                \n",
       "                \n",
       "                \n",
       "                <td id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4brow3_col0\"\n",
       "                 class=\"data row3 col0\" >\n",
       "                    52.0519\n",
       "                \n",
       "                \n",
       "            </tr>\n",
       "            \n",
       "            <tr>\n",
       "                \n",
       "                \n",
       "                <th id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4b\"\n",
       "                 class=\"row_heading level0 row4\" rowspan=1>\n",
       "                    geospatial_lon_min\n",
       "                \n",
       "                \n",
       "                \n",
       "                <td id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4brow4_col0\"\n",
       "                 class=\"data row4 col0\" >\n",
       "                    1.2520\n",
       "                \n",
       "                \n",
       "            </tr>\n",
       "            \n",
       "            <tr>\n",
       "                \n",
       "                \n",
       "                <th id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4b\"\n",
       "                 class=\"row_heading level0 row5\" rowspan=1>\n",
       "                    geospatial_lon_max\n",
       "                \n",
       "                \n",
       "                \n",
       "                <td id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4brow5_col0\"\n",
       "                 class=\"data row5 col0\" >\n",
       "                    4.1337\n",
       "                \n",
       "                \n",
       "            </tr>\n",
       "            \n",
       "            <tr>\n",
       "                \n",
       "                \n",
       "                <th id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4b\"\n",
       "                 class=\"row_heading level0 row6\" rowspan=1>\n",
       "                    time_coverage_start\n",
       "                \n",
       "                \n",
       "                \n",
       "                <td id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4brow6_col0\"\n",
       "                 class=\"data row6 col0\" >\n",
       "                    2017-09-15T02:00:00Z\n",
       "                \n",
       "                \n",
       "            </tr>\n",
       "            \n",
       "            <tr>\n",
       "                \n",
       "                \n",
       "                <th id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4b\"\n",
       "                 class=\"row_heading level0 row7\" rowspan=1>\n",
       "                    time_coverage_end\n",
       "                \n",
       "                \n",
       "                \n",
       "                <td id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4brow7_col0\"\n",
       "                 class=\"data row7 col0\" >\n",
       "                    2017-09-15T23:50:00Z\n",
       "                \n",
       "                \n",
       "            </tr>\n",
       "            \n",
       "            <tr>\n",
       "                \n",
       "                \n",
       "                <th id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4b\"\n",
       "                 class=\"row_heading level0 row8\" rowspan=1>\n",
       "                    provider\n",
       "                \n",
       "                \n",
       "                \n",
       "                <td id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4brow8_col0\"\n",
       "                 class=\"data row8 col0\" >\n",
       "                    STE\n",
       "                \n",
       "                \n",
       "            </tr>\n",
       "            \n",
       "            <tr>\n",
       "                \n",
       "                \n",
       "                <th id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4b\"\n",
       "                 class=\"row_heading level0 row9\" rowspan=1>\n",
       "                    date_update\n",
       "                \n",
       "                \n",
       "                \n",
       "                <td id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4brow9_col0\"\n",
       "                 class=\"data row9 col0\" >\n",
       "                    2017-09-17T13:01:08Z\n",
       "                \n",
       "                \n",
       "            </tr>\n",
       "            \n",
       "            <tr>\n",
       "                \n",
       "                \n",
       "                <th id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4b\"\n",
       "                 class=\"row_heading level0 row10\" rowspan=1>\n",
       "                    data_mode\n",
       "                \n",
       "                \n",
       "                \n",
       "                <td id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4brow10_col0\"\n",
       "                 class=\"data row10 col0\" >\n",
       "                    R\n",
       "                \n",
       "                \n",
       "            </tr>\n",
       "            \n",
       "            <tr>\n",
       "                \n",
       "                \n",
       "                <th id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4b\"\n",
       "                 class=\"row_heading level0 row11\" rowspan=1>\n",
       "                    parameters\n",
       "                \n",
       "                \n",
       "                \n",
       "                <td id=\"T_8d5d9d5a_adc2_11e7_8c00_64006a8d5c4brow11_col0\"\n",
       "                 class=\"data row11 col0\" >\n",
       "                    TEMP WDIR HCSP HCDT WSPD DEPH\n",
       "                \n",
       "                \n",
       "            </tr>\n",
       "            \n",
       "        </tbody>\n",
       "        </table>\n",
       "        "
      ],
      "text/plain": [
       "<pandas.formats.style.Styler at 0x7f4668d02c90>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "values = [index[dataset]['catalog_id'], '<a href='+index[dataset]['file_name']+'>'+index[dataset]['file_name']+'</a>', index[dataset]['geospatial_lat_min'], index[dataset]['geospatial_lat_max'],\n",
    "                 index[dataset]['geospatial_lon_min'], index[dataset]['geospatial_lon_max'], index[dataset]['time_coverage_start'],\n",
    "                 index[dataset]['time_coverage_end'], index[dataset]['provider'], index[dataset]['date_update'], index[dataset]['data_mode'],\n",
    "                 index[dataset]['parameters']]\n",
    "headers = ['catalog_id', 'file_name', 'geospatial_lat_min', 'geospatial_lat_max',\n",
    "                     'geospatial_lon_min', 'geospatial_lon_max',\n",
    "                     'time_coverage_start', 'time_coverage_end', \n",
    "                     'provider', 'date_update', 'data_mode', 'parameters']\n",
    "df = pd.DataFrame(values, index=headers, columns=[dataset])\n",
    "df.style"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>FILTERING CRITERIA</h3>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regarding the above glimpse, it is posible to filter by 12 criteria. As example we will setup next a filter to only download those files that has been updated in the last X hours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#packages\n",
    "from datetime import date\n",
    "import datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    1. Number of minutes "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "time_lapse = 10 #all files updated within the last 10 hours will be downloaded\n",
    "end_date = datetime.datetime.today()\n",
    "ini_date = end_date - datetime.timedelta(hours=time_lapse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    2. netCDF filtering/selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total: 333\n"
     ]
    }
   ],
   "source": [
    "#read file lines (iterate over them)\n",
    "selected_netCDFs = [];\n",
    "for netCDF in index: \n",
    "    file_name = netCDF['file_name'].decode('utf-8')\n",
    "    last_idx_slash = file_name.rfind('/')\n",
    "    ncdf_file_name = file_name[last_idx_slash+1:]\n",
    "    date_update = netCDF['date_update'].decode('utf-8')\n",
    "    date_format = \"%Y-%m-%dT%H:%M:%SZ\" \n",
    "    file_date = datetime.datetime.strptime(date_update, date_format)\n",
    "    #set up a selection criteria: i.e dates within the last 30 min\n",
    "    if ini_date < file_date < end_date:\n",
    "        selected_netCDFs.append(file_name)\n",
    "print(\"total: \" +str(len(selected_netCDFs)))   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> SELECTION DOWNLOAD </h3>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for nc in selected_netCDFs:\n",
    "    last_idx_slash = nc.rfind('/')\n",
    "    ncdf_file_name = nc[last_idx_slash+1:]\n",
    "    folders = nc.split('/')[3:len(file_name.split('/'))-1]\n",
    "    host = nc.split('/')[2] #distribution unit\n",
    "    \n",
    "    ftp=ftplib.FTP(host,user,password) \n",
    "    for folder in folders:\n",
    "        ftp.cwd(folder)\n",
    "    \n",
    "    local_file = open(ncdf_file_name, 'wb')\n",
    "    ftp.retrbinary('RETR '+ncdf_file_name, local_file.write)\n",
    "    local_file.close()\n",
    "    ftp.quit()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}