docs: update product chart for atmos products (#80)

feat: add `s3_filesystem` using `s3fs` to `utilities`
feat: add `'s3'` and `'opendap'` endpoint options to CMR query functions

README.rst

@@ -149,21 +149,24 @@ The granule boundaries limit the size of each ATL03 file and simplify the format
 Dependencies
 ############
 
- - `numpy: Scientific Computing Tools For Python <https://numpy.org>`_
- - `scipy: Scientific Tools for Python <https://docs.scipy.org/doc/>`_
- - `mpi4py: Python bindings of the Message Passing Interface (MPI) <https://mpi4py.readthedocs.io/en/stable/>`_
- - `h5py: Python interface for Hierarchal Data Format 5 (HDF5) <http://h5py.org>`_
- - `netCDF4: Python interface to the netCDF C library <https://unidata.github.io/netcdf4-python/>`_
- - `zarr: Chunked, compressed, N-dimensional arrays in Python <https://github.com/zarr-developers/zarr-python>`_
- - `pandas: Python Data Analysis Library <https://pandas.pydata.org/>`_
- - `scikit-learn: Machine Learning in Python <https://scikit-learn.org/stable/index.html>`_
- - `shapely: PostGIS-ish operations outside a database context for Python <http://toblerity.org/shapely/index.html>`_
- - `fiona: Python wrapper for vector data access functions from the OGR library <https://fiona.readthedocs.io/en/latest/manual.html>`_
- - `lxml: processing XML and HTML in Python <https://pypi.python.org/pypi/lxml>`_
- - `paramiko: Native Python SSHv2 protocol library <http://www.paramiko.org/>`_
- - `scp: scp module for paramiko <https://github.com/jbardin/scp.py>`_
- - `future: Compatibility layer between Python 2 and Python 3 <http://python-future.org/>`_
- - `yapc: Python version of the NASA Goddard Space Flight Center YAPC Program <https://github.com/tsutterley/yapc>`_
+- `numpy: Scientific Computing Tools For Python <https://numpy.org>`_
+- `scipy: Scientific Tools for Python <https://docs.scipy.org/doc//>`_
+- `mpi4py: Python bindings of the Message Passing Interface (MPI) <https://mpi4py.readthedocs.io/en/stable/>`_
+- `h5py: Python interface for Hierarchal Data Format 5 (HDF5) <http://h5py.org>`_
+- `netCDF4: Python interface to the netCDF C library <https://unidata.github.io/netcdf4-python/>`_
+- `zarr: Chunked, compressed, N-dimensional arrays in Python <https://github.com/zarr-developers/zarr-python>`_
+- `pandas: Python Data Analysis Library <https://pandas.pydata.org/>`_
+- `scikit-learn: Machine Learning in Python <https://scikit-learn.org/stable/index.html>`_
+- `dateutil: powerful extensions to datetime <https://dateutil.readthedocs.io/en/stable/>`_
+- `shapely: PostGIS-ish operations outside a database context for Python <http://toblerity.org/shapely/index.html>`_
+- `fiona: Python wrapper for vector data access functions from the OGR library <https://fiona.readthedocs.io/en/latest/manual.html>`_
+- `future: Compatibility layer between Python 2 and Python 3 <http://python-future.org/>`_
+- `boto3: Amazon Web Services (AWS) SDK for Python <https://boto3.amazonaws.com/v1/documentation/api/latest/index.html>`_
+- `s3fs: Pythonic file interface to S3 built on top of botocore <https://s3fs.readthedocs.io/en/latest/>`_
+- `lxml: processing XML and HTML in Python <https://pypi.python.org/pypi/lxml>`_
+- `paramiko: Native Python SSHv2 protocol library <http://www.paramiko.org>`_
+- `scp: scp module for paramiko <https://github.com/jbardin/scp.py>`_
+- `yapc: Python version of the NASA Goddard Space Flight Center YAPC Program <https://github.com/tsutterley/yapc>`_
 
 Download
 ########

doc/source/_assets/ASAS_product_chart.sh

@@ -0,0 +1,4 @@
+#!/bin/bash
+pdflatex ASAS_product_chart.tex
+convert -density 300 ASAS_product_chart.pdf -quality 100 ASAS_product_chart.png
+rm ASAS_product_chart.aux ASAS_product_chart.log ASAS_product_chart.pdf

doc/source/_assets/ASAS_product_chart.tex

@@ -0,0 +1,142 @@
+\documentclass[border=10pt]{standalone}
+\usepackage[default]{lato}
+\usepackage[T1]{fontenc}
+\usepackage{verbatim}
+\usepackage{xcolor}
+\usepackage{tikz}
+\usetikzlibrary{shapes,arrows,arrows.meta,mindmap,backgrounds,calc,positioning}
+
+\tikzset{
+	>={Latex[width=2mm,length=2mm]},
+	% Specifications for style of nodes:
+	base/.style = {rectangle, rounded corners, 
+	minimum width=2.5cm, minimum height=1cm,
+	draw=black, line width=0.3mm,
+	text centered, font=\small},
+	l0/.style ={rectangle,
+	minimum width=2.5cm, minimum height=2cm,
+	draw=gray50, line width=1mm,
+	text centered, font=\small},
+	l1/.style ={rectangle,
+	minimum width=2.5cm, minimum height=2cm,
+	draw=darkorange, line width=1mm,
+	text centered, font=\small},
+	l2/.style ={rectangle,
+	minimum width=2.5cm, minimum height=2cm,
+	draw=darkorchid, line width=1mm,
+	text centered, font=\small},
+	l3a/.style ={rectangle,
+	minimum width=2.5cm, minimum height=2cm,
+	draw=mediumseagreen, line width=1mm,
+	text centered, font=\small},
+	l3b/.style ={rectangle,
+	minimum width=2.5cm, minimum height=2cm,
+	draw=dodgerblue, line width=1mm,
+	text centered, font=\small},
+	ext/.style ={rectangle,
+	minimum width=1cm, minimum height=1cm,
+	draw=red, line width=1mm,
+	text centered, font=\small},
+}
+
+\definecolor{darkorange}{RGB}{255,140,0}
+\definecolor{mediumseagreen}{RGB}{60, 179, 113}
+\definecolor{darkorchid}{RGB}{153, 50, 204}
+\definecolor{dodgerblue}{RGB}{30, 144, 255}
+\definecolor{gray50}{RGB}{128,128,128}
+
+\begin{document}
+\resizebox{\textwidth}{!}{
+	\begin{tikzpicture}[node distance=2cm, every node/.style={fill=white, font=\small\sffamily}, align=center]
+	\linespread{1}
+		% draw blocks
+		\node (atl00)[l0]{\textbf{ATL00}\\Telemetered\\Data};
+		\node (atl01)[l1, below=0.5cm of atl00]{\textbf{ATL01}\\Reformatted\\Telemetry};
+		\node (atl02)[l1, right of=atl01, xshift=1cm]{\textbf{ATL02}\\Science Unit\\Converted\\Telemetry};
+		\node (ppd)[ext, below of=atl02, xshift=-1cm]{\textbf{PPD}};
+		\node (pod)[ext, below of=atl02, xshift=1cm]{\textbf{POD}};
+		% Level 2 products
+		\node (atl03)[l2, below of=ppd, xshift=1cm]{\textbf{ATL03}\\Geolocated\\Photons};
+		\node (atl04)[l2, right of=atl02, xshift=3.5cm]{\textbf{ATL04}\\Normalized\\Backscatter\\Profiles};
+		% connection coordinates for single arrow out of ATL03
+		\node(A)[shape=coordinate, below=0.5cm of atl03] {};
+		\node (B)[shape=coordinate, left=1.375cm of A] {};
+		\node (C)[shape=coordinate, left=2.75cm of B] {};
+		\node (D)[shape=coordinate, left=2.75cm of C] {};
+		\node (E)[shape=coordinate, right=1.375cm of A] {};
+		\node (F)[shape=coordinate, right=2.75cm of E] {};
+		\node (G)[shape=coordinate, right=2.75cm of F] {};
+		\node (H)[shape=coordinate, right=2.75cm of G] {};
+		% Level 3a products
+		\node (atl06)[l3a, below=0.5cm of D]{\textbf{ATL06}\\Land Ice\\Height};
+		\node (atl07)[l3a, below=0.5cm of C]{\textbf{ATL07}\\Sea Ice\\Height};
+		\node (atl10)[l3a, below=0.5cm of B]{\textbf{ATL10}\\Sea Ice\\Freeboard};
+		\node (atl08)[l3a, below=0.5cm of E]{\textbf{ATL08}\\Land/Vegetation\\Height};
+		\node (atl12)[l3a, below=0.5cm of F]{\textbf{ATL12}\\Ocean Surface\\Height};
+		\node (atl13)[l3a, below=0.5cm of G]{\textbf{ATL13}\\Inland Water\\Height};
+		\node (atl24)[l3a, dashed, below=0.5cm of H]{\textbf{ATL24}\\Bathymetric\\Elevation};
+		\node (atl09)[l3a, right of=atl03, xshift=3.5cm]{\textbf{ATL09}\\Calibrated\\Backscatter\\Profiles};
+		% connection coordinates for single arrow out of ATL09
+		\node (I)[shape=coordinate, below=0.5cm of atl09] {};
+		\node (J)[shape=coordinate, right=5.5cm of atl09] {};
+		% Level 3b products
+		\node (atl11)[l3b, below=0.5cm of atl06]{\textbf{ATL11}\\Land Ice Height\\Time Series};
+		\node (atl14)[l3b, below=0.5cm of atl11, xshift=-3cm]{\textbf{ATL14}\\Gridded Land\\ Ice Height};
+		\node (atl15)[l3b, below=0.5cm of atl11, xshift=0cm]{\textbf{ATL15}\\Gridded Land\\ Ice Height Change};
+		\node (atl20)[l3b, below=0.5cm of atl10, xshift=-1.5cm]{\textbf{ATL20}\\Gridded Sea\\ Ice Freeboard};
+		\node (atl21)[l3b, below=0.5cm of atl20, xshift=0cm]{\textbf{ATL21}\\Gridded Polar Sea\\ Surface Height};
+		\node (atl18)[l3b, dashed, below=0.5cm of atl08, xshift=0cm]{\textbf{ATL18}\\Gridded Land\\ and Vegetation\\Height};
+		\node (atl19)[l3b, below=0.5cm of atl12]{\textbf{ATL19}\\Gridded Ocean\\Topography};
+		\node (atl23)[l3b, dashed, below=0.5cm of atl19]{\textbf{ATL23}\\3-Month Gridded\\Ocean Topography};
+		\node (atl22)[l3b, below=0.5cm of atl13]{\textbf{ATL22}\\Mean Inland\\Water Height};
+		\node (atl16)[l3b, below=4.65cm of J]{\textbf{ATL16}\\Weekly Gridded\\Atmosphere};
+		\node (atl17)[l3b, below=0.5cm of atl16]{\textbf{ATL17}\\Monthly Gridded\\Atmosphere};
+		% Legend
+		\node (legend)[right=16cm of atl00, yshift=1cm, minimum height=0cm, minimum width=2.75cm]{\textbf{Legend}};
+		\node (level0)[l0, below=-0.1cm of legend, minimum height=0.75cm, minimum width=2.75cm]{\textbf{Level 0}};
+		\node (level1)[l1, below=0.1cm of level0, minimum height=0.75cm, minimum width=2.75cm]{\textbf{Level 1}};
+		\node (level2)[l2, below=0.1cm of level1, minimum height=0.75cm, minimum width=2.75cm]{\textbf{Level 2}};
+		\node (level3a)[l3a, below=0.1cm of level2, minimum height=0.75cm, minimum width=2.75cm]{\textbf{Level 3a}};
+		\node (level3b)[l3b, below=0.1cm of level3a, minimum height=0.75cm, minimum width=2.75cm]{\textbf{Level 3b}};
+		\node (extern)[ext, below=0.1cm of level3b, minimum height=0.75cm, minimum width=2.75cm]{\textbf{External}};
+		% draw arrows between nodes
+		\draw[line width=0.3mm,-Latex](atl00) -- (atl01);
+		\draw[line width=0.3mm,-Latex](atl01) -- (atl02);
+		\draw[line width=0.3mm,-Latex](atl02) -- (ppd);
+		\draw[line width=0.3mm,-Latex](atl02) -- (pod);
+		\draw[line width=0.3mm,-Latex](ppd) -- (atl03);
+		\draw[line width=0.3mm,-Latex](pod) -- (atl03);
+		\draw[line width=0.3mm,-Latex](atl02) -- (atl03);
+		\draw[line width=0.3mm,-Latex](atl02) -- (atl04);
+		\draw[line width=0.3mm,-Latex](atl04) -- (atl09);
+		\draw[line width=0.3mm,-](atl03.south) -- (A.north);
+		\draw[line width=0.3mm,-](atl09.south) -- (I.north);
+		\draw[line width=0.3mm,-](atl09.east) -- (J.west);
+		\draw[line width=0.3mm,-](A.west) -- (D.east);
+		\draw[line width=0.3mm,-](A.east) -- (H.west);
+		\draw[line width=0.3mm,-Latex](D.south) -- (atl06.north);
+		\draw[line width=0.3mm,-Latex](C.south) -- (atl07.north);
+		\draw[line width=0.3mm,-Latex](B.south) -- (atl10.north);
+		\draw[line width=0.3mm,-Latex](E.south) -- (atl08.north);
+		\draw[line width=0.3mm,-Latex](F.south) -- (atl12.north);
+		\draw[line width=0.3mm,-Latex](G.south) -- (atl13.north);
+		\draw[line width=0.3mm,-Latex](H.south) -- (atl24.north);
+		\draw[line width=0.3mm,-Latex](J.south) -- (atl16.north);
+		\draw[line width=0.3mm,-Latex](atl06) -- (atl11);
+		\draw[line width=0.3mm,-Latex](atl11) -- (atl14);
+		\draw[line width=0.3mm,-Latex](atl11) -- (atl15);
+		\draw[line width=0.3mm,-Latex](atl07) -- (atl20);
+		\draw[line width=0.3mm,-Latex](atl10) -- (atl20);
+		\draw[line width=0.3mm,-Latex](atl20) -- (atl21);
+		\draw[line width=0.3mm,-Latex](atl08) -- (atl18);
+		\draw[line width=0.3mm,-Latex](atl12) -- (atl19);
+		\draw[line width=0.3mm,-Latex](atl19) -- (atl23);
+		\draw[line width=0.3mm,-Latex](atl13) -- (atl22);
+		\draw[line width=0.3mm,-Latex](atl16) -- (atl17);
+	\end{tikzpicture}
+}
+\end{document}
+%%% Local Variables: ***
+%%% mode: latex ***
+%%% TeX-master: "ASAS_product_chart.tex" ***
+%%% End: ***

doc/source/api_reference/utilities.rst

@@ -27,8 +27,18 @@ General Methods
 
 .. autofunction:: icesat2_toolkit.utilities.url_split
 
+.. autofunction:: icesat2_toolkit.utilities.get_unix_time
+
+.. autofunction:: icesat2_toolkit.utilities.even
+
+.. autofunction:: icesat2_toolkit.utilities.ceil
+
+.. autofunction:: icesat2_toolkit.utilities.copy
+
 .. autofunction:: icesat2_toolkit.utilities.s3_client
 
+.. autofunction:: icesat2_toolkit.utilities.s3_filesystem
+
 .. autofunction:: icesat2_toolkit.utilities.s3_bucket
 
 .. autofunction:: icesat2_toolkit.utilities.s3_key
@@ -37,14 +47,6 @@ General Methods
 
 .. autofunction:: icesat2_toolkit.utilities.generate_presigned_url
 
-.. autofunction:: icesat2_toolkit.utilities.get_unix_time
-
-.. autofunction:: icesat2_toolkit.utilities.even
-
-.. autofunction:: icesat2_toolkit.utilities.ceil
-
-.. autofunction:: icesat2_toolkit.utilities.copy
-
 .. autofunction:: icesat2_toolkit.utilities.check_ftp_connection
 
 .. autofunction:: icesat2_toolkit.utilities.ftp_list

doc/source/getting_started/Citations.rst

@@ -54,6 +54,7 @@ This software is also dependent on other commonly used Python packages:
 - `fiona: Python wrapper for vector data access functions from the OGR library <https://fiona.readthedocs.io/en/latest/manual.html>`_
 - `future: Compatibility layer between Python 2 and Python 3 <http://python-future.org/>`_
 - `boto3: Amazon Web Services (AWS) SDK for Python <https://boto3.amazonaws.com/v1/documentation/api/latest/index.html>`_
+- `s3fs: Pythonic file interface to S3 built on top of botocore <https://s3fs.readthedocs.io/en/latest/>`_
 - `lxml: processing XML and HTML in Python <https://pypi.python.org/pypi/lxml>`_
 - `paramiko: Native Python SSHv2 protocol library <http://www.paramiko.org>`_
 - `scp: scp module for paramiko <https://github.com/jbardin/scp.py>`_

doc/source/getting_started/Contributing.rst

@@ -51,7 +51,7 @@ Steps to Contribute
 2) Either by cloning to your local system or working in `GitHub Codespaces <https://github.com/features/codespaces>`_, create a work environment to make your changes.
 3) Add your fork as the ``origin`` remote and the original project repository as the ``upstream`` remote.  While this step isn't a necessary, it allows you to keep your fork up to date in the future.
 4) Create a new branch to do your work.
-5) Make your changes on the new branch and add yourself to the list of `project contributors <./Citations.html#contributors>`_.
+5) Make your changes on the new branch and add yourself to the list of project `contributors <https://github.com/tsutterley/read-ICESat-2/blob/main/CONTRIBUTORS.rst>`_.
 6) Push your work to GitHub under your fork of the project.
 7) Submit a `Pull Request <https://github.com/tsutterley/read-ICESat-2/pulls>`_ from your forked branch to the project repository.
 

doc/source/getting_started/ICESat-2-Data-Products.rst

@@ -29,28 +29,28 @@ Each beam pair consists of a weak beam and a strong beam, with the strong beam a
 The spots illuminated on the ground from the ATLAS strong beams are numbered 1, 3, and 5, and the spots illuminated on the ground from the ATLAS weak beams are numbered 2, 4, and 6.
 
 The ICESat-2 observatory can be oriented in one of two positions with respect to the direction of travel.
-In the forward orientation, the weak beams lead the strong beams and a weak beam is on the left edge of the beam pattern (gt1l).
+In the forward orientation, the weak beams lead the strong beams and a weak beam is on the left edge of the beam pattern (``gt1l``).
 This is reversed in the backward orientation, and the strong beams lead the weak beams with a strong beam on the left edge of the beam pattern (:numref:`table2`).
 The beam strength, spot number, atmospheric profile and spacecraft orientation can all be found as HDF5 attributes of each beam group.
 
 .. _table2:
 .. table:: ATLAS beam mapping when in the forward and backward orientations
     :align: center
 
-    +----------------+------------------------+------------------------+
-    | | Ground Track | | Forward Orientation  | | Backward Orientation |
-    | | Designation  | | Beam Strength        | | Beam Strength        |
-    +================+========================+========================+
-    |      gt1l      |          Weak          |         Strong         |
-    +----------------+------------------------+------------------------+
-    |      gt1r      |         Strong         |          Weak          |
-    +----------------+------------------------+------------------------+
-    |      gt2l      |          Weak          |         Strong         |
-    +----------------+------------------------+------------------------+
-    |      gt2r      |         Strong         |          Weak          |
-    +----------------+------------------------+------------------------+
-    |      gt3l      |          Weak          |         Strong         |
-    +----------------+------------------------+------------------------+
-    |      gt3r      |         Strong         |          Weak          |
-    +----------------+------------------------+------------------------+
+    +--------------------+------------------------+------------------------+
+    | | Ground Track     | | Forward Orientation  | | Backward Orientation |
+    | | Designation      | | Beam Strength        | | Beam Strength        |
+    +====================+========================+========================+
+    |      ``gt1l``      |          Weak          |         Strong         |
+    +--------------------+------------------------+------------------------+
+    |      ``gt1r``      |         Strong         |          Weak          |
+    +--------------------+------------------------+------------------------+
+    |      ``gt2l``      |          Weak          |         Strong         |
+    +--------------------+------------------------+------------------------+
+    |      ``gt2r``      |         Strong         |          Weak          |
+    +--------------------+------------------------+------------------------+
+    |      ``gt3l``      |          Weak          |         Strong         |
+    +--------------------+------------------------+------------------------+
+    |      ``gt3r``      |         Strong         |          Weak          |
+    +--------------------+------------------------+------------------------+