docs/library/deflate: Add docs for deflate.DeflateIO.

Also update zlib & gzip docs to describe the micropython-lib modules.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>

docs/library/deflate.rst

@@ -0,0 +1,177 @@
+:mod:`deflate` -- deflate compression & decompression
+=====================================================
+
+.. module:: deflate
+ :synopsis: deflate compression & decompression
+
+This module allows compression and decompression of binary data with the
+`DEFLATE algorithm <https://en.wikipedia.org/wiki/DEFLATE>`_
+(commonly used in the zlib library and gzip archiver).
+
+**Availability:**
+
+* Added in MicroPython v1.21.
+
+* Decompression: Enabled via the ``MICROPY_PY_DEFLATE`` build option, on by default
+ on ports with the "extra features" level or higher (which is most boards).
+
+* Compression: Enabled via the ``MICROPY_PY_DEFLATE_COMPRESS`` build option, on
+ by default on ports with the "full features" level or higher (generally this means
+ you need to build your own firmware to enable this).
+
+Classes
+-------
+
+.. class:: DeflateIO(stream, format=AUTO, wbits=0, close=False, /)
+
+ This class can be used to wrap a *stream* which is any
+ :term:`stream-like <stream>` object such as a file, socket, or stream
+ (including :class:`io.BytesIO`). It is itself a stream and implements the
+ standard read/readinto/write/close methods.
+
+ The *stream* must be a blocking stream. Non-blocking streams are currently
+ not supported.
+
+ The *format* can be set to any of the constants defined below, and defaults
+ to ``AUTO`` which for decompressing will auto-detect gzip or zlib streams,
+ and for compressing it will generate a raw stream.
+
+ The *wbits* parameter sets the base-2 logarithm of the DEFLATE dictionary
+ window size. So for example, setting *wbits* to ``10`` sets the window size
+ to 1024 bytes. Valid values are ``5`` to ``15`` inclusive (corresponding to
+ window sizes of 32 to 32k bytes).
+
+ If *wbits* is set to ``0`` (the default), then a window size of 256 bytes
+ will be used (corresponding to *wbits* set to ``8``), except when
+ :ref:`decompressing a zlib stream <deflate_wbits_zlib>`.
+
+ See the :ref:`window size <deflate_wbits>` notes below for more information
+ about the window size, zlib, and gzip streams.
+
+ If *close* is set to ``True`` then the underlying stream will be closed
+ automatically when the :class:`deflate.DeflateIO` stream is closed. This is
+ useful if you want to return a :class:`deflate.DeflateIO` stream that wraps
+ another stream and not have the caller need to know about managing the
+ underlying stream.
+
+ If compression is enabled, a given :class:`deflate.DeflateIO` instance
+ supports both reading and writing. For example, a bidirectional stream like
+ a socket can be wrapped, which allows for compression/decompression in both
+ directions.
+
+Constants
+---------
+
+.. data:: deflate.AUTO
+ deflate.RAW
+ deflate.ZLIB
+ deflate.GZIP
+
+ Supported values for the *format* parameter.
+
+Examples
+--------
+
+A typical use case for :class:`deflate.DeflateIO` is to read or write a compressed
+file from storage:
+
+.. code:: python
+
+ import deflate
+
+ # Writing a zlib-compressed stream (uses the default window size of 256 bytes).
+ with open("data.gz", "wb") as f:
+ with deflate.DeflateIO(f, deflate.ZLIB) as d:
+ # Use d.write(...) etc
+
+ # Reading a zlib-compressed stream (auto-detect window size).
+ with open("data.z", "rb") as f:
+ with deflate.DeflateIO(f, deflate.ZLIB) as d:
+ # Use d.read(), d.readinto(), etc.
+
+Because :class:`deflate.DeflateIO` is a stream, it can be used for example
+with :meth:`json.dump` and :meth:`json.load` (and any other places streams can
+be used):
+
+.. code:: python
+
+ import deflate, json
+
+ # Write a dictionary as JSON in gzip format, with a
+ # small (64 byte) window size.
+ config = { ... }
+ with open("config.gz", "wb") as f:
+ with deflate.DeflateIO(f, deflate.GZIP, 6) as f:
+ json.dump(config, f)
+
+ # Read back that dictionary.
+ with open("config.gz", "rb") as f:
+ with deflate.DeflateIO(f, deflate.GZIP, 6) as f:
+ config = json.load(f)
+
+If your source data is not in a stream format, you can use :class:`io.BytesIO`
+to turn it into a stream suitable for use with :class:`deflate.DeflateIO`:
+
+.. code:: python
+
+ import deflate, io
+
+ # Decompress a bytes/bytearray value.
+ compressed_data = get_data_z()
+ with deflate.DeflateIO(io.BytesIO(compressed_data), deflate.ZLIB) as d:
+ decompressed_data = d.read()
+
+ # Compress a bytes/bytearray value.
+ uncompressed_data = get_data()
+ stream = io.BytesIO()
+ with deflate.DeflateIO(stream, deflate.ZLIB) as d:
+ d.write(uncompressed_data)
+ compressed_data = stream.getvalue()
+
+.. _deflate_wbits:
+
+Deflate window size
+-------------------
+
+The window size limits how far back in the stream the (de)compressor can
+reference. Increasing the window size will improve compression, but will
+require more memory.
+
+However, just because a given window size is used for compression, this does not
+mean that the stream will require the same size window for decompression, as
+the stream may not reference data as far back as the window allows (for example,
+if the length of the input is smaller than the window size).
+
+If the decompressor uses a smaller window size than necessary for the input data
+stream, it will fail mid-way through decompression with :exc:`OSError`.
+
+.. _deflate_wbits_zlib:
+
+The zlib format includes a header which specifies the window size used to
+compress the data (which due to the above, may be larger than the size required
+for the decompressor).
+
+If this header value is lower than the specified *wbits* value, then the header
+value will be used instead in order to reduce the memory allocation size. If
+the *wbits* parameter is zero (the default), then the header value will only be
+used if it is less than the maximum value of ``15`` (which is default value
+used by most compressors [#f1]_).
+
+In other words, if the source zlib stream has been compressed with a custom window
+size (i.e. less than ``15``), then using the default *wbits* parameter of zero
+will decompress any such stream.
+
+The gzip file format does not include the window size in the header.
+Additionally, most compressor libraries (including CPython's implementation
+of :class:`gzip.GzipFile`) will default to the maximum possible window size.
+This makes it difficult to decompress most gzip streams on MicroPython unless
+your board has a lot of free RAM.
+
+If you control the source of the compressed data, then prefer to use the zlib
+format, with a window size that is suitable for your target device.
+
+.. rubric:: Footnotes
+
+.. [#f1] The assumption here is that if the header value is the default used by
+ most compressors, then nothing is known about the likely required window
+ size and we should ignore it.

docs/library/gzip.rst

@@ -0,0 +1,106 @@
+:mod:`gzip` -- gzip compression & decompression
+===============================================
+
+.. module:: gzip
+ :synopsis: gzip compression & decompression
+
+|see_cpython_module| :mod:`python:gzip`.
+
+This module allows compression and decompression of binary data with the
+`DEFLATE algorithm <https://en.wikipedia.org/wiki/DEFLATE>`_ used by the gzip
+file format.
+
+.. note:: Prefer to use :class:`deflate.DeflateIO` instead of the functions in this
+ module as it provides a streaming interface to compression and decompression
+ which is convenient and more memory efficient when working with reading or
+ writing compressed data to a file, socket, or stream.
+
+**Availability:**
+
+* This module is **not present by default** in official MicroPython firmware
+ releases as it duplicates functionality available in the :mod:`deflate
+ <deflate>` module.
+
+* A copy of this module can be installed (or frozen)
+ from :term:`micropython-lib` (`source <https://github.com/micropython/micropython-lib/blob/master/python-stdlib/gzip/gzip.py>`_).
+ See :ref:`packages` for more information. This documentation describes that module.
+
+* Compression support will only be available if compression support is enabled
+ in the built-in :mod:`deflate <deflate>` module.
+
+Functions
+---------
+
+.. function:: open(filename, mode, /)
+
+ Wrapper around built-in :func:`open` returning a GzipFile instance.
+
+.. function:: decompress(data, /)
+
+ Decompresses *data* into a bytes object.
+
+.. function:: compress(data, /)
+
+ Compresses *data* into a bytes object.
+
+Classes
+-------
+
+.. class:: GzipFile(*, fileobj, mode)
+
+ This class can be used to wrap a *fileobj* which is any
+ :term:`stream-like <stream>` object such as a file, socket, or stream
+ (including :class:`io.BytesIO`). It is itself a stream and implements the
+ standard read/readinto/write/close methods.
+
+ When the *mode* argument is ``"rb"``, reads from the GzipFile instance will
+ decompress the data in the underlying stream and return decompressed data.
+
+ If compression support is enabled then the *mode* argument can be set to
+ ``"wb"``, and writes to the GzipFile instance will be compressed and written
+ to the underlying stream.
+
+ By default the GzipFile class will read and write data using the gzip file
+ format, including a header and footer with checksum and a window size of 512
+ bytes.
+
+ The **file**, **compresslevel**, and **mtime** arguments are not
+ supported. **fileobj** and **mode** must always be specified as keyword
+ arguments.
+
+Examples
+--------
+
+A typical use case for :class:`gzip.GzipFile` is to read or write a compressed
+file from storage:
+
+.. code:: python
+
+ import gzip
+
+ # Reading:
+ with open("data.gz", "rb") as f:
+ with gzip.GzipFile(fileobj=f, mode="rb") as g:
+ # Use g.read(), g.readinto(), etc.
+
+ # Same, but using gzip.open:
+ with gzip.open("data.gz", "rb") as f:
+ # Use f.read(), f.readinto(), etc.
+
+ # Writing:
+ with open("data.gz", "wb") as f:
+ with gzip.GzipFile(fileobj=f, mode="wb") as g:
+ # Use g.write(...) etc
+
+ # Same, but using gzip.open:
+ with gzip.open("data.gz", "wb") as f:
+ # Use f.write(...) etc
+
+ # Write a dictionary as JSON in gzip format, with a
+ # small (64 byte) window size.
+ config = { ... }
+ with gzip.open("config.gz", "wb") as f:
+ json.dump(config, f)
+
+For guidance on working with gzip sources and choosing the window size see the
+note at the :ref:`end of the deflate documentation <deflate_wbits>`.

docs/library/index.rst

@@ -64,6 +64,7 @@ library.
  collections.rst
  errno.rst
  gc.rst
+ gzip.rst
  hashlib.rst
  heapq.rst
  io.rst
@@ -95,6 +96,7 @@ the following libraries.
  bluetooth.rst
  btree.rst
  cryptolib.rst
+ deflate.rst
  framebuf.rst
  machine.rst
  micropython.rst
@@ -194,11 +196,11 @@ Extending built-in libraries from Python
 A subset of the built-in modules are able to be extended by Python code by
 providing a module of the same name in the filesystem. This extensibility
 applies to the following Python standard library modules which are built-in to
-the firmware: ``array``, ``binascii``, ``collections``, ``errno``, ``hashlib``,
-``heapq``, ``io``, ``json``, ``os``, ``platform``, ``random``, ``re``,
-``select``, ``socket``, ``ssl``, ``struct``, ``time`` ``zlib``, as well as the
-MicroPython-specific ``machine`` module. All other built-in modules cannot be
-extended from the filesystem.
+the firmware: ``array``, ``binascii``, ``collections``, ``errno``, ``gzip``,
+``hashlib``, ``heapq``, ``io``, ``json``, ``os``, ``platform``, ``random``,
+``re``, ``select``, ``socket``, ``ssl``, ``struct``, ``time`` ``zlib``, as well
+as the MicroPython-specific ``machine`` module. All other built-in modules
+cannot be extended from the filesystem.
 
 This allows the user to provide an extended implementation of a built-in library
 (perhaps to provide additional CPython compatibility or missing functionality).