5.2.1

📅 2021-06-25

• Fixed issue where a force ingest on a catalogue-only archive would fail.

5.2

📅 2021-06-22

• Added generic hooking mechanism (not coupled to a product type)

• Added 'muninn-update retype' to allow correction of product_type values.

• Added 'day', 'hour', 'minute', 'second', and 'time' subscripts for
  grouping by timestamps in muninn-summary.

• UUID values can now be used everywhere as boolean in query expressions.

• Add support for new type conversions of pg8000 >= 1.16.

• Allow filename in http result header for muninn-pull to be unquoted.

5.1

📅 2021-01-31

• Added support for oauth2 (Resource Owner Password Credentials Grant) for
  muninn-pull and added oauth2 support to the auth_file credentials file.

• url prefixes can now be used as keys in the auth_file credentials file
  (instead of only hostnames).

• muninn-pull will now auto-unzip a downloaded file if the downloaded filename
  equals core.physical_name + '.zip' (or '.ZIP').

• Improved memory efficiency of muninn-pull. Files are no longer read fully
  in memory when downloading.

• Added new plugin attribute 'hash_type', to specify hash algorithm
  (matching hashlib module). The 'use_hash' attribute has been deprecated.
  core.hash will now contain ':' as content. Existing core.hash
  values will still be treated as sha1 hashes for backward compatibility.

• Fixed issue where enabling indices on Geometry fields did not work.

• Archive.retrieve() and Archive.retrieve_by_name() now return a list of
  product paths. Archive.retrieve_by_uuid() now returns a single path.

• Added Archive.delete_properties_by_uuid().

• Added download_args, upload_args, copy_args, and transfer_config options
  to S3 storage backend.

• Added (optional) product type plugin attribute 'namespaces', which
  contains a list of names of non-core namespace names of properties that
  plugin.analyze may return. Those non-core properties are now also passed to
  the other plugin methods.

• Removed muninn-update --namespaces option, as this should now work
  automatically.

• Added product plugin method 'post_remove_hook' that is called after a
  successful remove.

• Changed muninn-update -a <action> command to munnin-update <action>.

• Changed muninn-ingest -c command to mean 'catalogue only'.

• Added basic 'in' and 'not in' operators (so that e.g. the following works:
  physical_name not in ["a", "b"])

• Namespace records can now be removed with Archive.update_properties() by
  passing: Struct({'mynamespace': None})

5.0

📅 2020-07-30

• Added a storage backend framework, including S3 and Swift backends.

• Changed configuration file format due to new storage backends:

  • renamed backend parameter to database in archive section
  • added storage parameter to archive section
  • added new sections for fs, s3, and 'swift`
  • moved root and use_symlinks parameters from archive section
    to fs section
  • added tmp_root parameter to s3 and swift sections

• Added support for pg8000 as postgresql driver. Added 'library'
  parameter to 'postgresql' section to specify library to use.

• Added a basic test/coverage framework.

• Changed the syntax for defining namespace properties, allowing
  one to specify if a property should be indexed in the database backend.
  The old approach is still supported for backward compatibility.

• Added -c/--catalogue-only option to muninn-remove.

• Added support for sub-queries in is_source_of and is_derived_from
  (specify a sub-query instead of a uuid).

• Added support for searching on UUID directly (without uuid==).

• Several fixes for NOT expressions in queries.

• Improved querying on NULL properties. For example, for 'property != b',
  we now also return products for which b is NULL.

• Added property_names argument to archive.retrieve_properties().

• Added archive.delete_properties().

• Fixed issue with intra-archive symlinks without enclosing dir (resulting
  in broken symlinks).

• Fixed issue where product hashes were not actually verified.

• Made product hashing compatible with Python 3 (and using UTF-8 encoding
  before hashing, instead of the current filesystem encoding).

4.4

📅 2019-04-03

• Added optional progress bars to muninn-ingest (requires 'tqdm' package).

• Added --parallel and --processes options to muninn-ingest.

• Added --processes option to muninn-update.

• Added property_names argument to archive.search().

• Cleaned up terminology: we now use 'properties' instead of 'attributes'
  everywhere when refering to product metadata.

• Fixed issue in sqlite backend where geometry covers()/intersects()
  operations returned true if geometry property was not defined.

4.3

📅 2018-10-04

• Added muninn-summary to display aggregate statistics.

• Removed muninn-search '-s' option in favor of muninn-summary.

• Fixed initialization of database when using sqlite backend.

• Strip operation now automatically restricts itself to products with a
  non-empty archive_path.

4.2

📅 2018-06-12

• Added -f/--force option to muninn-ingest.

• Improved formatting of duration in muninn-search summary output.

• Added optional progress bars to muninn-update (requires 'tqdm' package).

• Added --parallel and --keep options to muninn-update.

• Several fixes for muninn-search --paths.

• Archive.export() now returns the list of paths of the exported products. The
  export_ functions in the product type plugins must return the path of the
  exported product.

• Fixed issue where the database connection was not properly cleaned up when
  an error was raised during a database commit/rollback.

• Fixed issue where muninn-prepare --dryrun would not produce output for
  sqlite if the database file did not exist yet.

• Fixed issue where 'active' state was not always reset when a muninn-pull
  failed.

• Improve handling of creating tags/links that already exist
  (postgresql will now only raise an exception for contention cases).

• Added check on startup to verify that all mandatory attributes/methods are
  available for each product type plugin.

• Several smaller fixes (including Python3 compatibility).

4.1

📅 2017-11-16

• Added support for Python 3.

• Dropped support for spatialite 3.1 and earlier. spatialite 4.0.0 or higher
  is now required (which requires a version of sqlite3 that support
  extensions). Support for pyspatialite, which only worked with spatialite
  3.1, has been dropped. pysqlite2 is now the preferred module for sqlite
  support.

• Muninn now supports custom remote backends (used for muninn-pull).
  The ecmwfapi pull support has now been removed from the core muninn code and
  will become available as a separate plugin.

• Added muninn-info to display generic archive information (currently, the
  availables namespaces and product types)

• Added archive maintenance tool muninn-update that allows updating properties
  for a set of products (for when e.g. a product type plugin has changed).

• Added mod_spatialite_path option for sqlite backend.

• Command line tools now have a --verbose mode, displaying debug-level logging.

• Added 'paths' option to muninn-search that prints the physical path of the
  products found instead of the normal output.

• Added --keep option to muninn-ingest to ingest products that are already in
  the archive, maintaining its current relative path.

• Allow calling muninn-search -p '*' to fetch all fields of all namespaces.

• muninn.Archive.product_path() argument can be a product properties Struct.

• Added support for tabulate package to format the output.

• muninn.Archive.update_properties() takes an optional 'create_namespaces'
  parameter that, if set, will create any undefined namespaces for a product.

• Added product plugin method 'post_pull_hook' that is called after a
  successful pull.

• Close SQL connection after each operation (ingestion/search/etc.).

• Added support for separate auth_file that can contain credentials for
  downloads performed using muninn-pull.

• Added -f option to muninn-strip.

• Fixed issue where geometry columns in namespace extensions were not
  created when using the sqlite backend.

4.0

📅 2017-03-31

• Changed license of muninn to BSD.

• Muninn can now be used as a pure catalogue server.
  An archive root directory is still needed, but can remain empty.

• Changed the core schema:

  • core.archive_date is now optional and is only set if core.archive_path is
    set.
    It will reflect the time at which the product was stored in the archive.
  • added core.metadata_date which represents the last modification time of
    the metadata record. It will automatically be updated if the metadata is
    changed via the muninn interfaces.
  • added optional core.remote_url field which can contain urls of the form
    file://, http://, ftp://, etc. that point to a remote location of the
    product.

• The schema_name option for the postgresql backend has been removed.
  This is now replaced by a table_prefix option that is applicable for
  both the postgresql and sqlite backends.

• The sqlite backend will no longer remove the database file for a 'destroy'
  (it will only remove the tables). Both 'prepare' and 'destroy' can be
  performed on an existing sqlite database file.

• Added 'pull' command that allows downloading of products that are not yet
  in the archive. It will use the core.remote_url value to perform the
  download. A 'retrieve' still only works on products that are already in the
  archive.
  The pull method supports the default http/https/file/ftp protocols but also
  supports a custom ecmwfapi protocol that will use the ecmwf-api-client
  python package to download data from the ECMWF MARS archive.

• The muninn Struct type that is used in the Python interface to store
  metadata can now be initialized with a Python dict.

• Added create_properties() and generate_uuid() functions to the Python
  muninn.Archive interface. These are functions that should be used to create
  pure metadata records (without archived product) in a muninn archive.

• Added verify_hash() function to the Python muninn.Archive interface.

• muninn.Archive.generate_uuid() is now a static method.

• Improved preparing and destroying of muninn archives. This will now better
  distinguish between the archive directory and the catalogue.

• Verifying of the hash after an ingest of a product in the archive is now
  optional (--verify-hash argument to muninn-ingest).

• The muninn sqlite backend now adds a spatialindex on footprint.
  The indices for both backends have all been renamed to idx_<table>_<field>.

• muninn-prepare has a --dry-run mode, that just dumps the SQL that would be
  executed.

• The muninn sqlite backend now supports the escape character \ when using
  the ~= operator.

• Moved change history from README to separate CHANGELOG file.
  Renamed README.developer to EXTENSIONS and added DEVELOPER file.
  Include all documentation files with the source package.