v0.10.4

DL1 Data Handler Release v0.10.4

Upgrade to MARS v2-20-1 superstar files
Aligned MAGIC files with ctapipe conventions to share the same CTLearn config

Major Features And Improvements

Bug Fixes and Other Changes

• code clean up #112

Known Issues

• Increase of RAM usage in the DL1DataReaderSTAGE1, when the DL1 files are split by tel_id. Issue #104. At the moment it is recommended to use the DL1 files, where the telescope tables are split by tel_types.

v0.10.3

DL1 Data Handler Release v0.10.3

Bug fix for stage1 datasets (split by tel_types)

Major Features And Improvements

Bug Fixes and Other Changes

• fix the tel id select option for stage1 datasets (split by tel_types) #111

Known Issues

• Increase of RAM usage in the DL1DataReaderSTAGE1, when the DL1 files are split by tel_id. Issue #104. At the moment it is recommended to use the DL1 files, where the telescope tables are split by tel_types.

v0.10.2

DL1 Data Handler Release v0.10.2

Update of dependencies #110

Major Features And Improvements

Bug Fixes and Other Changes

• Fix hyperlinks in README for pypi upload

Known Issues

• Increase of RAM usage in the DL1DataReaderSTAGE1, when the DL1 files are split by tel_id. Issue #104. At the moment it is recommended to use the DL1 files, where the telescope tables are split by tel_types.

v0.10.1

DL1 Data Handler Release v0.10.1

Support latest official CTA DL1 data model v2.1.0 (still supporting Prod5 files with v1.1.0)
Compatibility with ctapipe v0.12.0

Major Features And Improvements

• Support multiple versions of the official CTA DL1 data model #107
• Moved the class weights calculation for unbalanced datasets from CTLearn to the DL1DataReader #108

Bug Fixes and Other Changes

• Formatted via black
• Updated the README with installation instructions
• Enhance version handling

Known Issues

• Increase of RAM usage in the DL1DataReaderSTAGE1, when the DL1 files are split by tel_id. Issue #104. At the moment it is recommended to use the DL1 files, where the telescope tables are split by tel_types.

v0.10.0

DL1 Data Handler Release v0.10.0

Support official CTA DL1 data model v1.1.0
Compatibility with ctapipe v0.10.5
Support MAGIC real data by @astrojarred
Deprecating DL1DH writer

Major Features And Improvements

• Subclassing the DL1Reader to support the official DL1 data model (for both versions split by tel_id and by tel_types) and the deprecated DL1DH DL1 data format
• Updated DLEventSource for MAGIC real data files. @astrojarred
• Added ability to save and load example identifiers in a separate file to skip the init of the DL1Reader.
• Make 'generate_runlist' and 'write_data' scripts to command-line tools.
• Updated example notebooks.
• Updated conda package build files and setup.py dependencies. Removed light setup (without ctapipe), because ctapipe is now scritly required.

Bug Fixes and Other Changes

• DL1a and/or DL1b data can be read by the subclasses of DL1Reader.

Known Issues

• Increase of RAM usage in the DL1DataReaderSTAGE1, when the DL1 files are split by tel_id. Issue #104. At the moment it is recommended to use the DL1 files, where the telescope tables are split by tel_types.
• Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The exception resulting from the call is currently being caught and ignored.

DL1 Data Handler 0.8.3

Release 0.8.3

Compatibility with ctlearn v0.5.0. Bugfixes w.r.t. v0.8.2.

Major Features And Improvements

Bug Fixes and Other Changes

• Fix counting of class examples for labels that are sequences. @aribrill
• added electrons to the particle transformation. @TjarkMiener

DL1 Data Handler 0.9.0

Release 0.9.0

Numerous bugfixes and improvements.
DLEventSource for MAGIC calibrated files by @sahilyadav27.
DLEventSource for MAGIC superstar files.
Expansion of the data format with image parameters by @LucaRomanato.
Compatibility with ctapipe v0.9.1.

Major Features And Improvements

• Updated DL1DataWriter to ctapipe v0.9.1 and expand the data format with image parameters.
• Make ctapipe's image_extractor and gain_selector configurable in the config.
• Added multiple parameter tables for different cleaning settings. Use MARS default ones for MAGIC superstar files.
• Added ability to only store selected tels (by tel_id).
• Updated example notebook (also for MAGIC data).
• Updated conda package build files and setup.py dependencies.

Bug Fixes and Other Changes

Known Issues

• The image parameters can't be read by the current DL1DataReader. This is planned for the next release DL1DH v0.9.1.
• Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The exception resulting from the call is currently being caught and ignored.

DL1 Data Handler 0.8.2

Release 0.8.2

Compatibility with ctapipe v0.8.0. Numerous bugfixes and improvements.

DL1DataReader compatible for the file format of DL1DH >= v0.7.4.

Major Features And Improvements

• Updated DL1DataWriter to ctapipe v0.8.0.
• gain_selector in ctapipe now. Redundant unit tests deleted.
• Numerous improvements for the ImageMapper (IM) class.
• ctapipe and ctapipe-extra optional dependencies for DL1DataReader and IM.
• Added and modified Transform functions for the data processor.
• Added telescope pointing for alt/az transformation
• Added 'sorting' transformation for stereo reading.
• Added and modified filters for the data processor in filters.py.
• Updated example notebook.
• Updated conda package build files and setup.py dependencies.

Bug Fixes and Other Changes

• Fixed FlashCam mapping bug with reading pixel positions from the data files.
• Normalization for nearest interpolation in IM.
• Removed rotation back to original orientation option in IM.
• Speeded up the data reading.
• Added parallel reads support.
• Sorting transformation fixed performance drop from the CTLearn cnn-rnn (stereo) model.

Known Issues

• Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The exception resulting from the call is currently being caught and ignored.

DL1 Data Handler 0.8.1 Legacy Version

Release 0.8.1-legacy

"Legacy" release adapting DL1DataReader for the file format of DL1DH <0.6.0.

Major Features And Improvements

• Added reader_legacy module containing a DL1DataReader adapted for the legacy file format.

Bug Fixes and Other Changes

• Updated dl1_data_handler_demo notebook for legacy reader.
• Updated transforms and utils modules to use legacy column names in relevant Transforms and filters.
• Added SortTelescopes Transform to sort telescopes by trigger or image size.
• Added event_multiplicity_filter to filter events based on multiplicity.

Known Issues

• The current version of ctapipe (v0.6.2) does not correctly support a version tag that can be read by pkg_resources (instead VERSION = "unknown") so it is not being written correctly to the output file headers.
• Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The exception resulting from the call is currently being caught and ignored.

DL1 Data Handler 0.8.0

Release 0.8.0

Integration of DL1DH Reader, Processor, and ImageMapper completed. Numerous bugfixes and improvements for compatibility with CTLearn.

Added option for serial/parallel writing modes to DL1DH Writer. Fixes to setuptools and conda build process.

Major Features And Improvements

• Finished changes to reader and image mapper APIs to be compatible with CTLearn
• Finished incorporating image mapper and DL1DH processor.
• Added example notebook.
• Added six new Transform functions for the data processor and moved to transform.py
• Added single threaded/parallel writes modes as a config option for DL1DataWriter.
• Updated conda package build files and setup.py dependencies.

Bug Fixes and Other Changes

• Fixed output_dir option of DataWriter to correctly write output files to directory.
• Added output_dir option to generate_runlists.py script.
• Moved example config files and runlists to /config folder
• Added separate expected_mc_events config setting for optimizing chunking on the MC_Events table.
• Fixed various issues with setuptools dependencies.
• Fixed FACT camera shape mismatch by upgrading to ctapipe-extra 0.2.17.
• Changed mapping table indices to start at 0 (removed 0 padding).
• Added option for camera rotation for LST, NectarCam, MAGICCam in image mapper.
• Fixed output_dir option in write_data.py script.
• Moved config files to config folder
• Changed indexing convention in image mapper.
• Changed get_image() to return a vector when map method is axial addressing
• Fixed status badge

Known Issues

• The current version of ctapipe (v0.6.2) does not correctly support a version tag that can be read by pkg_resources (instead VERSION = "unknown") so it is not being written correctly to the output file headers.
• Calling file.close() in the destructor of CTAMLDataDumper causes a KeyError on some data files due to an empty NodeManager registry. The cause of this exception is unclear. The exception resulting from the call is currently being caught and ignored.