0.4.0
Summary
This is a major update of the interface to the DIRAC grid for protopipe.
It is strongly recommended to any protopipe user that runs analyses on the DIRAC platform to apply this upgrade.
IMPORTANT: Due to the nature of protopipe the installation of the interface is also needed to create the directory structure for a new analysis, which makes some use-cases like pure benchmarking more user-friendly (recommended). It is planned to migrate this interface into protopipe starting from its next minor release.
Following the upgrade to Python3 of DIRAC and CTADIRAC, now the interface can be installed as a Python3-based package.
A set of easy-to-use command-line tools has been defined.
Unit-testing has slightly improved, even though most inner operations are based on DIRAC.
The full-scale analysis framework has been reviewed and improved in light of these changes.
The interface provides provenance for the analyses stored on the DIRAC FileCatalog, a centralized logging system along with improved debugging and an improved single-analysis structure which synchronizes with the new benchmarking module of protopipe.
The interface provides also automatic builds of Docker containers for both development and released content.
For more details, please find below the complete list of merged Pull Request since the last release.
Contributors
- Michele Peresano(@HealthyPear)
- Karl Kosack (@kosack)
What is changed since 0.3.1
🚀 General features
- Upload also grid.yaml at submission (#74) @HealthyPear
- added --particle and --n_file_per_job options (#72) @kosack
- Build and publish Docker images to DockerHub (#54) @HealthyPear
- Upgrade to python 3 (#50) @HealthyPear
- Add unit-test for file merging (#51) @HealthyPear
- Save each particle type in its sub-folder throughout the analysis (#47) @HealthyPear
- Add directory syncing after downloading (#45) @HealthyPear
- Add option to submit jobs with data reprocessing (#39) @HealthyPear
- Improve pilot for better job status debugging (#37) @HealthyPear
- Add new script to filter lists of files depending on where their replicas reside (#43) @HealthyPear
- Improve interface to allow support for different containers (#32) @HealthyPear
- Upload analysis config for provenance (#33) @HealthyPear
- Improve script for merging files (#31) @HealthyPear
- Upload models' configuration files and remove cleaning mode from filenames (#30) @HealthyPear
- Expose --debug flag (#29) @HealthyPear
- Upgrade for use of ctapipe 0.11.0 (#27) @HealthyPear
- Upgrade for usage of ctapipe 0.10.5 (#26) @HealthyPear
🐛 Bug Fixes
- Fix/upload models (#69) @kosack
- Upload analysis config file before submitting jobs (#67) @HealthyPear
- Define global list of replicating sites (#68) @HealthyPear
- add try/catch around main() in submit and log errors (#64) @kosack
- fixed logging error (#63) @kosack
- Fix gfal2 version (#61) @HealthyPear
- Check for job submission failure and output correct warning (#60) @kosack
- Fix extraction of run numbers (#46) @HealthyPear
- Fix inconsistency between different use-cases (#40) @HealthyPear
- Improve script for merging files (#31) @HealthyPear
- Fix single file output name inconsistency (#22) @HealthyPear
🧰 Maintenance
- added some better logging output for SUBMIT_JOBS (#70) @kosack
- Upgrade to python 3 (#50) @HealthyPear
- Add issue templates (#53) @HealthyPear
- Improve interface to allow support for different containers (#32) @HealthyPear
- Update readme for coverage report (fixed version) (#36) @HealthyPear
- Update README for coverage report (#35) @HealthyPear
- Add first unit test (#34) @HealthyPear
