v1.1.10

v1.1.10 Release (#432)

• Sync up read the docs config with dev environments using poetry (#399)
• Print usage on command line when no args are provided (#404)
• Add sacct fallback to slurm adapter to improve robustness of job tracking (#405)
• Update Flurm Job State mappings for flux versions >= 0.26 (#407)
• Bump certifi from 2021.10.8 to 2022.12.7 to address security issue (#409)
• Bump cryptography from 37.0.1 to 38.0.3 to address security issue (#410)
• Add missing shbang in unscheduled scripts from lsf adapter (#411)
• Update poetry lockfile to address dependabot flagged security issues (#412)
• Fix for Dockerfile smell DL3006 (#418)
• Port Maestro documentation to mkdocs and expand coverage of features and tutorials (#403)
• Update version info to be driven from pyproject.toml exclusively, and hook up to command line (#419)
• Pin mermaid to < 10.x due to api change (#422)
• Bump lock file certifi from 2022.12.7 to 2023.7.22 to address security issue (#426)
• Refactor flux adapter to avoid using pickle to talk to flux brokers installed in external environments (#415)
  Also adds flux integration tests to exercise against real flux brokers
• Add pager functionality to status command (#420)
• Patch broken flux job cancellation (#428)
• Insulate slurm adapters from user customization of squeue and sacct output formats (#431)
  Also adds live unit and integration tests for slurm adapter

---

Co-authored-by: Francesco Di Natale frank.dinatale1988@gmail.com
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Bruno P. Kinoshita kinow@users.noreply.github.com
Co-authored-by: Charles Doutriaux doutriaux1@llnl.gov
Co-authored-by: Giovanni Rosa grosa23@yahoo.com
Co-authored-by: Brian Gunnarson 49216024+bgunnar5@users.noreply.github.com

v1.1.9: 1.1.9 Release (#398)

This release covers extensive improvements to the SLURM, LSF, and Flux adapters for scheduling to clusters as well as a new Rich-enabled status view. Other features include the addition of hashing workspaces, improvements to pgen, and many other background improvements/bugfixes.

v1.1.8

This patch release covers a critical bugfix that caused a regression in the ability to use variables and parameters in the scheduling parameters for a study step as well as makes the Specification class more easily inherited. This release also includes some improved documentation for parameters and parameter generation.

Other changes include the inclusion of the long description in the setup.py file to allow PyPi to render the markdown readme.

Inheritable validation and spec module (#280) [6edda08] -- authored by Benjamin Bay
Modified validation logic to skip over variable tokens. (#279) [5ca2b29] -- authored by Benjamin Bay
Updated the docs release version to 1.1.7 (#278) [3795de0] -- authored by jsemler
Replace unicode quote with ascii quote (#277) [f7e5b5b] -- authored by Jeremy White
Pgen docs (#275) [2998cc4] -- authored by Jeremy White
Addition of README as long description. (#269) [4d65e76] -- authored by Francesco Di Natale

v1.1.7

Official 1.1.7 release (#268)

* Addition of user enabled workspace hashing (#145)

* Addition of hashing to Study parameterization.

* Addition of the hashws option to argparse.

* Addition of a warning note for users who use labels in steps.

* Update setup.py to 1.1.4dev

* More generalized FluxScriptAdapter (#149)

* Addition of a more general flux ScriptAdapter.

* Addition of some casting from int to str

* Corrected "gpus" to "ngpus"

* Rework jobspec construction to make a valid jobspec.

* Check for empty value for cores per task.

* README tweak to update quickstart link. (#139)

* typos. fixes #141 (#154)

* Correction of flake8 style errors [new version of flake8].

* Update to setup.py to reflect dev version 1.0

* Correction to safe pathing for missed cases and make_safe_path enhancements. (#157)

* Made pickle and log path string safe for pathing.

* Tweaks to make_safe_path to include a base path.

* Updates to make_safe_path usage

* Correction to not modify the iterator copy.

* Correction to fix the format of output status time to avoid a comma that breaks printing. (#160)

* Addition of a utility function for formatting times to H:M:S

* _StepRecord time methods now call the new utility function.

* Tweaks to add days to the format to avoid 3 digit hours.

* Tweak to formatting.

* Made the day format more parsable.

* Removal of _stage_linear since it is now not needed. (#156)

* Removal of _stage_linear since it is now not needed.

* Addition of linear LULESH samples.

* Update the dev to 1.1.

* Addition of pargs for passing parameters to custom parameter generation (#152)

* Addition of a utility method to create a dictionary from a list of key-value pairs.

* Addition of the pargs interface for passing parameters to custom parameter generation.

* Addition of a Monte Carlo example that accepts pargs.

* Addition of pargs check for dependency on pgen.

* Addition of clearer error message for malformed parameters.

* Update setup.py

* do not overwrite log file. (#162)

Signed-off-by: Peter Robinson <robinson96@llnl.gov>

* Added confirmation message after launching a study (#163)

* Enhancements to store relative pathing in the metadata file. (#165)

* Changes to make workspaces reflect relative pathing based on step names.

* Addition of an alternative output format based on step combinations.

* Addition of tag to LULESH git dependency. (#169)

* Script Adapter Plugin (#167) (#170)

Fixes #167 

* added pytest to requirements

added Pipfile and pipenv settings

* Added property key to Abstract.ScriptAdapter (#167)
Also added impementation and tests to verify that existing functionality isn't changed

* updated factory to use key when registering adapters(#167)

* cleanedup linelength

* cleaned up imports to be specific to module (#167)

* added tests to verify exception for unknown adapter

* moved adapters tests to individual files

* added test to verify scriptadapter functionality (#167)

updated gitignore to have testing and pycharm ignores

testing existing adapters in factory (#167)

added test to verify factories.keys matches get_valid_adapters (#167)

added copyright to file

* updated __init__ modules to do dynamic includes

* removed unneeeded imports

* updated dependency versions

* fixed all flake8 errors

* updated to run flake8 and pytest when run locally

* updated tests to have documentation about purpose and function as requested in #170

* fixed line length

* Removal of nose from requirements.

* updated to remove nose from the requirements

* PyYAML vulnerability fix (#171)

* Locking the version of PyYAML to be above 2.1 because of an arbitrary code execution vulnerability.

* Addition of a version condition to pyyaml to patch a vulnerability.

* Update of Pipfile.lock to match Pipefile.

* Minor tweak to indentation for flake8 failure.

* fixed pyyaml to requirements (#172)

* Addition of a loader to the yaml load call. (#174)

Fixes #173 

* Addition of a loader to the yaml load call.

* Addition of a catch if the loader attribute is missing.

* Correction to install enum34 for Python versions < 3.4 (#176)

* Moved enum34 to condition dependent on Python<3.4.

* Addition of conditional enum34 install for requirements.txt.

* Correction of requirements.txt syntax for python version.

* Addition of a Dockerfile for tutorials and ease of trying out. (#178)

* Addition of a Dockerfile for quick tutorials.

* Tweaks for Docker and addition of git.

* Tweak to Docker file for caching.

* Addition of Docker documentation.

* Tweaks to Docker documentation.

* Removal of markdown ##

* Take out shebang from shell definition and add it when script is written. (#181)

* Take out shebang from shell definiton and at it when script is written.

* Include shebang in cmd and fix format of string written to file.

* Tweaks to fix malformed log statements. (#182)

* Correction to message when stating no to launch.

* Enhance shell batch setting to apply to scheduler scripts. (#183)

* Extension of shebang feature to allow users to specify shells.

* Addition of debug message to print kwargs.

* Addition of kwargs.

* Addition of basic batch settings to LULESH sample.

* Addition of kwargs to Flux adapters.

* Docstring tweaks.

* Docstring update.

* Fixes the addition of the shebang header for SLURM (#184)

* Docstring correction for LocalAdapter.

* Correction to addition of exec line at top of scripts.

* Correction to an accidental reassignment of cmd.

* Removal of an assignment of self._exec in SLURM adapter.

* Change to transition adapter returns to Record objects. (#177)

* Addition of a Record class for storing general data.

* Addition of SubmissionRecord type.

* Update to the order of for record parameters.

* Changes to StepRecord to expect SubmissionRecord returns.

* Updates to SLURM and local adapters to use SubmissionRecords.

* Slight tweak to LocalAdapter docstring.

* Tweak to have SubmissionRecord initialize its base.

* Addition of CancellationRecord class.

* Changes to CancellationRecord to map based on status.

* Additional interface additions and tweaks.

* Changes to have cancel use CancellationRecords.

* Update to ExecutionGraph to use records.

* Updates to SLURM and local adapters to use SubmissionRecords.

* Slight tweak to LocalAdapter docstring.

* Addition of CancellationRecord class.

* Additional interface additions and tweaks.

* Changes to have cancel use CancellationRecords.

* Cherry pick of execution commit.

* Removal of redundant "get" definiton.

* Addition of a SLURM enabled LULESH sample specification.

* Addition of output for stdout and stderr for Local adapter.

* Correction of file to open.

* Addition of 3.7 to testing stack.

* Added 3.7 to tox.ini.

* Removal of py37 in testing.

* Addition of py37 to travisCI (#187)

* Correction to 3.7 specification.

* Removed py37 from tox testing.

* Readded py37 to tox and removed duplicate from travis.

* Addition of build status badge.

* Update SLURM sample spec to add missing walltime.

* Addition of dumping the environment to a YAML file. (#190)

* Addition of documentation that covers the set up of a simple study (#168)

* Addition of simple Hello World spec.

* Addition of basics page to index.

* Addition of hello_world documentation.

* Additions to hello_world.

* More documentation in single step section.

* Continued edits to Hello World.

* Addition of parameter section.

* Addition of a note about %% token.

* Addition of directory structure.

* Continuation of parameter documentation.

* Removal of the depends key.

* Addition of the env section description.

* Addition of a link to Docker documentation for Dockerfiles.

* Addition of single parameter hello world.

* Correction of double colons.

* Correction of indentation.

* Addition of print out to verify output.

* Addition of sample specifications for multi and single params.

* Addition of more documentation for single param.

* Additional output to show parameter results.

* Correction to formatting.

* Addition of samples.

* Addition of simple Hello World spec.

* Addition of basics page to index.

* Addition of hello_world documentation.

* Additions to hello_world.

* More documentation in single step section.

* Continued edits to Hello World.

* Addition of parameter section.

* Addition of a note about %% token.

* Addition of directory structure.

* Continuation of parameter documentation.

* Removal of the depends key.

* Addition of the env section description.

* Addition of a link to Docker documentation for Dockerfiles.

* Addition of single parameter hello world.

* Correction of double colons.

* Correction of indentation.

* Addition of print out to verify output.

* Addition of sample specifications for multi and single params.

* Addition of more documentation for single param.

* Additional output to show parameter results.

* Correction to formatting.

* Updates to docstrings for data structures.

* Updates to clear Sphinx warnings.

* Removal of escape on the *args becuase of flake8 failure.

* Clean up of existing hello world specs.

* Addition of multistep example spec.

* Removal of * to fix sphinx errors.

* Correction to some docstrings.

* Tweaks to specs for consistent naming.

* Finished multi-step parameterized example.

* Tweaks to hello world docs.

* Addition of link to examples on GitHub.

* Correction of link to examples.

* Correction of link to examples (again).

* Removal of Pipfile.lock.

* Additions to gitignore for vscode and pipenv.

* Marking for v1.1.4 release.

* Corrected a missed merge for release v1.1.4

* Extend the Specification interface to break out loading from streams. (#198)

* Closes #198 

* Addition of loading specification "from_str".

* Updates to Specification docstrings.

* Updates to abstract Specification to change from str to stream.

* Updates to YAMLSpecification to use the new stream API.

* Removal of IOString

* Update to the YAMLSpecification load stream method.

* Quickfix: Addition of the accidental removal of the path member variable.

* Updating the version to 1.1.5dev (forgotten previously).

* Correction to versioning for install.

* Moved SLURM parameters to be placed in front of the submitted script. (#202)

Fixes #201

* Addition of version information to package and command line (#205)

* Addition of version information.

* Tweak to have setup.py pull from __version__

* Addition of command line arg to print version.

* Pinning version for release 1.1.5

* Addition of 1.1.5a to line up with PyPi labeling.

* Increment up to get rid of a0

* Addition of a simple example and logo to the README. (#208)

* Addition of the Maestro logo.

* Logo and hello world addition.

* Addition parameter section.

* Slight tweak to parameter section.

* Addition of a reference to samples folder.

* Update __init__.py to tick version to dev version.

* Addition of Spack shield.

* Enhances pgen to have access to environment specified in a study. (#209)

* Add the OUTPUT_PATH and SPECROOT to kwargs for pgen.

* Addition of the spec constructed environment.

* Remove "study_env" from pgen kwargs.

* Update to pgen function parameters.

* Update lulesh examples to have pgen vars.

* Correction to docstring ordering.

* Updates to add scheduled workflows and reorganize.

* Correction to HPC wikipedia link.

* Updates to the classifiers for setup.py

* Addition of long text setting.

* Correction of missing quote.

* Drop support for Python 3.4 (#218)

* Removal of enum34 and Python 3.4 classifiers.

PyYAML no longer supports Python 3.4 which is forcing Maestro to
also drop support as it has a direct dependency.

* Addition of python requirements, download url, and maintainer.

* Re-add py2.7. Note: py2.7 unofficially supported.

* Re-add enum34 for py2.7.

* Removal of py34 from tox tests.

* Removal of py3.4 from travis.

* Applies workspace substitution to the restart command. (#219)

Fixes #217 

* Sub in the new restart command.

* Addition of restart workspaces to sub.

* Fix for WORKSPACE substitutions into restart.

* Correction to override restart instead of cmd.

* Test/interfaces/lsf (#215)

* Implementation of a ScriptAdapter for the IBM LSF Scheduler.

Initial implementation of an LSF adapter.

Addition of LSF to the interface factory.

Correction to time format LSF correction.

Tweak to correct for casting

Further tweaks to the LSFScriptAdapter

Adjustments to the states the LSF adapter can return.

More tweaks to LSF states and status checking.

Update to the batch setting docstring

Tweak to make wallclock time entries two digits

Bugfix to the previous commit.

Signed-off-by: Francesco Di Natale <frank.dinatale1988@gmail.com>

* Addition of GPU support.

* Addition of a cancel method.

* Addition of reservation submissions.

* Tweak to use the -nrs flag for jsrun.

* Changes to resource allocation parameters.

* Removal of some batch headers for LSF.

* Tweak to error code for NOJOBS status.

* Tweak to skip lines that are part of prev status line.

* Correction of --nrs

* Correction of task batch key to nodes.

* Tweaks to _substitute_parallel_command.

Now only pass in step.run by copy and append the popped keys as
step resources "snodes" and "sprocs".

* Correction to LSF adapter to correct node specification.

* Tweaks to checking status of LSF jobs.

A tweak to formatting of the output for bjobs. With the new format
we get termination reason, which allows us to check for a timed
out status.

* Correction to bjobs formatting and nojob check.

* Corrections to how nodes and procs are being passed.

* Correction of the bkill command with multiple job ids.

* Tweaks to check_jobs for LSF adapter.

* Implementation of a ScriptAdapter for the IBM LSF Scheduler.

Initial implementation of an LSF adapter.

Addition of LSF to the interface factory.

Correction to time format LSF correction.

Tweak to correct for casting

Further tweaks to the LSFScriptAdapter

Adjustments to the states the LSF adapter can return.

More tweaks to LSF states and status checking.

Update to the batch setting docstring

Tweak to make wallclock time entries two digits

Bugfix to the previous commit.

Signed-off-by: Francesco Di Natale <frank.dinatale1988@gmail.com>

* Addition of GPU support.

* Addition of a cancel method.

* Tweak to use the -nrs flag for jsrun.

* Changes to resource allocation parameters.

* Removal of some batch headers for LSF.

* Tweak to error code for NOJOBS status.

* Tweak to skip lines that are part of prev status line.

* Correction of task batch key to nodes.

* Tweaks to _substitute_parallel_command.

Now only pass in step.run by copy and append the popped keys as
step resources "snodes" and "sprocs".

* Correction to LSF adapter to correct node specification.

* Tweaks to checking status of LSF jobs.

A tweak to formatting of the output for bjobs. With the new format
we get termination reason, which allows us to check for a timed
out status.

* Correction to bjobs formatting and nojob check.

* Corrections to how nodes and procs are being passed.

* Correction of the bkill command with multiple job ids.

* Tweaks to check_jobs for LSF adapter.

* Addition of LSF to key for LSFScriptAdapter.

* Correction of lsf key.

* Addition of a debug statement to catch status command.

* Correction of bjobs command.

* Additions to status checks.

* Rearraging some debug logging.

* Testing to see if .split works.

* Further LSF tweaks.

* Revert back to split with strip.

* Removal of -q option due to excessive filtering.

* Correction of a missed merge

* Correction to use new Records structures.

* Style fix for line 207.

* Correction to SubmissionRecord creation.

* Decode output in check_status to enforce str type.

* Decode output in submit.

* Addition of retcode to a logger statement.

* Sets log and err out for SLURM to parameterized step name (#220)

Fixes #213 

* First attempt at log name fix.

* Correction to header formatting.

* Update to dev0 to differentiate for pre-release.

* Ticked up version to 1.1.7dev1.

* Update Maestro logo link to full link for PyPi.

* Small Tweak to the rtd.io link

* An update to Maestro's description.

* Adding Neptune to the list on Planets (#222)

Signed-off-by: Adrien M. Bernede <bernede1@llnl.gov>

* Small README tweak.

* Updated the study step name in the README.md file (#227)

* Updated the package version in the Sphinx docs (#229)

* Added a link to Maestro's documentation (#231)

* Added a link to the documentation in the README.md

* Added a documentation section to the README.md

* Improve the performance of expansion (#232)

* Addition of override for ExectionGraph to not check cycles.

* Addition of documentation for justification of override.

* Addition of a newline due to style.

* Addition of dill as a dependency.

* Fix pickle and unpickle to use dill.

* Updated the description in the setup.py file (#233)

* Added dill to the requirements.txt file (#235)

* Fix to add PID to local log names. (#241)

* Refactor to move DAG operations to the back end Conductor (#242)

* Removal of SimObject references.

* Addition of PickleInterface.

* Derive Study and ExecutionGraph from PickleInterface.

* Some style clean up.

* Clean up unused dill import.

* Checking in to develop on another machine.

* Start of pre-check.

* Removal of precheck.

* Tweaks to Maestro and Conductor.

* Initial interface and refactor to Conductor class.

* Refactor of monitor_study

* Tweaks to backend conductor to use Conductor class.

* Tweaks to Maestro frontend to use Conductor class.

* Minor bug fixes in Conductor class.

* Minor tweaks to user cancel in Conductor class.

* Port status to the Conductor class.

* Continued additions to port to Conductor class.

* Slight fix to fix flake8 violation.

* Removal of named argument *, python2.7 does not support it.

* Refactor to remove parser and logging from Conductor class.

* Style clean up to fix flake8 errors.

* Updates to the docstrings for PickleInterface.

* Updates to the Conductor docstrings.

* Small flake8 error fix.

* Added pre-commit to enable flake8 checks (#244)

Added pre-commit to enable flake8 checks before a commit is accepted.

Also reordered requirements.txt to more
easily determine which are for development.

* Bugfix for logging that didn't appear in submodules (#247)

* Improved logging setup.

* Transition to a LoggerUtil class.

* Addition of docstring to LoggerUtility + cleanup.

* Added spec verification via jsonschema

added checks for valid keys

branch updates

working on validation

added schema file

updates

fixed spec

fixed spec

added jsonschema to deps

updates

ran black on yamlspecification.py

specified newest jsonschema version

added manifest

added include_package_data to setup.py

reformatted json

experimental package_data

fixed path

fixed path

fixed path again

reverted newline

added check for empty strings

reworked exception logic

implemented reviewer suggestions, shifted exception logic, renamed redundant variables

renamed variable

removed unused import

added missing `self.verify_environment()` call

Co-Authored-By: Francesco Di Natale <frank.dinatale1988@gmail.com>

paths and git dependencies are now array types

Co-Authored-By: Francesco Di Natale <frank.dinatale1988@gmail.com>

removed redundant logic

swapped number type to integer

moved env schema validation to top, which avoids some types of ambiguous errors

removed test yaml

removed some additionalProperties restrictions

unknown commit

removed debug print

* Reformatted and added color to logger messages. (#252)

Closes #248 

Added color to logging and converted some info messages to debug.

added colors and cleaned up logger

corrected formatting

added to dependencies

reverted message log level change

added debug format

debug logging format now works

flake8 fix

* Bug fix for unintentional variable section requirement. (#256)

Closes #255. A bug fix that corrects an unintentional assumption that the variable section in a specification will always exist.

* Update to broken venv documentation link.

* Addition of a simple dry-run capability. (#259)

* Addition of a simple dry-run capability.

* Addition of a DRYRUN state.

* Tweak to reduce sleep time for dry run.

* Renamed dryrun to dry to reduce redundancy.

* Enable autoyes when dry running is invoked.

* enabled raw sbatch errors to be logged (#262)

* enabled raw sbatch errors to be logged

* tweaks/correction suggested by Frank

* reduced line length

* fixed flake8 error in slurm-adapter

* Tweaks and fixes to the SLURM header. (#263)

* Tweaks and fixes to the SLURM header.

Adds the ability to specify GPUs, fixes reservation pairing with
accounts, and now uses a ChainMap to handle the internal key
conflicts between the batch and step settings. Also introduces
the exclusive key. Changed to full parameter names for clarity.

* Addition of chainmap package for python2.7

* A check to see if procs is included in the header.

Fixes #234 and includes ntasks in the header if the batch section
includes the key.

* modified executiongraph to round datetimes to nearest second

* Adds a check for UNKNOWN state. (#266)

Fixed #264 -- When testing #263, SLURM ran out of memory
during the completing stage and aborted the jobs and left the
job in an unknown state. This PR fixes this issue by defaulting to
a failure when the status is found to be UNKNOWN.

* Adds a check for UNKNOWN state.

* Correction of bad variable name "state"

* Tweak to treat UNKNOWN as failed.

* Change marking of failed state to unknown.

* Some fixes for style and credit from #265

* Official 1.1.7 release.

Co-authored-by: Elsa Gonsiorowski, PhD <gonsie@me.com>
Co-authored-by: robinson96 <robinson96@llnl.gov>
Co-authored-by: jsemler <624509+jsemler@users.noreply.github.com>
Co-authored-by: Kevin Athey <drizzt@acm.org>
Co-authored-by: Joe Koning <koning@users.noreply.github.com>
Co-authored-by: Adrien Bernede <51493078+adrienbernede@users.noreply.github.com>
Co-authored-by: Kevin Athey <athey1@llnl.gov>
Co-authored-by: Bay <bay1@llnl.gov>
Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com>
Co-authored-by: crkrenn <6069975+crkrenn@users.noreply.github.com>
Co-authored-by: Christopher R. Krenn <crkrenn@rztopaz188.llnl.gov>

v1.1.6

Additions that include:

• Enabling of loading specifications from streams and stream-like objects.
• Correction to dependency versions to fix known security issues in PyYAML
• Corrections to the construction of safe pathing of workspaces
• User enabled workspace hashing to counter long parameter lists
• Addition of automatic adapter detection
• A prototype generalized Flux adapter

v1.1.5dev: Addition of version information to package and command line (#205)

* Addition of version information.

* Tweak to have setup.py pull from __version__

* Addition of command line arg to print version.

v1.1.4dev1.1

Removal of _stage_linear since it is now not needed. (#156)

* Removal of _stage_linear since it is now not needed.

* Addition of linear LULESH samples.

* Update the dev to 1.1.

v1.1.4dev1.0

Update to setup.py to reflect dev version 1.0