0.12.0

This release includes a wide range of performance improvements and internal refactoring that will enable the addition of an "aggregation" feature in subsequent releases (not yet available). Performance of a sample workflow that checks status, runs, and submits a FlowProject with 1000 jobs, 3 operations, and 2 label functions has improved roughly 4x compared to the 0.11.0 release.

Added

• Code is formatted with black and isort pre-commit hooks (#365).
• Add official support for Python version 3.9 (#365).
• Documentation has been added for all public classes and methods (#387, #389).
• Added internal support for aggregates of jobs (#334, #348, #351, #364, #383, #390, #415, #422, #430).
• Added code coverage to continuous integration (#405).

Changed

• Command line interface always uses --job-id instead of --jobid (#363, #386).
• CPUEnvironment and GPUEnvironment classes are deprecated (#381).
• Docstrings are now written in numpydoc style (#392).
• Default environment for the University of Minnesota Mangi cluster changed from SLURM to Torque (#393).
• Run commands are evaluted lazily (#70, #396).
• Deprecated method export_job_statuses (#402).
• Improved internal caching of scheduler status (#410).
• Refactored status fetching code (#368, #417).
• Optimization: Directives are no longer deep-copied (#420, #421).
• The use_buffered_mode config option is deprecated. Buffering is always internally enabled (#425).
• Evaluate directives when called instead of when defined (#398, #402).
• Various internal refactorings and optimizations (#371, #373, #374, #375, #376, #377, #378, #379, #380, #400, #410, #416, #423, #426).
• Scheduler is now an abstract base class (#426).
• flow.scheduling.fakescheduler has been renamed to flow.scheduling.fake_scheduler (#426).
• Arguments to submit have been changed for all scheduler classes (#426).
• Python 3.6 is only tested with oldest dependencies (#436).
• Drop support for tqdm versions older than 4.48.1 (#436, #440).
• Drop support for Jinja2 versions older than 2.10.0 (#436).

Fixed

• Ensure that directives are always evaluated before running or submitting (#408, #409).
• Cache the fully qualified domain name during environment detection to fix a performance issue on macOS (#339, #394).
• Ensure that next CLI command displays eligible jobs for the exact operation name provided (#443).

Removed

• Removed the deprecated method flow.util.misc.write_human_readable_statepoints (#397).
• Removed the deprecated argument --no-parallelize (#424).
• Removed the deprecated env argument from submission methods (#424).
• flow.render_status.Renderer class has been removed. FlowProject.print_status no longer returns the renderer (#426).
• Removed deprecated status.py module (#426).
• Removed the --test argument from FlowProject.submit (#439).

0.11.0

Added

• Added classes _Directives and _Directive that serve as a smart mapping for directives specified by the environment or user (#265, #283).
• Added support for pre-commit hooks (#333).
• Add environment profile for University of Minnesota, Minnesota Supercomputing Institute, Mangi supercomputer (#353).

Changed

• Make FlowCondition class private (#307, #315).
• Deprecate JobOperation class, make SubmissionJobOperation a private class and deprecate the following methods of FlowProject: script, run_operations, submit_operations, next_operations. (#313)
• Deprecate the following methods: FlowGroup.eligible, FlowGroup.complete, BaseFlowOperation.eligible, BaseFlowOperation.complete (#337).

Fixed

• Serial execution on Summit correctly counts total node requirements (#342).
• Fixed performance regression in job submission in large workspaces (#354).

Removed

• Drop support for Python 3.5 (#305). The signac project will follow the NEP 29 deprecation policy going forward.
• Remove the deprecated methods always, make_bundles, and JobOperation.get_id (#312).

0.10.1

Fixed

• Fix issue with the submission of bundled operations on cluster environments that do not allow slashes ('/') in cluster scheduler job names (#343).

0.10.0

Added

• Add FlowGroup (one or more operations can be grouped within an execution environment) (#114).
• Add official support for University of Michigan Great Lakes cluster (#185).
• Add official support for Bridges AI cluster (#222).
• Add IgnoreConditions option for submit(), run() and script() (#38).
• Add pytest support for Testing Framework (#227, #232).
• Add markdown and html format support for print_status() (#113, #163).
• Add memory flag option for default Slurm scheduler (#256).
• Add optional environment variable to specify submission script separator (#262).
• Add status_parallelization configuration to specify the parallelization used for fetching status (#264, #271).

Changed

• Raises ValueError when an operation function is passed to FlowProject.pre() and FlowProject.post(), or a non-operation function passed to FlowProject.pre.after() (#248, #249).
• The option to provide the env argument to submit and submit_operations has been deprecated (#245).
• The command line option --cmd for script has been deprecated and will trigger a DeprecationWarning upon use until removed (#243, #218).
• Raises ValueError when --job-name is passed by the user because that interferes with status checking (#164, #241).
• Submitting with --memory no longer assumes a unit of gigabytes on Bridges and Comet clusters (#257).
• Buffering is enabled by default, improving the performance of status checks (#273).
• Deprecate the use of no_parallelize argument while printing status (#264, #271).
• Submission via the command-line interface now calls the FlowProject.submit function instead of bypassing it for FlowProject.submit_operations (#238, #286).
• Updated Great Lakes GPU request syntax (#299).

Fixed

• Ensure that label names are used when displaying status (#263).
• Fix node counting for large resource sets on Summit (#294).

Removed

• Removed ENVIRONMENT global variable in the flow.environment module (#245).
• Removed vendored tqdm module and replaced it with a requirement (#247).

0.9.0

Added

• Add official support for Python version 3.8 (#190, #210).
• Add descriptive error message when tag is not set and cannot be autogenerated for conditions (#195).
• Add "fork" directive to enforce the execution of operations within a subprocess (#159).
• Operation graph detection based on function comparison (#178).
• Exceptions raised during operations always show tracebacks of user code (#169, #171).

Changed

• Raise a warning when a condition's tag is not set and raise an error if this occurs during graph detection (#195).
• Raise errors if a forked process or @cmd operation returns a non-zero exit code. (#170, #172).

Removed

• Drop support for Python version 2.7 (#157, #158, #201).
• The "always" condition has been deprecated and will trigger a DeprecationWarning upon use until removed (#179).
• Removed deprecated UnknownEnvironment in favor of StandardEnvironment (#204).
• Removed support for decommissioned INCITE Titan and Eos computers (#204).
• Removed support for the legacy Python-based submission script generation (#200).
• Removed legacy compatibility layers for Python 2, signac < 1.0, and soft dependencies (#205).
• Removed deprecated support for implied operation names with the run command (#205).

0.8.0

Added

• Add feature for integrated profiling of status updates (status --profile) to aid with the optimization of a FlowProject implementation (#107, #110).
• The status view is generated with Jinja2 templates and thus more easily customizable (#67, #111).
• Automatically show an overview of the number of eligible jobs for each operation in status view (#134).
• Allow the provision of multiple operation-functions to the pre.after and *.copy_from conditions (#120).
• Add option to specify the operation execution order (#121).
• Add a testing module to easily initialize a test project (#130).
• Enable option to always show the full traceback with show_traceback = on within the [flow] section of the signac configuration (#61, #144).
• Add full launcher support for job submission on XSEDE Stampede2 for large parallel single processor jobs (#85, #91).

Fixes

• Both the nranks and omp_num_threads directives properly support callables (#118).
• Show submission error messages in combination with a TORQUE scheduler (#103, #104).
• Fix issue that caused the "Fetching operation status" progress bar to be inaccurate (#108).
• Fix erroneous line in the torque submission template (#126).
• Ensure default parameter range detection in status printing succeeds for nested state points (#154).
• Fix issue with the resource set calculation on INCITE Summit (#101).

Changed

• Packaged environments are now available by default. Set import_packaged_environments = off within the [flow] section of the signac configuration to revert to previous behavior.

• The following methods of the FlowProject class have been deprecated and will trigger a DeprecationWarning upon use until their removal:

  • classify (use labels() instead)
  • next_operation (use next_operations() instead)
  • export_job_stati (replaced by export_job_statuses)
  • eligible_for_submission (removed without replacement)
  • update_aliases (removed without replacement)

• The support for Python version 2.7 is deprecated.

Removed

• The support for Python version 3.4 has been dropped.
• Support for signac version 0.9 has been dropped.

0.7.1

Added

• Add function to automatically print all varying state point parameters in the detailed status view triggered by providing option -p/--parameters without arguments (#19, #87).
• Add clear environment notification when submitting job scripts (#43, #88).

Fixes

• Fix issue where the scheduler status of job-operations would not be properly updated for ineligible operations (#96).

Fixes (compute environments)

• Fix issue with the TORQUE scheduler that occured when there was no job scheduled at all on the system (for any user) (#92, #93).

Changed

• The performance of status updates has been significantly improved (up to a factor of 1,000 for large data spaces) by applying a more efficient caching strategy (#94).

0.7.0

Added

• Add legend explaining the scheduler-related symbols to the detailed status view (#68).
• Allow the specification of the number of tasks per resource set and additional jsrun arguments for Summit scripts.

Fixes (general)

• Fixes issue where callable cmd-directives were not evaluated (#47).
• Fixes issue where the source file of wrapped functions was not determined correctly (#55).
• Fix a Python 2.7 incompatibility and another unrelated issue with the TORQUE scheduler driver (#54, #81).
• Fixes issue where providing the wrong argument type to Project.submit() would go undetected and lead to unexpected behavior (#58).
• Fixes issue where using the buffered mode would lead to confusing error messages when condition-functions would raise an AttributeError exception.
• Fixes issue with erroneous unused-directive-keys-warning.

Fixes (compute environments)

• Fixes issues with the Summit environment resource set calculation for parallel operations under specific conditions (#63).
• Fix the node size specified in the template for the ORNL Eos system (#77).
• Fixes issue with a missing --gres directive when using the GPU-shared partition on the XSEDE Bridges system (#59).
• Fixed University of Michigan Flux hostname pattern to ignore the Flux Hadoop cluster (#82).
• Remove the Ascent environment (host decommissioned).

Note: The official support for Python 3.4 will be dropped beginning with version 0.8.0.