v0.5.0

Highlights

✨ pytask v0.5.0 is released and contains two big features. ✨

🤖 Provisional Nodes and Task Generators

pytask now has mechanisms to define tasks that produce an unknown number of products, which are provisional nodes. For example, when a task splits data by rows into chunks of roughly 100MB.

If you want further to process each chunk of data with a separate task, use task generators to create as many tasks as there are chunks.

Everything is explained in the how-to guide Provisional Nodes and Task Generators.

🚀 Support for HPC and your favorite cloud provider (AWS, GCP, Azure) using dask / coiled

pytask can build your project on virtual machines from the cloud provider of your choice. Read more about it in the release announcement of pytask-parallel v0.5.0.

Removals

• The markers @pytask.mark.depends_on and @pytask.mark.produces are removed. Please define dependencies and products using the new approaches.
• Instead of @pytask.mark.task use from pytask import task and @task.

What's Changed

• Fix type hints for Task.execute() and TaskWithoutPath.execute(). by @tobiasraabe in #548
• [automated] Update plugin list by @github-actions in #550
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #549
• Remove depends_on and produces markers. by @tobiasraabe in #551
• Remove @pytask.mark.task and switch to @task. by @tobiasraabe in #552
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #554
• Forbid strings as values for paths in config file. by @tobiasraabe in #553
• Use new-style hook wrappers and require pluggy 1.3. by @tobiasraabe in #555
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #556
• Fix issue with @task(after=...) in notebooks and terminals. by @tobiasraabe in #557
• Fix path in "Node is dir" error message by @egerlach in #561
• Bump the github-actions group with 2 updates by @dependabot in #560
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #559
• Make universal-pathlib 0.2.2 an official dependency. by @tobiasraabe in #566
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #562
• Bump the github-actions group with 1 update by @dependabot in #564
• Fix typos in defining_dependencies_products.md by @ChristianZimpelmann in #563
• Improve handling of task_files. by @tobiasraabe in #568
• Remove hooks related to the DAG. by @tobiasraabe in #569
• Remove redundant calls of PNode.state(). by @tobiasraabe in #571
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #573
• Remove pytask_execute_create_scheduler hook. by @tobiasraabe in #575
• Implement task generators and provisional nodes. by @tobiasraabe in #487
• Fix interaction with --pdb, --trace and tasks that return. by @tobiasraabe in #579
• Simplify the code related to tracebacks. by @tobiasraabe in #581
• [automated] Update plugin list by @github-actions in #584
• Improve typing of capture.py. by @tobiasraabe in #587
• Improve linting. by @tobiasraabe in #586
• Reset class variables of ExecutionReport and Traceback. by @tobiasraabe in #588
• Fix error introduced by #588. by @tobiasraabe in #590
• Invalidate cache to check whether remote files exist. by @tobiasraabe in #591
• Resolve root paths and module names for imported files. by @tobiasraabe in #589
• Use uv to speed up CI. by @tobiasraabe in #567
• Bump the github-actions group with 1 update by @dependabot in #578
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #577
• Recreate PythonNodes every time. by @tobiasraabe in #593
• Publish NodeLoadError. by @tobiasraabe in #594
• Manage project with rye. by @tobiasraabe in #596
• Bump peter-evans/create-pull-request from 6.0.2 to 6.0.3 in the github-actions group by @dependabot in #597
• Replace requests with httpx. by @tobiasraabe in #598
• Stop unwrapping coiled functions. by @tobiasraabe in #595
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #592
• Add fixture for changing cwd. by @tobiasraabe in #599
• Refactor tests using subprocesses. by @tobiasraabe in #600
• Bump peter-evans/create-pull-request from 6.0.3 to 6.0.4 in the github-actions group by @dependabot in #602
• [automated] Update plugin list by @github-actions in #601
• Fix example about capturing warnings in docs. by @tobiasraabe in #603
• Fix PythonNode examples in docs. by @tobiasraabe in #604
• Improve infra and add step for type checking. by @tobiasraabe in #605
• Bump peter-evans/create-pull-request from 6.0.4 to 6.0.5 in the github-actions group by @dependabot in #608
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #607
• Allow tasks to be pending. by @tobiasraabe in #609
• Remove status from pytask_execute_task_log_start. by @tobiasraabe in #611
• Validate the names of data catalogs. by @tobiasraabe in #612
• Improve documentation for data catalogs. by @tobiasraabe in #606
• [automated] Update plugin list by @github-actions in #613

New Contributors

• @egerlach made their first contribution in #561
• @ChristianZimpelmann made their first contribution in #563

Full Changelog: v0.4.5...v0.5.0