Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: enable template histogram creation and collection to accept list of tasks #421

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

alexander-held
Copy link
Member

@alexander-held alexander-held commented Jul 6, 2023

This updates the templates.build, templates.collect, and templates.postprocess implementations to accept a list of templates to produce via a new template_list keyword argument. The full list can be obtained via route.required_templates. By default, all templates will still be processed.

This allows users to parallelize the template histogram production / collection, partially addressing #401. An example of this would be the following:

from dask.distributed import Client, LocalCluster, wait

def produce_single_template(template):
    cabinetry.templates.build(config, template_list=[template])

template_list = cabinetry.route.required_templates(config)

with LocalCluster(n_workers=2) as cluster:
    client = Client(cluster)
    wait(client.map(produce_single_template, template_list))

compared to the non-parallelized version, which is unchanged:

cabinetry.templates.build(config)

breaking change:

  • renamed route.apply_to_all_templates to route.apply_to_templates and changed signature: now takes a list of tasks, which can be generated by route.required_templates
* templates.build, templates.collect, and templates.postprocess now accept a list of tasks
* breaking change: route. apply_to_all_templates renamed to route.apply_to_templates
* breaking change: route.apply_to_templates now requires a list of tasks as argument

Comment on lines +43 to +45
TemplateHistogramInformation = Tuple[
Dict[str, Any], Dict[str, Any], Dict[str, Any], Optional[Literal["Up", "Down"]]
]
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This probably would be good to replace by a lightweight class to wrap the information.

@alexander-held alexander-held force-pushed the feat/histogram-production-refactor branch from 1d83bf7 to fd93dbb Compare September 19, 2023 18:00
@alexander-held alexander-held force-pushed the feat/histogram-production-refactor branch from 1d29183 to 2f75c8a Compare October 5, 2023 02:47
@alexander-held alexander-held force-pushed the feat/histogram-production-refactor branch from 2f75c8a to 5536dd8 Compare November 24, 2023 15:54
@alexander-held
Copy link
Member Author

This will require a different approach to handle remote workers without access to a shared filesystem. To handle this, histograms would have to be passed back and saved at the head node. The current API does not surface the call to utils._name_and_save, which is quite deep inside internal APIs.

Currently route.apply_to_templates does generic function execution. This might have to be changed to support handling return values and passing those back to the higher level, e.g. templates.build for saving. The downside of this is that saving would only happen (in the non-parallel version) after all templates are constructed (and passed back as histograms).

to-do / think about:

  • is writing at the very end maybe be a problem if some late templates crash
  • memory (or other) issues from keeping all histograms in memory?
  • writing only once at the very end means all histograms can easily be written into a single file (this is a bigger refactor)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant