Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: async app CLI #159

Merged
merged 2 commits into from
Mar 8, 2024
Merged

feature: async app CLI #159

merged 2 commits into from
Mar 8, 2024

Conversation

ClemDoum
Copy link
Collaborator

@ClemDoum ClemDoum commented Feb 2, 2024

PR description

Create the async app CLI, which is the counter part of the HTTP service for async tasks.
The CLI can be used to deploy an async worker pool running async tasks.

For now the CLI only contains one endpoint to display the package version:

icij-worker --version

and another one to start a worker pool.

When launching a worker pool the name of the async app to run has to be provided, the worker configuration will be read from env variables (starting by ICIJ_WORKER_).

icij-worker workers start "neo4j_app.tests.conftest.APP"

otherwise more options can be provided:

icij-worker workers start -c worker_config.json -n 2 --backend multiprocessing neo4j_app.tests.conftest.APP

Later the CLI could be enriched with utils to stop workers, list workers running on the machine, expose the async task defined in on the async app and so on.

Changes

neo4j-app

Added

  • implemented the async app CLI using typer. Implemented the icij-worker workers start and icij-worker --version CLI entry points

Changed

  • refactored the multiprocessing backend to avoid context manager usage when running the CLI (context manager are necessary for when starting the worker pool from the HTTP service)
  • updated RegistrableMixin to use the Registrable.registry_key attribute value when no name is provided to register a component
  • implemented RegistrableConfig.from_env and updated implemented RegistrableConfig.parse_file so that registered instances of RegistrableConfig subclasses can be directly created from the generic RegistrableConfig.from_env/parse_file. The env or the file are inspected to read the registry key and the appropriate subclass is retrieved from the registry and instanciated
  • moved setup_loggers to neo4j_app.core.utils.logging
  • renamed the http service loggers_enter dependency to http_loggers_enter

Fixed

  • updated Worker.work_forever to swallow KeyboardInterrupt and exit gracefully so that dependencies teardown can complete (this update was necessary due to an update in signel handling in chore: isolate the async app and HTTP app #157)

Left for later

  • worker pool stop, pgrep/pkill can be used for now

@ClemDoum ClemDoum changed the title feature: worker pool CLI feature: async app CLI Feb 2, 2024
@ClemDoum ClemDoum force-pushed the feature/worker-pool-cli branch 2 times, most recently from 03645e0 to 03369ed Compare February 2, 2024 14:24
@ClemDoum ClemDoum self-assigned this Feb 2, 2024
@ClemDoum ClemDoum requested a review from a team February 2, 2024 14:35
@ClemDoum ClemDoum marked this pull request as ready for review February 2, 2024 14:35
@ClemDoum ClemDoum force-pushed the feature/worker-pool-cli branch from 03369ed to 22326da Compare March 8, 2024 14:05
@ClemDoum ClemDoum force-pushed the feature/worker-pool-cli branch from 22326da to 2aad0f5 Compare March 8, 2024 14:14
@ClemDoum ClemDoum merged commit b46f3ca into main Mar 8, 2024
8 checks passed
@ClemDoum ClemDoum deleted the feature/worker-pool-cli branch March 8, 2024 14:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant