Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for specifying worker version #717

Merged
merged 15 commits into from
Sep 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,27 +52,27 @@ To work with the repo and contribute changes, the basic process is as follows:

## Getting Started

In order to be able to contribute code changes, you will first need to [create a Github fork](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/fork-a-repo) of the official OWP repo.
In order to be able to contribute code changes, you will first need to [create a Github fork](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/fork-a-repo) of the official OWP repo.

Next, set up your authentication mechanism with Github for your command line (or IDE). You can either [create an SSH key pair](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent) and [add the public key](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account) to your Github account, or you can set up a [Personal Access Token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#using-a-personal-access-token-on-the-command-line) if you plan to clone the repo locally via HTTPS.
Next, set up your authentication mechanism with Github for your command line (or IDE). You can either [create an SSH key pair](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent) and [add the public key](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account) to your Github account, or you can set up a [Personal Access Token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#using-a-personal-access-token-on-the-command-line) if you plan to clone the repo locally via HTTPS.

After that, [clone a local development repo](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository) from your fork, using a command similar to one of the following:
After that, [clone a local development repo](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository) from your fork, using a command similar to one of the following:

# SSH-based clone command. Change URL to match your fork as appropriate
git clone git@github.com:your_user/DMOD.git dmod

# HTTPS-based clone command. Change URL to match your fork as appropriate
git clone https://github.com/your_user/DMOD.git dmod

You can now change directories into the local repo, which will have the default branch - `master` - checked out.
You can now change directories into the local repo, which will have the default branch - `master` - checked out.

# Move into the repo directory "dmod"
cd dmod

# You can verify the branch by examining the output of ...
git status

> [!IMPORTANT]
> [!IMPORTANT]
> Git's will add a [Git remote](https://git-scm.com/book/en/v2/Git-Basics-Working-with-Remotes) named `origin` to the clone's configuration that points to the cloned-from repo. Because of this, the recommended convention is to clone your local repo(s) from your personal fork, thus making `origin` point to your fork. This is assumed to be the case in other parts of the documentation.

Next, add the upstream OWP DMOD repo as a second remote for the local clone. The standard convention used in this doc and elsewhere is to name that remote `upstream`. Doing the addition will look something like:
Expand Down Expand Up @@ -130,7 +130,7 @@ Especially if making more frequent, smaller commits as suggested above, it is a

# The fetch is probably unnecesssary unless you work from multiple local repos
git fetch

# Assuming your branch of interest is still checked out:
git status

Expand All @@ -143,7 +143,7 @@ Once a code contribution is finished, make sure all changes have been pushed to

#### PR Review and Requested Revisions

Once the PR is submitted, it will be reviewed by one or more other repo contributors. Often conversations will be had within the Github PR if reviewers have questions or request revisions be made to the proposed changes. If revisions are requested, you will need to make those in your locally copy of the feature/fix branch, and then re-push that branch (and the updates) to your personal fork. Then, use the PR page in Github to re-request review.
Once the PR is submitted, it will be reviewed by one or more other repo contributors. Often conversations will be had within the Github PR if reviewers have questions or request revisions be made to the proposed changes. If revisions are requested, you will need to make those in your locally copy of the feature/fix branch, and then re-push that branch (and the updates) to your personal fork. Then, use the PR page in Github to re-request review.

## Keeping Forks Up to Date

Expand Down
20 changes: 10 additions & 10 deletions doc/GIT_USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,17 +10,17 @@ Note that this document goes into detail on the Git strategy and branching model

## Branching Model

- The DMOD repo uses a branching model based on [Gitflow](https://nvie.com/posts/a-successful-git-branching-model/) that has two primary long-term branches:
- The DMOD repo uses a branching model based on [Gitflow](https://nvie.com/posts/a-successful-git-branching-model/) that has two primary long-term branches:
- **master**: the main development and integration branch containing the latest completed development work intended for the next released version
- **production**: the branch representing the latest code verified as production-ready and pointing to the most recently release, official version
- Rebasing is used to integrate changes across branches, rather than merge commits
- This allows the repo to maintain a more robust and complete history
- Most interaction with the official OWP DMOD repo is done via pull requests (PRs) to the `master` branch
- Independent branches for features or bug fixes are created off `master` to contain development work that is in progress
- Once work in a feature/fix branch is complete (or at least thought complete), it is used to create a PR
- Once work in a feature/fix branch is complete (or at least thought complete), it is used to create a PR
- PRs and their linked branches are reviewed and, once approved, have their changes integrated back into `master`
- Typically feature/fix branches exist in personal clones and personal Github forks, but not in the official OWP repo
- Release branches (e.g., `release-X` for pending version `X`) will be created whenever it is time to officially release a new version
- Release branches (e.g., `release-X` for pending version `X`) will be created whenever it is time to officially release a new version
- These effectively are release candidates, with branches created from `master`
- The release branches are managed by the core OWP contributors team
- They do exist in the official OWP repo
Expand All @@ -30,7 +30,7 @@ Note that this document goes into detail on the Git strategy and branching model
### Feature Branches from `master`
This illustrates the relationship between feature branches and `master`. They should be created from `master` and independently contain commits from their feature. Once done, the changes will be reintegrated back into `master` via rebasing.

```mermaid
```mermaid
%%{init: { 'logLevel': 'debug', 'theme': 'base', 'gitGraph': { 'showBranches': true, 'showCommitLabel':true, 'mainBranchName': 'master'}}}%%
gitGraph
commit id:"feature1.1"
Expand All @@ -52,7 +52,7 @@ This illustrates the relationship between feature branches and `master`. They s

The resulting state of `master` after rebasing the two new feature branches would be:

```mermaid
```mermaid
%%{init: { 'logLevel': 'debug', 'theme': 'base', 'gitGraph': { 'showBranches': true, 'showCommitLabel':true, 'mainBranchName': 'master'}}}%%
gitGraph
commit id:"feature1.1"
Expand All @@ -69,7 +69,7 @@ The resulting state of `master` after rebasing the two new feature branches woul
This illustrates the relationship between `production`, `master`, and `release-v2`. Notice that `production` has already been tagged with version `v1` at the start. Commits for `feature1` and `feature2` at some point are integrated into `master`. When it is time to prepare to release version `v2`, `release-v2` is created. A few bug fix commits were needed in `release-v2`. After that, all the changes in `release-v2` are integrated into `production`, and `production` is tagged `v2`. All the changes are also integrated back into `master`.


```mermaid
```mermaid
%%{init: { 'logLevel': 'debug', 'theme': 'base', 'gitGraph': { 'showBranches': true, 'showCommitLabel':true, 'mainBranchName': 'master'}}}%%
gitGraph
commit id:"v1-commit"
Expand All @@ -95,7 +95,7 @@ This illustrates the relationship between `production`, `master`, and `release-v

The resulting state of `production` is:

```mermaid
```mermaid
%%{init: { 'logLevel': 'debug', 'theme': 'base', 'gitGraph': { 'showBranches': true, 'showCommitLabel':true, 'mainBranchName': 'production'}}}%%
gitGraph
commit id:"v1-commit" tag:"v1"
Expand All @@ -110,7 +110,7 @@ The resulting state of `production` is:

The resulting state of `master` is essentially the same:

```mermaid
```mermaid
%%{init: { 'logLevel': 'debug', 'theme': 'base', 'gitGraph': { 'showBranches': true, 'showCommitLabel':true, 'mainBranchName': 'master'}}}%%
gitGraph
commit id:"v1-commit"
Expand Down Expand Up @@ -160,12 +160,12 @@ Once the utility is available, install the _pre-commit_-configured hook scripts
pre-commit install
```

The hook scripts will now run when code is committed.
The hook scripts will now run when code is committed.

Alternatively, you can run the hook scripts manually via:

```shell
pre-commit run --all-files
```

For more information, see [_pre-commit_'s documentation](https://pre-commit.com/index.html).
For more information, see [_pre-commit_'s documentation](https://pre-commit.com/index.html).
6 changes: 3 additions & 3 deletions doc/RELEASE_MANAGEMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,11 +39,11 @@ The release process for DMOD can be summarized fairly simply:

# Versions

The versioning for DMOD is a little complicated.
The versioning for DMOD is a little complicated.

DMOD contains the sources of several independently-versioned Python packages; e.g., `dmod-core-0.19.0`. As long as this code remains organized as multiple separate packages, the package versions need to continue to be maintained individually.
DMOD contains the sources of several independently-versioned Python packages; e.g., `dmod-core-0.19.0`. As long as this code remains organized as multiple separate packages, the package versions need to continue to be maintained individually.

DMOD contains other source code wholly separate from these package, such as helper scripts, Dockerfiles, stack configurations, and other files. These are not contained in some inner organizational unit with its own versioning, and many (if not all) of them are particularly relevant to DMOD deployment.
DMOD contains other source code wholly separate from these package, such as helper scripts, Dockerfiles, stack configurations, and other files. These are not contained in some inner organizational unit with its own versioning, and many (if not all) of them are particularly relevant to DMOD deployment.

As such, DMOD utilizes another, independent versioning scheme for itself as a whole.

Expand Down
10 changes: 5 additions & 5 deletions docker/main/ngen/customize/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Locally Customizing the ngen Worker

It simply isn't possible to bundle every possible ngen-compatible BMI module into the job worker image Dockerfile. As such, the source Dockerfile doesn't try to include everything and only integrates a small number of OWP-developed BMI modules.
It simply isn't possible to bundle every possible ngen-compatible BMI module into the job worker image Dockerfile. As such, the source Dockerfile doesn't try to include everything and only integrates a small number of OWP-developed BMI modules.

But, it is possible to use other BMI modules outside this small subset in the DMOD job worker images.

Expand All @@ -15,7 +15,7 @@ In summary, it is possible to:

## Supply `requirements.txt` File

If a `requirements.txt` file is present within this directory, it will be used by an additional call to `pip` during the image build process, installing the Python packages listed within the file. This is likely the easiest way to incorporate more Python BMI modules, as long as they are publicly accessible.
If a `requirements.txt` file is present within this directory, it will be used by an additional call to `pip` during the image build process, installing the Python packages listed within the file. This is likely the easiest way to incorporate more Python BMI modules, as long as they are publicly accessible.

Keep in mind that, even if ready-made packages are not available via something like PyPI, `pip` supports [installing directly from version control systems](https://pip.pypa.io/en/stable/topics/vcs-support/) like Git.

Expand All @@ -30,12 +30,12 @@ For this to work for a provided Git repo, a few conditions must hold true:
- the Git repository must be accessible at the given URL anonymously
- the script doesn't provide a branch, so whatever the default branch is (e.g., `master` or `main`) must be suitable
- the contents of the repo must be set up to build with **CMake**
- no extra, deliberate configuration of **CMake** variables should be necessary
- no extra, deliberate configuration of **CMake** variables should be necessary
- (except `CMAKE_BUILD_TYPE` and `CMAKE_INSTALL_PREFIX`, which are pre-set in the script)
- running `cmake --build <build_dir>` will build anything/everything required
- i.e., it must not be necessary to build a specific **CMake** `target`

## Use Manual Customization Script
## Use Manual Customization Script

If the above methods are insufficient, it is possible to write a more bespoke script for configuring whatever customization is needed within the image, while also avoiding commiting this script directly to the repo. This allows for finer-grained control but also puts more responsibility on the user. To do this:

Expand All @@ -45,4 +45,4 @@ If the above methods are insufficient, it is possible to write a more bespoke sc
1. `/dmod/bin/` for executables
2. `/dmod/bmi_module_data/` for static module data and configs
3. `/dmod/shared_libs/` for compiled shared libraries
4. Note that there is also a Python virtual environment at `/dmod/venv/`, though this should be active in the environment when the script is run; i.e., installing packages using `pip` should get things there without any extra steps
4. Note that there is also a Python virtual environment at `/dmod/venv/`, though this should be active in the environment when the script is run; i.e., installing packages using `pip` should get things there without any extra steps
2 changes: 2 additions & 0 deletions python/lib/client/dmod/client/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@ def _create_ngen_based_exec_parser(subcommand_container: Any, parser_name: str,
The newly created and associated subparser.
"""
new_parser = subcommand_container.add_parser(parser_name)
new_parser.add_argument('--worker-version', dest='worker_version', default="latest",
help="Specify version of worker (e.g., Docker image tag) to use.")
new_parser.add_argument('--partition-config-data-id', dest='partition_cfg_data_id', default=None,
help='Provide data_id for desired partition config dataset.')
paradigms = [p for p in AllocationParadigm]
Expand Down
2 changes: 1 addition & 1 deletion python/lib/client/dmod/client/_version.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = '0.9.0'
__version__ = '0.10.0'
2 changes: 1 addition & 1 deletion python/lib/client/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ dependencies = [
"dmod.core>=0.17.0",
"websockets>=8.1",
"pydantic>=1.10.8,~=1.10",
"dmod.communication>=0.20.0",
"dmod.communication>=0.22.0",
"dmod.externalrequests>=0.6.0",
"dmod.modeldata>=0.12.0",
]
Expand Down
2 changes: 1 addition & 1 deletion python/lib/communication/dmod/communication/_version.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = '0.21.1'
__version__ = '0.22.0'
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
from .partial_realization_config import PartialRealizationConfig
from abc import ABC, abstractmethod
from typing import List, Optional
from pydantic import PrivateAttr
from pydantic import Field, PrivateAttr

from dmod.core.meta_data import (
DataCategory,
Expand Down Expand Up @@ -37,6 +37,7 @@ class AbstractNgenRequest(DmodJobRequest, ABC):
"""

request_body: NGENRequestBody
worker_version: str = Field("latest", description="The desired version of the applicable worker for the request.")

_hydrofabric_data_requirement = PrivateAttr(None)
_forcing_data_requirement = PrivateAttr(None)
Expand Down
Loading
Loading