Cleanup and centralize ngen-related images #408
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Move ngen entrypoint args to use of explicit flags and to use of single composite config dataset arg/name/directory.
Updating entrypoint.sh to copy various job-related configs into output dataset for record keeping and reproducibility.
Updating Dockerfile to give name to the build stage used as main "ngen" worker image for model exec jobs, and updating docker-build.yml build setup accordingly to explicitly target this stage.
Moving reusable functions and logic not (yet) in functions from ngen entrypoint.sh to functions within new funcs.sh, and sourcing the new file from the entrypoint.
Moving entrypoint script (and renaming) for partitioner image to ngen directory/build context in preparation for working from the same Dockerfile.
Updating rocky-ngen-deps build stage in ngen Dockerfile so that Boost root directory (after downloaded Boost headers are untarred) is chowned to USER value, so the directory can be deleted later.
Fixing function to actually load the secrets for ACCESS and SECRET to variables.
Creating custom build stage for use as the ngen-partitioner image target.
Updating docker-build.yml to use context and Dockerfile from main ngen image and specify new custom target set up for partitioner image.
Removing old stand-alone partitioner Dockerfile after moving to stage within main ngen Dockerfile.
Adjusting entrypoint setup to reduce layers and use standardized name for entrypoint script once it is inside the image/container.
Adjusting entrypoint setup to use standardized name for entrypoint script once it is inside the image/container.
Moving and renaming entrypoint script for ngen_cal image to be within context directory for ngen image build, in preparation for utilizing a stage within the main Dockerfile for the calibration image.
Fixing logic in script for how things were handled when running a serial execution.
Fixing issue in funcs.sh related to when to check for a partition config dataset directory, which previously would only happen if the MPI node count was greater than 1 (i.e., while an MPI job running all on a single node would still expect the partition config).
Updating main ngen Dockerfile with new build stage for use as target for ngen_cal worker image.
robertbartel
force-pushed
the
f/update_ngen_docker/cleanup_and_centralize
branch
from
1afeb5d
to
498466a
Compare
aaraney
approved these changes
Sep 5, 2023
robertbartel
deleted the
f/update_ngen_docker/cleanup_and_centralize
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Combining the builds for ngen, ngen-cal, and ngen partitioning into a single Dockerfile and build context. Differentiating them through targeted Docker build stages. Updating the scripts used by the images to combine parts of reusable logic, and to make other general improvements.
Depends on #405 and should not be merged until after that.