Cleanup and centralize ngen-related images #408
Commits on Aug 28, 2023
-
Updates to ngen Docker image entrypoint script.
Move ngen entrypoint args to use of explicit flags and to use of single composite config dataset arg/name/directory.
-
Have ngen entrypoint copy configs to output.
Updating entrypoint.sh to copy various job-related configs into output dataset for record keeping and reproducibility.
-
-
Explicit build stage name for ngen Docker image.
Updating Dockerfile to give name to the build stage used as main "ngen" worker image for model exec jobs, and updating docker-build.yml build setup accordingly to explicitly target this stage.
-
Move ngen entrypoint common logic and functions.
Moving reusable functions and logic not (yet) in functions from ngen entrypoint.sh to functions within new funcs.sh, and sourcing the new file from the entrypoint.
-
-
-
-
Move partitioner image entrypoint.
Moving entrypoint script (and renaming) for partitioner image to ngen directory/build context in preparation for working from the same Dockerfile.
-
-
Update ngen build to let Boost be cleaned up.
Updating rocky-ngen-deps build stage in ngen Dockerfile so that Boost root directory (after downloaded Boost headers are untarred) is chowned to USER value, so the directory can be deleted later.
-
Fix ngen funcs for handling object store secrets.
Fixing function to actually load the secrets for ACCESS and SECRET to variables.
-
Create ngen Dockerfile partitioner build stage.
Creating custom build stage for use as the ngen-partitioner image target.
-
Update ngen-partitioner compose build config.
Updating docker-build.yml to use context and Dockerfile from main ngen image and specify new custom target set up for partitioner image.
-
Delete obsolete ngen-partitioner Dockerfile.
Removing old stand-alone partitioner Dockerfile after moving to stage within main ngen Dockerfile.
-
Adjust entrypoint in partitioner image.
Adjusting entrypoint setup to reduce layers and use standardized name for entrypoint script once it is inside the image/container.
-
Adjust entrypoint in ngen image.
Adjusting entrypoint setup to use standardized name for entrypoint script once it is inside the image/container.
-
Moving ngen-cal image entrypoint script.
Moving and renaming entrypoint script for ngen_cal image to be within context directory for ngen image build, in preparation for utilizing a stage within the main Dockerfile for the calibration image.
-
-
Fix ngen entrypoint handling for serial run.
Fixing logic in script for how things were handled when running a serial execution.
-
Fix func to init and sanity ngen exec.
Fixing issue in funcs.sh related to when to check for a partition config dataset directory, which previously would only happen if the MPI node count was greater than 1 (i.e., while an MPI job running all on a single node would still expect the partition config).
-
-
Update ngen Dockerfile for ngen_cal image.
Updating main ngen Dockerfile with new build stage for use as target for ngen_cal worker image.
-
