-
Notifications
You must be signed in to change notification settings - Fork 122
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add logic to carma script to account for env var in carma-config #2409
Add logic to carma script to account for env var in carma-config #2409
Conversation
@MishkaMN https://github.com/usdot-fhwa-stol/carma-platform/blob/623f636dd6b2ccbacc6b9cd97546de9f1f92d7e9/engineering_tools/carma |
Discussed offline with @paulbourelly999 and @adev4a to include an option for the user to specify the env file directory for the docker-compose |
|
<!-- Thanks for the contribution, this is awesome. --> # PR Details ## Description Supports usdot-fhwa-stol/carma-platform#2409 This change copies the hidden `.env` file specifically to the carma-config image. It also adds `cdasim_config` folder to necessary `.gitignore` files such that the respective images can be built without any permission issue common with cdasim_config folders. ## Related GitHub Issue NA <!--- This project only accepts pull requests related to open GitHub issues or Jira Keys --> <!--- If suggesting a new feature or change, please discuss it in an issue first --> <!--- If fixing a bug, there should be an issue describing it with steps to reproduce --> <!--- Please DO NOT name partially fixed issues, instead open an issue specific to this fix --> <!--- Please link to the issue here: --> ## Related Jira Key https://usdot-carma.atlassian.net/browse/ARC-82 <!-- e.g. CAR-123 --> ## Motivation and Context During 4.5.0 release, we modifed the way DOCKER_ORG and DOCKER_TAG are specified by centralizing them into a single .env file in each carma-config folder for docker-compose.yml to load. However, currnently, when a carma-config image is built by our GitHub Actions CI processes, the resulting image still contains the {DOCKER_ORG} and {DOCKER_TAG} environment variable placeholders in the docker-compose.yml and docker-compose-background.yml files. This breaks the carma config install <carma-config image> functionality of the carma shell script. The carma start all functionality breaks with this as well. Workaround: If .env file exists in the same directory from where `carma` is called, then everything works fine. However, this is not a desired functionality as we want to call carma from anywhere <!--- Why is this change required? What problem does it solve? --> ## How Has This Been Tested? locally in VM. Results of each commands working on the main PR and its comments <!--- Please describe in detail how you tested your changes. --> <!--- Include details of your testing environment, and the tests you ran to --> <!--- see how your change affects other areas of the code, etc. --> ## Types of changes <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: --> - [X] Defect fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that cause existing functionality to change) ## Checklist: <!--- Go over all the following points, and put an `x` in all the boxes that apply. --> <!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> - [X] I have added any new packages to the sonar-scanner.properties file - [ ] My change requires a change to the documentation. - [X] I have updated the documentation accordingly. - [X] I have read the [**CONTRIBUTING**](https://github.com/usdot-fhwa-stol/carma-platform/blob/develop/Contributing.md) document. - [X] I have added tests to cover my changes. - [X] All new and existing tests passed.
PR Details
Description
This PR fixes the carma script to be able to run from any directory without relying on .env file to exist in the same directory.
Depends on this PR: usdot-fhwa-stol/carma-config#352
The proposed change on a high level:
.env
file into thecarma-config
container. Since.env
file specifies carma version now, it makes sense to include it in the config, which is intended to track carma version.carma
script now modifies that variable as below (highest to lowest precedence):carma
is calledcarma config
currently setThis way,
carma
script always has default values from carma-config wherever it's called. Additionally, user can still use the variable as docker intended.NOTE: docker actually uses local
.env
file only as the last resort such as if no.env
file is specified, so it doesn't always use. Currently I implemented such that it uses .env as 2nd to last precedence as long as it exists, as it makes sense in my head. For anyone who is interested, here is the docker env file precedence.extra_args
incarma start all extra_args
as input todocker-compose up
command. Now it works as intended where the user can input extra flags--env-file
or--progress
etc.Resulting user experience:
Most user experience stays the same, but here are specific breakdowns of the usage and logics under the hood. "New Behaviors" describe different actions required by the user, whereas "Old Behaviors" describe similar user experience as old.
NEW Behavior: docker-compose needed to be updated to latest (or more than 2.23.0) which is now 2.27.1
NEW Behavior: If multiple
.env
files are specified, such as local.env
,.env
files in COMPOSE_ENV_FILES etc, all of them are loaded including.env
fromcarma-config
. If overlapping values exist, above precedence is used to overwrite.NOTE: if
--env-file
argument is used incarma start all --env-file /path/to/env
, docker ignores COMPOSE_ENV_FILES. It is still same behavior as that.OLD Behavior:
carma config edit
usescarma-base
that tries to use same version as specified in its.env
inside. If fails, just uses most recentcarma-base
. In the end, it shouldn't matter which version ofcarma-base
is used as it is only for editingvolume
Old behavior:
carma config install <image>
tries to pull down images specified in.env
of<config-image>
. It doesn't make sense to use any other.env
here as the user is trying to download specific version in<config-image>
OLD Behavior + fixes:
carma exec
carma-platform
orcarma-messenger
using version specified inside current.env
incarma-config
.carma-platform
orcarma-messenger
images.NOTE: Here, previously the script tried to use
usdotfhwastol/carma-base:latest
instead which often doesn't exist and script failed. This was undesirable such as when custom image likelocal/carma-platform:latest
is used, then the script used to fail.carma-base
image.NOTE: notice here that
carma exec
is uses only.env
file from carma-config to open the container, which how it worked before. Using any other.env
file is overengineering and not needed.carma exec
has extra argument of-i
to specify specific image anyways.OLD Behavior: In most cases, if the user wants to change the image name/tags (or any other variable), they can simply modify the .env file and rebuild
<config-image>
orcarma config edit
and modify the/opt/carma/vehicle/config/.env
file and restart. Or even hardcoding as before in the docker-compose.yml work as well without worrying about any.env
files. However, they can now use different.env
files now as mentioned before, which has higher precedence than the one in<carma-config>
OLD Behavior:
carma
script is still backwards compatible to<config-image>
before (and including) carma-system-4.5.0, which didn't have.env
files. I accomplished it by keeping old relevant logics in__get_image_from_config
(most new ones are backwards compatible anyways).NOTE: I suppressed some potential error messages that would have occured due to older versions as they are not errors (such as not finding
.env
file in<config-image>
and there will be other actual error messages if there was an actual issue).Related GitHub Issue
NA
Related Jira Key
https://usdot-carma.atlassian.net/browse/ARC-82
Motivation and Context
During 4.5.0 release, we modifed the way DOCKER_ORG and DOCKER_TAG are specified by centralizing them into a single .env file in each carma-config folder for docker-compose.yml to load. However, currently, when a carma-config image is built by our GitHub Actions CI processes, the resulting image still contains the {DOCKER_ORG} and {DOCKER_TAG} environment variable placeholders in the docker-compose.yml and docker-compose-background.yml files. This breaks all of the
carma *
functionality of the carma shell script.Workaround: If .env file exists in the same directory from where
carma
is called, then everything works fine. However, this is not a desired functionality as we want to call carma from anywhere.How Has This Been Tested?
locally in VM.
Types of changes
Checklist:
I also updated carma installation guide here to use the most recent version of docker-compose instead of hardcoding it