[gitlab] release OTel to public facing repos on relevant tags + protections #30641
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
short review
|
[Fast Unit Tests Report] On pipeline 48164298 (CI Visibility). The following jobs did not run any unit tests: Jobs:
If you modified Go files and expected unit tests to run in these jobs, please double check the job logs. If you think tests should have been executed reach out to #agent-devx-help |
Regression Detector |
chouquette
approved these changes
Nov 4, 2024
| variables: | ||
| AGENT_REPOSITORY: agent-dev | ||
| IMG_REGISTRIES: dev | ||
| - if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+$/ |
There was a problem hiding this comment.
This could use a comment to clarify the difference in the pattern/what's expected to match
| rules: | ||
| !reference [.on_internal_final-ot] | ||
| before_script: | ||
| - if [[ "$VERSION" == "" ]]; then VERSION="$(inv agent.version --major-version 7 --url-safe --pipeline-id $PARENT_PIPELINE_ID)" || exit $?; fi |
There was a problem hiding this comment.
nitpicky, but could you drop the --agent-version 7? They already default to 7 and we plan on removing those parameters soon
github-actions
bot
added
medium review
chouetz
approved these changes
Nov 4, 2024
| @@ -28,6 +28,30 @@ | |||
| DSD_REPOSITORY: dogstatsd | |||
| IMG_REGISTRIES: public | |||
|
|
|||
| .manual_on_deploy_auto_on_rc-ot: | |||
There was a problem hiding this comment.
🥜 the logic about _ and - is not really strict and sometimes confusing on job names. I would recommend to use _ everywhere if possible (and it allows to select the full name with a double click)
Suggested change
| .manual_on_deploy_auto_on_rc-ot: | |
| .manual_on_deploy_auto_on_rc_ot: |
There was a problem hiding this comment.
I actually tried to use the following convention:
job_name-variant
So that anything with _ underscores can be regarded as the actual job name, of which there may be other variants. And the - hyphen is used to separate the job name from the particular variant. If you feel strongly about it, I can change it, but I still find it visually useful.
There was a problem hiding this comment.
I do understand your point. On my side, I often never really know when to put underscore or hyphen, because:
- the convention is not used everywhere (See for instance convention here, here and here(
- it's not super easy to know what is a variant.
However, this is probably not related to this PR: I was trying to limit some naming conventions, and as there is no strict rule now, I can let you do as you prefer 😄
| @@ -44,6 +68,20 @@ | |||
| DSD_REPOSITORY: dogstatsd | |||
| IMG_REGISTRIES: public | |||
|
|
|||
| # Rule for job that are triggered on_success on RC pipelines for OTel Beta | |||
| .on_rc-ot: | |||
There was a problem hiding this comment.
same as previously
Suggested change
| .on_rc-ot: | |
| .on_rc_ot: |
| IMG_REGISTRIES: dev | ||
| - if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/ && $FORCE_MANUAL != "true" | ||
| when: on_success | ||
| variables: |
There was a problem hiding this comment.
🥜 If these are the default values we can maybe set them in the main variables section of the .gitlab-ci.yml (can maybe be a separate PR on our side as it would change a lot of existing conditions)
There was a problem hiding this comment.
These are all temporary jobs that will be removed once GA is reached for the OTel offering, so not sure it's worth it.
There was a problem hiding this comment.
Yes like the other comment, this is maybe something we could change to simplify the gitlab configuration and it's not directly related to your change so you can ignore this
| @@ -62,13 +100,32 @@ | |||
| DSD_REPOSITORY: dogstatsd | |||
| IMG_REGISTRIES: public | |||
|
|
|||
| # Rule for job that can be triggered manually on final build, deploy to prod repository on stable branch deploy, else to dev repository | |||
| # For OTel Beta builds | |||
| .on_final-ot: | |||
There was a problem hiding this comment.
Suggested change
| .on_final-ot: | |
| .on_final_ot: |
| @@ -77,6 +134,19 @@ | |||
| CWS_INSTRUMENTATION_REPOSITORY: ci/datadog-agent/cws-instrumentation-release | |||
| IMG_REGISTRIES: internal-aws-ddbuild | |||
|
|
|||
| # Rule to deploy to our internal repository, on stable branch deploy | |||
| .on_internal_final-ot: | |||
There was a problem hiding this comment.
Suggested change
| .on_internal_final-ot: | |
| .on_internal_final_ot: |
| deploy_containers-a7-ot: | ||
| extends: .deploy_containers-a7-base-ot | ||
| rules: | ||
| !reference [.manual_on_deploy_auto_on_rc-ot] |
There was a problem hiding this comment.
Suggested change
| !reference [.manual_on_deploy_auto_on_rc-ot] | |
| !reference [.manual_on_deploy_auto_on_rc_ot] |
| deploy_containers-a7-ot-rc: | ||
| extends: .deploy_containers-a7-base-ot | ||
| rules: | ||
| !reference [.on_rc-ot] |
There was a problem hiding this comment.
Suggested change
| !reference [.on_rc-ot] | |
| !reference [.on_rc_ot] |
| extends: .deploy_containers-a7-base-ot | ||
| rules: | ||
| !reference [.on_internal_final-ot] |
There was a problem hiding this comment.
Suggested change
| !reference [.on_internal_final-ot] | |
| !reference [.on_internal_final_ot] |
| deploy_containers-a7-ot_internal-rc: | ||
| extends: .deploy_containers-a7-base-ot | ||
| rules: | ||
| !reference [.on_internal_rc-ot] |
There was a problem hiding this comment.
Suggested change
| !reference [.on_internal_rc-ot] | |
| !reference [.on_internal_rc_ot] |
| extends: .deploy_containers-a7-base-ot | ||
| rules: | ||
| !reference [.on_final-ot] |
There was a problem hiding this comment.
Suggested change
| !reference [.on_final-ot] | |
| !reference [.on_final_ot] |
github-actions
bot
added
long review
Gitlab CI Configuration Changes
|
| Removed | Modified | Added | Renamed |
|---|---|---|---|
| 0 | 2 | 0 | 0 |
Updated: .gitlab/deploy_containers/deploy_containers_a7.yml
Modified Jobs
.deploy_containers-a7-base-ot
.deploy_containers-a7-base-ot:
dependencies: []
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
- rules:
- - allow_failure: true
- when: manual
script:
- GITLAB_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $GITLAB_TOKEN write_api)
|| exit $?; export GITLAB_TOKEN
- "if [[ \"$BUCKET_BRANCH\" == \"nightly\" && ( \"$IMG_SOURCES\" =~ \"$SRC_AGENT\"\
\ || \"$IMG_SOURCES\" =~ \"$SRC_DCA\" || \"$IMG_SOURCES\" =~ \"$SRC_CWS_INSTRUMENTATION\"\
\ || \"$IMG_VARIABLES\" =~ \"$SRC_AGENT\" || \"$IMG_VARIABLES\" =~ \"$SRC_DCA\"\
\ || \"$IMG_VARIABLES\" =~ \"$SRC_CWS_INSTRUMENTATION\" ) ]]; then\n export ECR_RELEASE_SUFFIX=\"\
-nightly\"\nelse\n export ECR_RELEASE_SUFFIX=\"${CI_COMMIT_TAG+-release}\"\n\
fi\n"
- IMG_VARIABLES="$(sed -E "s#(${SRC_AGENT}|${SRC_DSD}|${SRC_DCA}|${SRC_CWS_INSTRUMENTATION})#\1${ECR_RELEASE_SUFFIX}#g"
<<<"$IMG_VARIABLES")"
- IMG_SOURCES="$(sed -E "s#(${SRC_AGENT}|${SRC_DSD}|${SRC_DCA}|${SRC_CWS_INSTRUMENTATION})#\1${ECR_RELEASE_SUFFIX}#g"
<<<"$IMG_SOURCES")"
- inv pipeline.trigger-child-pipeline --project-name DataDog/public-images --git-ref
main --timeout 1800 --variable IMG_VARIABLES --variable IMG_REGISTRIES --variable
IMG_SOURCES --variable IMG_DESTINATIONS --variable IMG_SIGNING --variable APPS
--variable BAZEL_TARGET --variable DDR --variable DDR_WORKFLOW_ID --variable TARGET_ENV
--variable DYNAMIC_BUILD_RENDER_TARGET_FORWARD_PARAMETERS
stage: deploy_containers
tags:
- arch:amd64
variables:
- AGENT_REPOSITORY: agent
- IMG_REGISTRIES: public
IMG_SIGNING: ''
IMG_VARIABLES: ''
SRC_AGENT: registry.ddbuild.io/ci/datadog-agent/agent
SRC_CWS_INSTRUMENTATION: registry.ddbuild.io/ci/datadog-agent/cws-instrumentation
SRC_DCA: registry.ddbuild.io/ci/datadog-agent/cluster-agent
SRC_DSD: registry.ddbuild.io/ci/datadog-agent/dogstatsd.on_internal_final
.on_internal_final:
- if: $BUCKET_BRANCH == "beta"
when: never
- if: $BUCKET_BRANCH != "beta" && $BUCKET_BRANCH != "stable"
when: never
- allow_failure: true
+ if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+$/
variables:
AGENT_REPOSITORY: ci/datadog-agent/agent-release
CLUSTER_AGENT_REPOSITORY: ci/datadog-agent/cluster-agent-release
CWS_INSTRUMENTATION_REPOSITORY: ci/datadog-agent/cws-instrumentation-release
DSD_REPOSITORY: ci/datadog-agent/dogstatsd-release
IMG_REGISTRIES: internal-aws-ddbuild
when: manualdeploy_containers-a7_internal
deploy_containers-a7_internal:
before_script:
- if [[ "$VERSION" == "" ]]; then VERSION="$(inv agent.version --major-version 7
--url-safe --pipeline-id $PARENT_PIPELINE_ID)" || exit $?; fi
- export IMG_BASE_SRC="${SRC_AGENT}:v${PARENT_PIPELINE_ID}-${CI_COMMIT_SHORT_SHA}"
- export IMG_LINUX_SOURCES="${IMG_BASE_SRC}-7${JMX}-amd64,${IMG_BASE_SRC}-7${JMX}-arm64"
- export IMG_WINDOWS_SOURCES="${IMG_BASE_SRC}-7${JMX}-win1809${FLAVOR}-amd64,${IMG_BASE_SRC}-7${JMX}-winltsc2022${FLAVOR}-amd64"
- if [[ "$FLAVOR" == "-linux" ]]; then export IMG_SOURCES="${IMG_LINUX_SOURCES}";
elif [[ "$FLAVOR" == "-servercore" ]]; then export IMG_SOURCES="${IMG_WINDOWS_SOURCES}";
else export IMG_SOURCES="${IMG_LINUX_SOURCES},${IMG_WINDOWS_SOURCES}"; fi
- export IMG_DESTINATIONS="${AGENT_REPOSITORY}:${VERSION}${FLAVOR}${JMX}"
dependencies: []
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
rules:
- if: $BUCKET_BRANCH == "beta"
when: never
- if: $BUCKET_BRANCH != "beta" && $BUCKET_BRANCH != "stable"
when: never
- allow_failure: true
+ if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+$/
variables:
AGENT_REPOSITORY: ci/datadog-agent/agent-release
CLUSTER_AGENT_REPOSITORY: ci/datadog-agent/cluster-agent-release
CWS_INSTRUMENTATION_REPOSITORY: ci/datadog-agent/cws-instrumentation-release
DSD_REPOSITORY: ci/datadog-agent/dogstatsd-release
IMG_REGISTRIES: internal-aws-ddbuild
when: manual
script:
- GITLAB_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $GITLAB_TOKEN write_api)
|| exit $?; export GITLAB_TOKEN
- "if [[ \"$BUCKET_BRANCH\" == \"nightly\" && ( \"$IMG_SOURCES\" =~ \"$SRC_AGENT\"\
\ || \"$IMG_SOURCES\" =~ \"$SRC_DCA\" || \"$IMG_SOURCES\" =~ \"$SRC_CWS_INSTRUMENTATION\"\
\ || \"$IMG_VARIABLES\" =~ \"$SRC_AGENT\" || \"$IMG_VARIABLES\" =~ \"$SRC_DCA\"\
\ || \"$IMG_VARIABLES\" =~ \"$SRC_CWS_INSTRUMENTATION\" ) ]]; then\n export ECR_RELEASE_SUFFIX=\"\
-nightly\"\nelse\n export ECR_RELEASE_SUFFIX=\"${CI_COMMIT_TAG+-release}\"\n\
fi\n"
- IMG_VARIABLES="$(sed -E "s#(${SRC_AGENT}|${SRC_DSD}|${SRC_DCA}|${SRC_CWS_INSTRUMENTATION})#\1${ECR_RELEASE_SUFFIX}#g"
<<<"$IMG_VARIABLES")"
- IMG_SOURCES="$(sed -E "s#(${SRC_AGENT}|${SRC_DSD}|${SRC_DCA}|${SRC_CWS_INSTRUMENTATION})#\1${ECR_RELEASE_SUFFIX}#g"
<<<"$IMG_SOURCES")"
- inv pipeline.trigger-child-pipeline --project-name DataDog/public-images --git-ref
main --timeout 1800 --variable IMG_VARIABLES --variable IMG_REGISTRIES --variable
IMG_SOURCES --variable IMG_DESTINATIONS --variable IMG_SIGNING --variable APPS
--variable BAZEL_TARGET --variable DDR --variable DDR_WORKFLOW_ID --variable TARGET_ENV
--variable DYNAMIC_BUILD_RENDER_TARGET_FORWARD_PARAMETERS
stage: deploy_containers
tags:
- arch:amd64
variables:
IMG_SIGNING: ''
IMG_VARIABLES: ''
JMX: -jmx
SRC_AGENT: registry.ddbuild.io/ci/datadog-agent/agent
SRC_CWS_INSTRUMENTATION: registry.ddbuild.io/ci/datadog-agent/cws-instrumentation
SRC_DCA: registry.ddbuild.io/ci/datadog-agent/cluster-agent
SRC_DSD: registry.ddbuild.io/ci/datadog-agent/dogstatsddeploy_containers_latest-a7_internal
deploy_containers_latest-a7_internal:
dependencies: []
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
parallel:
matrix:
- IMG_DESTINATIONS: ${AGENT_REPOSITORY}:7-jmx
IMG_SOURCES: '%BASE%-amd64,%BASE%-arm64,%BASE%-win1809-amd64,%BASE%-winltsc2022-amd64'
IMG_VARIABLES: BASE=${SRC_AGENT}:v${PARENT_PIPELINE_ID}-${CI_COMMIT_SHORT_SHA}-7-jmx
rules:
- if: $BUCKET_BRANCH == "beta"
when: never
- if: $BUCKET_BRANCH != "beta" && $BUCKET_BRANCH != "stable"
when: never
- allow_failure: true
+ if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+$/
variables:
AGENT_REPOSITORY: ci/datadog-agent/agent-release
CLUSTER_AGENT_REPOSITORY: ci/datadog-agent/cluster-agent-release
CWS_INSTRUMENTATION_REPOSITORY: ci/datadog-agent/cws-instrumentation-release
DSD_REPOSITORY: ci/datadog-agent/dogstatsd-release
IMG_REGISTRIES: internal-aws-ddbuild
when: manual
script:
- GITLAB_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $GITLAB_TOKEN write_api)
|| exit $?; export GITLAB_TOKEN
- "if [[ \"$BUCKET_BRANCH\" == \"nightly\" && ( \"$IMG_SOURCES\" =~ \"$SRC_AGENT\"\
\ || \"$IMG_SOURCES\" =~ \"$SRC_DCA\" || \"$IMG_SOURCES\" =~ \"$SRC_CWS_INSTRUMENTATION\"\
\ || \"$IMG_VARIABLES\" =~ \"$SRC_AGENT\" || \"$IMG_VARIABLES\" =~ \"$SRC_DCA\"\
\ || \"$IMG_VARIABLES\" =~ \"$SRC_CWS_INSTRUMENTATION\" ) ]]; then\n export ECR_RELEASE_SUFFIX=\"\
-nightly\"\nelse\n export ECR_RELEASE_SUFFIX=\"${CI_COMMIT_TAG+-release}\"\n\
fi\n"
- IMG_VARIABLES="$(sed -E "s#(${SRC_AGENT}|${SRC_DSD}|${SRC_DCA}|${SRC_CWS_INSTRUMENTATION})#\1${ECR_RELEASE_SUFFIX}#g"
<<<"$IMG_VARIABLES")"
- IMG_SOURCES="$(sed -E "s#(${SRC_AGENT}|${SRC_DSD}|${SRC_DCA}|${SRC_CWS_INSTRUMENTATION})#\1${ECR_RELEASE_SUFFIX}#g"
<<<"$IMG_SOURCES")"
- inv pipeline.trigger-child-pipeline --project-name DataDog/public-images --git-ref
main --timeout 1800 --variable IMG_VARIABLES --variable IMG_REGISTRIES --variable
IMG_SOURCES --variable IMG_DESTINATIONS --variable IMG_SIGNING --variable APPS
--variable BAZEL_TARGET --variable DDR --variable DDR_WORKFLOW_ID --variable TARGET_ENV
--variable DYNAMIC_BUILD_RENDER_TARGET_FORWARD_PARAMETERS
stage: deploy_containers
tags:
- arch:amd64
variables:
IMG_SIGNING: ''
IMG_VARIABLES: ''
SRC_AGENT: registry.ddbuild.io/ci/datadog-agent/agent
SRC_CWS_INSTRUMENTATION: registry.ddbuild.io/ci/datadog-agent/cws-instrumentation
SRC_DCA: registry.ddbuild.io/ci/datadog-agent/cluster-agent
SRC_DSD: registry.ddbuild.io/ci/datadog-agent/dogstatsdAdded Jobs
.manual_on_deploy_auto_on_rc-ot
.manual_on_deploy_auto_on_rc-ot:
- allow_failure: true
if: $BUCKET_BRANCH != "beta" && $BUCKET_BRANCH != "stable"
variables:
AGENT_REPOSITORY: agent-dev
IMG_REGISTRIES: dev
when: manual
- if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
&& $FORCE_MANUAL != "true"
variables:
AGENT_REPOSITORY: agent
IMG_REGISTRIES: public
when: on_success
- allow_failure: true
if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
&& $FORCE_MANUAL == "true"
variables:
AGENT_REPOSITORY: agent
IMG_REGISTRIES: public
when: manual
- allow_failure: true
variables:
AGENT_REPOSITORY: agent
IMG_REGISTRIES: public
when: manual.on_final-ot
.on_final-ot:
- if: $BUCKET_BRANCH == "beta"
when: never
- allow_failure: true
if: $BUCKET_BRANCH != "beta" && $BUCKET_BRANCH != "stable" && $CI_COMMIT_TAG =~
/^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+$/
variables:
AGENT_REPOSITORY: agent-dev
IMG_REGISTRIES: dev
when: manual
- allow_failure: true
if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+$/
variables:
AGENT_REPOSITORY: agent
IMG_REGISTRIES: public
when: manual.on_internal_final-ot
.on_internal_final-ot:
- if: $BUCKET_BRANCH == "beta"
when: never
- if: $BUCKET_BRANCH != "beta" && $BUCKET_BRANCH != "stable"
when: never
- allow_failure: true
if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+$/
variables:
AGENT_REPOSITORY: ci/datadog-agent/agent-release
IMG_REGISTRIES: internal-aws-ddbuild
when: manual.on_internal_rc-ot
.on_internal_rc-ot:
- allow_failure: true
if: $FORCE_MANUAL == "true" && $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
variables:
AGENT_REPOSITORY: ci/datadog-agent/agent-release
IMG_REGISTRIES: internal-aws-ddbuild
when: manual
- if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
variables:
AGENT_REPOSITORY: ci/datadog-agent/agent-release
IMG_REGISTRIES: internal-aws-ddbuild
when: on_success.on_rc-ot
.on_rc-ot:
- allow_failure: true
if: $FORCE_MANUAL == "true" && $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
variables:
AGENT_REPOSITORY: agent
IMG_REGISTRIES: public
when: manual
- if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
variables:
AGENT_REPOSITORY: agent
IMG_REGISTRIES: public
when: on_successdeploy_containers-a7-ot
deploy_containers-a7-ot:
dependencies: []
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
rules:
- allow_failure: true
if: $BUCKET_BRANCH != "beta" && $BUCKET_BRANCH != "stable"
variables:
AGENT_REPOSITORY: agent-dev
IMG_REGISTRIES: dev
when: manual
- if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
&& $FORCE_MANUAL != "true"
variables:
AGENT_REPOSITORY: agent
IMG_REGISTRIES: public
when: on_success
- allow_failure: true
if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
&& $FORCE_MANUAL == "true"
variables:
AGENT_REPOSITORY: agent
IMG_REGISTRIES: public
when: manual
- allow_failure: true
variables:
AGENT_REPOSITORY: agent
IMG_REGISTRIES: public
when: manual
script:
- GITLAB_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $GITLAB_TOKEN write_api)
|| exit $?; export GITLAB_TOKEN
- "if [[ \"$BUCKET_BRANCH\" == \"nightly\" && ( \"$IMG_SOURCES\" =~ \"$SRC_AGENT\"\
\ || \"$IMG_SOURCES\" =~ \"$SRC_DCA\" || \"$IMG_SOURCES\" =~ \"$SRC_CWS_INSTRUMENTATION\"\
\ || \"$IMG_VARIABLES\" =~ \"$SRC_AGENT\" || \"$IMG_VARIABLES\" =~ \"$SRC_DCA\"\
\ || \"$IMG_VARIABLES\" =~ \"$SRC_CWS_INSTRUMENTATION\" ) ]]; then\n export ECR_RELEASE_SUFFIX=\"\
-nightly\"\nelse\n export ECR_RELEASE_SUFFIX=\"${CI_COMMIT_TAG+-release}\"\n\
fi\n"
- IMG_VARIABLES="$(sed -E "s#(${SRC_AGENT}|${SRC_DSD}|${SRC_DCA}|${SRC_CWS_INSTRUMENTATION})#\1${ECR_RELEASE_SUFFIX}#g"
<<<"$IMG_VARIABLES")"
- IMG_SOURCES="$(sed -E "s#(${SRC_AGENT}|${SRC_DSD}|${SRC_DCA}|${SRC_CWS_INSTRUMENTATION})#\1${ECR_RELEASE_SUFFIX}#g"
<<<"$IMG_SOURCES")"
- inv pipeline.trigger-child-pipeline --project-name DataDog/public-images --git-ref
main --timeout 1800 --variable IMG_VARIABLES --variable IMG_REGISTRIES --variable
IMG_SOURCES --variable IMG_DESTINATIONS --variable IMG_SIGNING --variable APPS
--variable BAZEL_TARGET --variable DDR --variable DDR_WORKFLOW_ID --variable TARGET_ENV
--variable DYNAMIC_BUILD_RENDER_TARGET_FORWARD_PARAMETERS
stage: deploy_containers
tags:
- arch:amd64
variables:
IMG_SIGNING: ''
IMG_VARIABLES: ''
SRC_AGENT: registry.ddbuild.io/ci/datadog-agent/agent
SRC_CWS_INSTRUMENTATION: registry.ddbuild.io/ci/datadog-agent/cws-instrumentation
SRC_DCA: registry.ddbuild.io/ci/datadog-agent/cluster-agent
SRC_DSD: registry.ddbuild.io/ci/datadog-agent/dogstatsddeploy_containers-a7-ot-rc
deploy_containers-a7-ot-rc:
dependencies: []
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
rules:
- allow_failure: true
if: $FORCE_MANUAL == "true" && $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
variables:
AGENT_REPOSITORY: agent
IMG_REGISTRIES: public
when: manual
- if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
variables:
AGENT_REPOSITORY: agent
IMG_REGISTRIES: public
when: on_success
script:
- GITLAB_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $GITLAB_TOKEN write_api)
|| exit $?; export GITLAB_TOKEN
- "if [[ \"$BUCKET_BRANCH\" == \"nightly\" && ( \"$IMG_SOURCES\" =~ \"$SRC_AGENT\"\
\ || \"$IMG_SOURCES\" =~ \"$SRC_DCA\" || \"$IMG_SOURCES\" =~ \"$SRC_CWS_INSTRUMENTATION\"\
\ || \"$IMG_VARIABLES\" =~ \"$SRC_AGENT\" || \"$IMG_VARIABLES\" =~ \"$SRC_DCA\"\
\ || \"$IMG_VARIABLES\" =~ \"$SRC_CWS_INSTRUMENTATION\" ) ]]; then\n export ECR_RELEASE_SUFFIX=\"\
-nightly\"\nelse\n export ECR_RELEASE_SUFFIX=\"${CI_COMMIT_TAG+-release}\"\n\
fi\n"
- IMG_VARIABLES="$(sed -E "s#(${SRC_AGENT}|${SRC_DSD}|${SRC_DCA}|${SRC_CWS_INSTRUMENTATION})#\1${ECR_RELEASE_SUFFIX}#g"
<<<"$IMG_VARIABLES")"
- IMG_SOURCES="$(sed -E "s#(${SRC_AGENT}|${SRC_DSD}|${SRC_DCA}|${SRC_CWS_INSTRUMENTATION})#\1${ECR_RELEASE_SUFFIX}#g"
<<<"$IMG_SOURCES")"
- inv pipeline.trigger-child-pipeline --project-name DataDog/public-images --git-ref
main --timeout 1800 --variable IMG_VARIABLES --variable IMG_REGISTRIES --variable
IMG_SOURCES --variable IMG_DESTINATIONS --variable IMG_SIGNING --variable APPS
--variable BAZEL_TARGET --variable DDR --variable DDR_WORKFLOW_ID --variable TARGET_ENV
--variable DYNAMIC_BUILD_RENDER_TARGET_FORWARD_PARAMETERS
stage: deploy_containers
tags:
- arch:amd64
variables:
IMG_SIGNING: ''
IMG_VARIABLES: ''
SRC_AGENT: registry.ddbuild.io/ci/datadog-agent/agent
SRC_CWS_INSTRUMENTATION: registry.ddbuild.io/ci/datadog-agent/cws-instrumentation
SRC_DCA: registry.ddbuild.io/ci/datadog-agent/cluster-agent
SRC_DSD: registry.ddbuild.io/ci/datadog-agent/dogstatsd
VERSION: 7-ot-beta-rcdeploy_containers-a7-ot_internal
deploy_containers-a7-ot_internal:
before_script:
- if [[ "$VERSION" == "" ]]; then VERSION="$(inv agent.version --url-safe --pipeline-id
$PARENT_PIPELINE_ID)" || exit $?; fi
- export IMG_SOURCES="${SRC_AGENT}:v${PARENT_PIPELINE_ID}-${CI_COMMIT_SHORT_SHA}-7-ot-beta${JMX}-amd64,${SRC_AGENT}:v${PARENT_PIPELINE_ID}-${CI_COMMIT_SHORT_SHA}-7-ot-beta${JMX}-arm64"
- export IMG_DESTINATIONS="${AGENT_REPOSITORY}:${VERSION}-ot-beta${JMX}"
dependencies: []
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
parallel:
matrix:
- JMX:
- ''
- -jmx
rules:
- if: $BUCKET_BRANCH == "beta"
when: never
- if: $BUCKET_BRANCH != "beta" && $BUCKET_BRANCH != "stable"
when: never
- allow_failure: true
if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+$/
variables:
AGENT_REPOSITORY: ci/datadog-agent/agent-release
IMG_REGISTRIES: internal-aws-ddbuild
when: manual
script:
- GITLAB_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $GITLAB_TOKEN write_api)
|| exit $?; export GITLAB_TOKEN
- "if [[ \"$BUCKET_BRANCH\" == \"nightly\" && ( \"$IMG_SOURCES\" =~ \"$SRC_AGENT\"\
\ || \"$IMG_SOURCES\" =~ \"$SRC_DCA\" || \"$IMG_SOURCES\" =~ \"$SRC_CWS_INSTRUMENTATION\"\
\ || \"$IMG_VARIABLES\" =~ \"$SRC_AGENT\" || \"$IMG_VARIABLES\" =~ \"$SRC_DCA\"\
\ || \"$IMG_VARIABLES\" =~ \"$SRC_CWS_INSTRUMENTATION\" ) ]]; then\n export ECR_RELEASE_SUFFIX=\"\
-nightly\"\nelse\n export ECR_RELEASE_SUFFIX=\"${CI_COMMIT_TAG+-release}\"\n\
fi\n"
- IMG_VARIABLES="$(sed -E "s#(${SRC_AGENT}|${SRC_DSD}|${SRC_DCA}|${SRC_CWS_INSTRUMENTATION})#\1${ECR_RELEASE_SUFFIX}#g"
<<<"$IMG_VARIABLES")"
- IMG_SOURCES="$(sed -E "s#(${SRC_AGENT}|${SRC_DSD}|${SRC_DCA}|${SRC_CWS_INSTRUMENTATION})#\1${ECR_RELEASE_SUFFIX}#g"
<<<"$IMG_SOURCES")"
- inv pipeline.trigger-child-pipeline --project-name DataDog/public-images --git-ref
main --timeout 1800 --variable IMG_VARIABLES --variable IMG_REGISTRIES --variable
IMG_SOURCES --variable IMG_DESTINATIONS --variable IMG_SIGNING --variable APPS
--variable BAZEL_TARGET --variable DDR --variable DDR_WORKFLOW_ID --variable TARGET_ENV
--variable DYNAMIC_BUILD_RENDER_TARGET_FORWARD_PARAMETERS
stage: deploy_containers
tags:
- arch:amd64
variables:
IMG_SIGNING: ''
IMG_VARIABLES: ''
SRC_AGENT: registry.ddbuild.io/ci/datadog-agent/agent
SRC_CWS_INSTRUMENTATION: registry.ddbuild.io/ci/datadog-agent/cws-instrumentation
SRC_DCA: registry.ddbuild.io/ci/datadog-agent/cluster-agent
SRC_DSD: registry.ddbuild.io/ci/datadog-agent/dogstatsddeploy_containers-a7-ot_internal-rc
deploy_containers-a7-ot_internal-rc:
before_script:
- if [[ "$VERSION" == "" ]]; then VERSION="$(inv agent.version --url-safe --pipeline-id
$PARENT_PIPELINE_ID)" || exit $?; fi
- export IMG_SOURCES="${SRC_AGENT}:v${PARENT_PIPELINE_ID}-${CI_COMMIT_SHORT_SHA}-7-ot-beta${JMX}-amd64,${SRC_AGENT}:v${PARENT_PIPELINE_ID}-${CI_COMMIT_SHORT_SHA}-7-ot-beta${JMX}-arm64"
- export IMG_DESTINATIONS="${AGENT_REPOSITORY}:${VERSION}-ot-beta${JMX}"
dependencies: []
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
parallel:
matrix:
- JMX:
- ''
- -jmx
rules:
- allow_failure: true
if: $FORCE_MANUAL == "true" && $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
variables:
AGENT_REPOSITORY: ci/datadog-agent/agent-release
IMG_REGISTRIES: internal-aws-ddbuild
when: manual
- if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
variables:
AGENT_REPOSITORY: ci/datadog-agent/agent-release
IMG_REGISTRIES: internal-aws-ddbuild
when: on_success
script:
- GITLAB_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $GITLAB_TOKEN write_api)
|| exit $?; export GITLAB_TOKEN
- "if [[ \"$BUCKET_BRANCH\" == \"nightly\" && ( \"$IMG_SOURCES\" =~ \"$SRC_AGENT\"\
\ || \"$IMG_SOURCES\" =~ \"$SRC_DCA\" || \"$IMG_SOURCES\" =~ \"$SRC_CWS_INSTRUMENTATION\"\
\ || \"$IMG_VARIABLES\" =~ \"$SRC_AGENT\" || \"$IMG_VARIABLES\" =~ \"$SRC_DCA\"\
\ || \"$IMG_VARIABLES\" =~ \"$SRC_CWS_INSTRUMENTATION\" ) ]]; then\n export ECR_RELEASE_SUFFIX=\"\
-nightly\"\nelse\n export ECR_RELEASE_SUFFIX=\"${CI_COMMIT_TAG+-release}\"\n\
fi\n"
- IMG_VARIABLES="$(sed -E "s#(${SRC_AGENT}|${SRC_DSD}|${SRC_DCA}|${SRC_CWS_INSTRUMENTATION})#\1${ECR_RELEASE_SUFFIX}#g"
<<<"$IMG_VARIABLES")"
- IMG_SOURCES="$(sed -E "s#(${SRC_AGENT}|${SRC_DSD}|${SRC_DCA}|${SRC_CWS_INSTRUMENTATION})#\1${ECR_RELEASE_SUFFIX}#g"
<<<"$IMG_SOURCES")"
- inv pipeline.trigger-child-pipeline --project-name DataDog/public-images --git-ref
main --timeout 1800 --variable IMG_VARIABLES --variable IMG_REGISTRIES --variable
IMG_SOURCES --variable IMG_DESTINATIONS --variable IMG_SIGNING --variable APPS
--variable BAZEL_TARGET --variable DDR --variable DDR_WORKFLOW_ID --variable TARGET_ENV
--variable DYNAMIC_BUILD_RENDER_TARGET_FORWARD_PARAMETERS
stage: deploy_containers
tags:
- arch:amd64
variables:
IMG_SIGNING: ''
IMG_VARIABLES: ''
SRC_AGENT: registry.ddbuild.io/ci/datadog-agent/agent
SRC_CWS_INSTRUMENTATION: registry.ddbuild.io/ci/datadog-agent/cws-instrumentation
SRC_DCA: registry.ddbuild.io/ci/datadog-agent/cluster-agent
SRC_DSD: registry.ddbuild.io/ci/datadog-agent/dogstatsddeploy_containers_latest-a7-ot
deploy_containers_latest-a7-ot:
dependencies: []
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
parallel:
matrix:
- IMG_DESTINATIONS: ${AGENT_REPOSITORY}:${VERSION}-ot-beta
IMG_SOURCES: ${SRC_AGENT}:v${PARENT_PIPELINE_ID}-${CI_COMMIT_SHORT_SHA}-7-ot-beta-amd64,${SRC_AGENT}:v${PARENT_PIPELINE_ID}-${CI_COMMIT_SHORT_SHA}-7-ot-beta-arm64
- IMG_DESTINATIONS: ${AGENT_REPOSITORY}:${VERSION}-ot-beta-jmx
IMG_SOURCES: ${SRC_AGENT}:v${PARENT_PIPELINE_ID}-${CI_COMMIT_SHORT_SHA}-7-ot-beta-jmx-amd64,${SRC_AGENT}:v${PARENT_PIPELINE_ID}-${CI_COMMIT_SHORT_SHA}-7-ot-beta-jmx-arm64
rules:
- if: $BUCKET_BRANCH == "beta"
when: never
- allow_failure: true
if: $BUCKET_BRANCH != "beta" && $BUCKET_BRANCH != "stable" && $CI_COMMIT_TAG =~
/^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+$/
variables:
AGENT_REPOSITORY: agent-dev
IMG_REGISTRIES: dev
when: manual
- allow_failure: true
if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+$/
variables:
AGENT_REPOSITORY: agent
IMG_REGISTRIES: public
when: manual
script:
- GITLAB_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $GITLAB_TOKEN write_api)
|| exit $?; export GITLAB_TOKEN
- "if [[ \"$BUCKET_BRANCH\" == \"nightly\" && ( \"$IMG_SOURCES\" =~ \"$SRC_AGENT\"\
\ || \"$IMG_SOURCES\" =~ \"$SRC_DCA\" || \"$IMG_SOURCES\" =~ \"$SRC_CWS_INSTRUMENTATION\"\
\ || \"$IMG_VARIABLES\" =~ \"$SRC_AGENT\" || \"$IMG_VARIABLES\" =~ \"$SRC_DCA\"\
\ || \"$IMG_VARIABLES\" =~ \"$SRC_CWS_INSTRUMENTATION\" ) ]]; then\n export ECR_RELEASE_SUFFIX=\"\
-nightly\"\nelse\n export ECR_RELEASE_SUFFIX=\"${CI_COMMIT_TAG+-release}\"\n\
fi\n"
- IMG_VARIABLES="$(sed -E "s#(${SRC_AGENT}|${SRC_DSD}|${SRC_DCA}|${SRC_CWS_INSTRUMENTATION})#\1${ECR_RELEASE_SUFFIX}#g"
<<<"$IMG_VARIABLES")"
- IMG_SOURCES="$(sed -E "s#(${SRC_AGENT}|${SRC_DSD}|${SRC_DCA}|${SRC_CWS_INSTRUMENTATION})#\1${ECR_RELEASE_SUFFIX}#g"
<<<"$IMG_SOURCES")"
- inv pipeline.trigger-child-pipeline --project-name DataDog/public-images --git-ref
main --timeout 1800 --variable IMG_VARIABLES --variable IMG_REGISTRIES --variable
IMG_SOURCES --variable IMG_DESTINATIONS --variable IMG_SIGNING --variable APPS
--variable BAZEL_TARGET --variable DDR --variable DDR_WORKFLOW_ID --variable TARGET_ENV
--variable DYNAMIC_BUILD_RENDER_TARGET_FORWARD_PARAMETERS
stage: deploy_containers
tags:
- arch:amd64
variables:
IMG_SIGNING: ''
IMG_VARIABLES: ''
SRC_AGENT: registry.ddbuild.io/ci/datadog-agent/agent
SRC_CWS_INSTRUMENTATION: registry.ddbuild.io/ci/datadog-agent/cws-instrumentation
SRC_DCA: registry.ddbuild.io/ci/datadog-agent/cluster-agent
SRC_DSD: registry.ddbuild.io/ci/datadog-agent/dogstatsd
VERSION: 7Removed Jobs
- deploy_containers-ot
- deploy_containers_latest-ot
Changes Summary
| Removed | Modified | Added | Renamed |
|---|---|---|---|
| 2 | 4 | 10 | 0 |
ℹ️ Diff available in the job log.
|
/merge |
|
🚂 MergeQueue: pull request added to the queue The median merge time in Use |
truthbk
added a commit
that referenced
this pull request
Nov 4, 2024
This was referenced Nov 5, 2024
truthbk
added a commit
that referenced
this pull request
Nov 6, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR introduces a tag-based workflow for the OT beta image gitlab builds. It allows us to share the release branches (and workflow for the most part) with the regular release process and trigger our own pipelines on bespoke tags to produce the OT beta images, with relevant tag naming/versioning.
The idea is the new conditions and jobs introduced here can allow the regular build pipelines and workflows to remain unchanged, while enabling an almost identical workflow (albeit on slight differently named tags
7.X.Y-vA.B.C) which will produce and publish the images for OT beta in a similar fashion and following the patterns we are familar with.Motivation
Integrate the OT beta builds into the usual release workflow.
Describe how to test/QA your changes
No code, purely gitlab changes. Validating the images are correctly published and the right conditions respected remain the key points to validate. Proof will, mostly be in the pudding, though this has already been tested with a few RC builds.
Possible Drawbacks / Trade-offs
Should be mostly transparent, small change to one A7 build condition that I believe will be a NOP for most usecases.
Additional Notes