Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[gitlab] release OTel to public facing repos on relevant tags + protections #30641

Merged
merged 2 commits into from
Nov 4, 2024
Merged

[gitlab] release OTel to public facing repos on relevant tags + protections #30641

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7f27c3d
[gitlab] release OTel to public facing repos on relevant tags + prote…
truthbk Oct 31, 2024
c28417f
[gitlab] contaiener_deploy: address OTel feedback
truthbk Nov 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 85 additions & 1 deletion .gitlab/deploy_containers/conditions.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,30 @@
DSD_REPOSITORY: dogstatsd
IMG_REGISTRIES: public

.manual_on_deploy_auto_on_rc-ot:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🥜 the logic about _ and - is not really strict and sometimes confusing on job names. I would recommend to use _ everywhere if possible (and it allows to select the full name with a double click)

Suggested change
.manual_on_deploy_auto_on_rc-ot:
.manual_on_deploy_auto_on_rc_ot:

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually tried to use the following convention:

job_name-variant

So that anything with _ underscores can be regarded as the actual job name, of which there may be other variants. And the - hyphen is used to separate the job name from the particular variant. If you feel strongly about it, I can change it, but I still find it visually useful.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do understand your point. On my side, I often never really know when to put underscore or hyphen, because:

  • the convention is not used everywhere (See for instance convention here, here and here(
  • it's not super easy to know what is a variant.

However, this is probably not related to this PR: I was trying to limit some naming conventions, and as there is no strict rule now, I can let you do as you prefer 😄

- if: $BUCKET_BRANCH != "beta" && $BUCKET_BRANCH != "stable"
when: manual
allow_failure: true
variables:
AGENT_REPOSITORY: agent-dev
IMG_REGISTRIES: dev
- if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/ && $FORCE_MANUAL != "true"
when: on_success
variables:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🥜 If these are the default values we can maybe set them in the main variables section of the .gitlab-ci.yml (can maybe be a separate PR on our side as it would change a lot of existing conditions)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are all temporary jobs that will be removed once GA is reached for the OTel offering, so not sure it's worth it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes like the other comment, this is maybe something we could change to simplify the gitlab configuration and it's not directly related to your change so you can ignore this

AGENT_REPOSITORY: agent
IMG_REGISTRIES: public
- if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/ && $FORCE_MANUAL == "true"
when: manual
allow_failure: true
variables:
AGENT_REPOSITORY: agent
IMG_REGISTRIES: public
- when: manual
allow_failure: true
variables:
AGENT_REPOSITORY: agent
IMG_REGISTRIES: public

# Rule for job that are triggered on_success on RC pipelines
.on_rc:
- if: $FORCE_MANUAL == "true" && $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
Expand All @@ -44,6 +68,20 @@
DSD_REPOSITORY: dogstatsd
IMG_REGISTRIES: public

# Rule for job that are triggered on_success on RC pipelines for OTel Beta
.on_rc-ot:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as previously

Suggested change
.on_rc-ot:
.on_rc_ot:

- if: $FORCE_MANUAL == "true" && $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
when: manual
allow_failure: true
variables:
AGENT_REPOSITORY: agent
IMG_REGISTRIES: public
- if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
when: on_success
variables:
AGENT_REPOSITORY: agent
IMG_REGISTRIES: public

# Rule for job that can be triggered manually on final build, deploy to prod repository on stable branch deploy, else to dev repository
.on_final:
- if: $BUCKET_BRANCH == "beta"
Expand All @@ -62,13 +100,32 @@
DSD_REPOSITORY: dogstatsd
IMG_REGISTRIES: public

# Rule for job that can be triggered manually on final build, deploy to prod repository on stable branch deploy, else to dev repository
# For OTel Beta builds
.on_final-ot:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
.on_final-ot:
.on_final_ot:

- if: $BUCKET_BRANCH == "beta"
when: never
- if: $BUCKET_BRANCH != "beta" && $BUCKET_BRANCH != "stable" && $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+$/
when: manual
allow_failure: true
variables:
AGENT_REPOSITORY: agent-dev
IMG_REGISTRIES: dev
- if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+$/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could use a comment to clarify the difference in the pattern/what's expected to match

when: manual
allow_failure: true
variables:
AGENT_REPOSITORY: agent
IMG_REGISTRIES: public

# Rule to deploy to our internal repository, on stable branch deploy
.on_internal_final:
- if: $BUCKET_BRANCH == "beta"
when: never
- if: $BUCKET_BRANCH != "beta" && $BUCKET_BRANCH != "stable"
when: never
- when: manual
- if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+$/
when: manual
allow_failure: true
variables:
AGENT_REPOSITORY: ci/datadog-agent/agent-release
Expand All @@ -77,6 +134,19 @@
CWS_INSTRUMENTATION_REPOSITORY: ci/datadog-agent/cws-instrumentation-release
IMG_REGISTRIES: internal-aws-ddbuild

# Rule to deploy to our internal repository, on stable branch deploy
.on_internal_final-ot:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
.on_internal_final-ot:
.on_internal_final_ot:

- if: $BUCKET_BRANCH == "beta"
when: never
- if: $BUCKET_BRANCH != "beta" && $BUCKET_BRANCH != "stable"
when: never
- if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+$/
when: manual
allow_failure: true
variables:
AGENT_REPOSITORY: ci/datadog-agent/agent-release
IMG_REGISTRIES: internal-aws-ddbuild

# Rule to deploy to our internal repository on RC
.on_internal_rc:
- if: $FORCE_MANUAL == "true" && $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
Expand All @@ -96,3 +166,17 @@
DSD_REPOSITORY: ci/datadog-agent/dogstatsd-release
CWS_INSTRUMENTATION_REPOSITORY: ci/datadog-agent/cws-instrumentation-release
IMG_REGISTRIES: internal-aws-ddbuild

# Rule to deploy to our internal repository on RC for OTel Agent Beta
.on_internal_rc-ot:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
.on_internal_rc-ot:
.on_internal_rc_ot:

- if: $FORCE_MANUAL == "true" && $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
when: manual
allow_failure: true
variables:
AGENT_REPOSITORY: ci/datadog-agent/agent-release
IMG_REGISTRIES: internal-aws-ddbuild
- if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
when: on_success
variables:
AGENT_REPOSITORY: ci/datadog-agent/agent-release
IMG_REGISTRIES: internal-aws-ddbuild
40 changes: 32 additions & 8 deletions .gitlab/deploy_containers/deploy_containers_a7.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,26 +48,32 @@ include:
.deploy_containers-a7-base-ot:
extends: .docker_publish_job_definition
stage: deploy_containers
rules:
- when: manual
allow_failure: true
variables:
AGENT_REPOSITORY: agent
IMG_REGISTRIES: public
dependencies: []

deploy_containers-a7:
extends: .deploy_containers-a7_external
rules:
!reference [.manual_on_deploy_auto_on_rc]

deploy_containers-a7-ot:
extends: .deploy_containers-a7-base-ot
rules:
!reference [.manual_on_deploy_auto_on_rc-ot]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
!reference [.manual_on_deploy_auto_on_rc-ot]
!reference [.manual_on_deploy_auto_on_rc_ot]


deploy_containers-a7-rc:
extends: .deploy_containers-a7_external
rules:
!reference [.on_rc]
variables:
VERSION: 7-rc

deploy_containers-a7-ot-rc:
extends: .deploy_containers-a7-base-ot
rules:
!reference [.on_rc-ot]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
!reference [.on_rc-ot]
!reference [.on_rc_ot]

variables:
VERSION: 7-ot-beta-rc

deploy_containers-dogstatsd:
extends: .docker_publish_job_definition
stage: deploy_containers
Expand Down Expand Up @@ -95,8 +101,24 @@ deploy_containers-a7_internal-rc:
VERSION: 7-rc


deploy_containers-ot:
deploy_containers-a7-ot_internal:
extends: .deploy_containers-a7-base-ot
rules:
!reference [.on_internal_final-ot]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
!reference [.on_internal_final-ot]
!reference [.on_internal_final_ot]

before_script:
- if [[ "$VERSION" == "" ]]; then VERSION="$(inv agent.version --major-version 7 --url-safe --pipeline-id $PARENT_PIPELINE_ID)" || exit $?; fi
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpicky, but could you drop the --agent-version 7? They already default to 7 and we plan on removing those parameters soon

- export IMG_SOURCES="${SRC_AGENT}:v${PARENT_PIPELINE_ID}-${CI_COMMIT_SHORT_SHA}-7-ot-beta${JMX}-amd64,${SRC_AGENT}:v${PARENT_PIPELINE_ID}-${CI_COMMIT_SHORT_SHA}-7-ot-beta${JMX}-arm64"
- export IMG_DESTINATIONS="${AGENT_REPOSITORY}:${VERSION}-ot-beta${JMX}"
parallel:
matrix:
- JMX:
- ""
- "-jmx"

deploy_containers-a7-ot_internal-rc:
extends: .deploy_containers-a7-base-ot
rules:
!reference [.on_internal_rc-ot]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
!reference [.on_internal_rc-ot]
!reference [.on_internal_rc_ot]

before_script:
- if [[ "$VERSION" == "" ]]; then VERSION="$(inv agent.version --major-version 7 --url-safe --pipeline-id $PARENT_PIPELINE_ID)" || exit $?; fi
- export IMG_SOURCES="${SRC_AGENT}:v${PARENT_PIPELINE_ID}-${CI_COMMIT_SHORT_SHA}-7-ot-beta${JMX}-amd64,${SRC_AGENT}:v${PARENT_PIPELINE_ID}-${CI_COMMIT_SHORT_SHA}-7-ot-beta${JMX}-arm64"
Expand Down Expand Up @@ -155,8 +177,10 @@ deploy_containers_latest-dogstatsd:
IMG_SOURCES: ${SRC_DSD}:v${PARENT_PIPELINE_ID}-${CI_COMMIT_SHORT_SHA}-amd64,${SRC_DSD}:v${PARENT_PIPELINE_ID}-${CI_COMMIT_SHORT_SHA}-arm64
IMG_DESTINATIONS: ${DSD_REPOSITORY}:7,${DSD_REPOSITORY}:latest

deploy_containers_latest-ot:
deploy_containers_latest-a7-ot:
extends: .deploy_containers-a7-base-ot
rules:
!reference [.on_final-ot]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
!reference [.on_final-ot]
!reference [.on_final_ot]

variables:
VERSION: 7
parallel:
Expand Down
7 changes: 6 additions & 1 deletion .gitlab/trigger_release/trigger_release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@ trigger_auto_staging_release:
rules:
- if: $DDR == "true"
when: never
- if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+(-rc\.[0-9]+){0,1}$/
when: never
- !reference [.on_deploy]

trigger_manual_prod_release:
Expand All @@ -49,4 +51,7 @@ trigger_manual_prod_release:
# The jobs in the downstream pipeline will all be manual, so following
# the created pipeline would likely cause this job to timeout
NO_FOLLOW: "--no-follow"
rules: !reference [.on_deploy_stable_or_beta_manual_auto_on_stable]
rules:
- if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-v[0-9]+\.[0-9]+\.[0-9]+(-rc\.[0-9]+){0,1}$/
when: never
- !reference [.on_deploy_stable_or_beta_manual_auto_on_stable]
Loading