Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add nightly vcpkg publishing #2025

Merged
merged 58 commits into from
Apr 6, 2021
Merged
Show file tree
Hide file tree
Changes from 34 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
6310a64
Tool changes to support nightly test package release
danieljurek Mar 25, 2021
2f6d0b5
Initial wiring up and refactor vcpkg-publish.yml
danieljurek Mar 29, 2021
6e85903
Use vcpkg-clone.yml
danieljurek Mar 31, 2021
6ee7833
Disable publishing for verification of nightly builds
danieljurek Mar 31, 2021
70df80e
Complete the comment
danieljurek Mar 31, 2021
71e5a2f
Dependency is enforced at the stage level, not the deployment/job level
danieljurek Mar 31, 2021
ea249f7
Remove environment, no approval needed
danieljurek Mar 31, 2021
346ebc1
Add ability to enable test release parameters
danieljurek Mar 31, 2021
20ae668
deploy -> job
danieljurek Mar 31, 2021
3cbf955
deployment -> job
danieljurek Mar 31, 2021
f95e5a0
Remove strategy
danieljurek Mar 31, 2021
208c4a1
Download pipeline artifacts
danieljurek Mar 31, 2021
256eaac
task -> download
danieljurek Mar 31, 2021
e474192
Set appropriate working directory
danieljurek Mar 31, 2021
229c6f3
git status
danieljurek Mar 31, 2021
b70c239
Set working directory
danieljurek Mar 31, 2021
e88e5f6
Add pipeline for nightly vcpkg PR update
danieljurek Apr 1, 2021
a16df9f
Correct vcpkg-clone.yml path
danieljurek Apr 1, 2021
78bf4e3
Add identity parameters to git merge commands
danieljurek Apr 1, 2021
a7973e7
Remove GitIdentityParameters, the merge command does not suppor tthem…
danieljurek Apr 1, 2021
dd8a067
Split lines
danieljurek Apr 1, 2021
89951ea
Write-Host
danieljurek Apr 1, 2021
899b9ca
Template for nightly branch name, update comments, Check for scheduli…
danieljurek Apr 1, 2021
b32309c
include archetype-cpp-release.yml changes
danieljurek Apr 1, 2021
70e7de2
Re-enable publishing stage
danieljurek Apr 1, 2021
d715111
Close the and
danieljurek Apr 1, 2021
aea9d4a
Runtime condition for integration stage
danieljurek Apr 1, 2021
e13eff5
Move integration below package publishing stages
danieljurek Apr 1, 2021
bc92280
try -- to disambiguate
danieljurek Apr 1, 2021
20eca31
^ -> ~
danieljurek Apr 1, 2021
b22b38d
Remove --
danieljurek Apr 1, 2021
9ded0b1
Documentation and variable naming
danieljurek Apr 1, 2021
3876e9e
Move documentation comment
danieljurek Apr 1, 2021
e804577
Merge branch 'master' into djurek/vcpkg-nightly
danieljurek Apr 1, 2021
c735b55
Update eng/pipelines/templates/steps/generate-nightly-branch-name.yml
danieljurek Apr 2, 2021
099d4a9
Apply suggestions from code review
danieljurek Apr 2, 2021
6333c71
Review feedback
danieljurek Apr 2, 2021
ea2cbd3
Add ability to skip all release steps
danieljurek Apr 2, 2021
b5554a2
Equal sign
danieljurek Apr 2, 2021
061331b
The right number of dashes
danieljurek Apr 2, 2021
7538b60
Checkout before merge
danieljurek Apr 2, 2021
26402e2
Checkout the default branch
danieljurek Apr 2, 2021
45208ef
--no-ff
danieljurek Apr 2, 2021
b1a117c
git reset
danieljurek Apr 2, 2021
807cd3f
Merge branch 'master' into djurek/vcpkg-nightly
danieljurek Apr 5, 2021
53c74b7
Skip publishing for artifacts which are not publishign to vcpkg
danieljurek Apr 5, 2021
a9fb1f5
Merge branch 'master' into djurek/vcpkg-nightly
danieljurek Apr 5, 2021
5423d0a
Add git config --unset
danieljurek Apr 5, 2021
e5041c0
Merge branch 'master' into djurek/vcpkg-nightly
danieljurek Apr 5, 2021
ef16616
Merge branch 'master' into djurek/vcpkg-nightly
danieljurek Apr 5, 2021
8b35b33
Apply suggestions from code review
danieljurek Apr 6, 2021
4bea21b
Review feedback
danieljurek Apr 6, 2021
4d4befd
Merge branch 'djurek/vcpkg-nightly' of github.com:Azure/azure-sdk-for…
danieljurek Apr 6, 2021
d4e0acf
Review feedback
danieljurek Apr 6, 2021
2f5018f
Merge branch 'master' into djurek/vcpkg-nightly
danieljurek Apr 6, 2021
586b66f
Remove extra reset
danieljurek Apr 6, 2021
9922631
Merge branch 'djurek/vcpkg-nightly' of github.com:Azure/azure-sdk-for…
danieljurek Apr 6, 2021
dd22981
TestRelease -> DailyRelease
danieljurek Apr 6, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 64 additions & 0 deletions eng/pipelines/cpp-nightly.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@

danieljurek marked this conversation as resolved.
Show resolved Hide resolved
jobs:
- job:
displayName: Nightly vcpkg PR
pool:
name: azsdk-pool-mms-win-2019-general
vmImage: MMS2019

variables:
# The branch in the azure-sdk/vcpkg repo that is updated nigthly with the
# latest set of ports
TargetPrBranch: azure-sdk-for-cpp-nightly-verification
danieljurek marked this conversation as resolved.
Show resolved Hide resolved

steps:
- checkout: self

- template: /eng/pipelines/templates/steps/generate-nightly-branch-name.yml

- template: /eng/pipelines/templates/steps/vcpkg-clone.yml
parameters:
Workspace: $(Pipeline.Workspace)
RepoOwner: azure-sdk
PRBranchName: $(TargetPrBranch)

- template: /eng/common/pipelines/templates/steps/set-default-branch.yml
parameters:
WorkingDirectory: $(Pipeline.Workspace)/vcpkg

# Git merge does not support the -c parameter. For the scope of this job
# it's ok to configure the user.name and user.email at a global level
- pwsh: |
danieljurek marked this conversation as resolved.
Show resolved Hide resolved
git config --global user.name "azure-sdk"
git config --global user.email "azuresdk@microsoft.com"
displayName: Configure git identity

- task: Powershell@2
inputs:
filePath: eng/scripts/Update-VcpkgNightlyBranch.ps1
arguments: >-
-SourceBranchName $(NightlyBranchName)
-TargetBranchName $(TargetPrBranch)
-MainBranchName $(DefaultBranch)
pwsh: true
workingDirectory: $(Pipeline.Workspace)/vcpkg
displayName: Update vcpkg Nightly Branch

# Set $(HasChanges) to $true so create-pull-request.yml will push the
# branch.
- pwsh: Write-Host "##vso[task.setvariable variable=HasChanges]$true"
danieljurek marked this conversation as resolved.
Show resolved Hide resolved
displayName: Set $(HasChanges) to $true

# This step will skip creating the PR if the PR already exists
- template: /eng/common/pipelines/templates/steps/create-pull-request.yml
parameters:
BaseBranchName: $(DefaultBranch)
PRBranchName: $(TargetPrBranch)
RepoOwner: Microsoft
RepoName: vcpkg
WorkingDirectory: $(Pipeline.Workspace)/vcpkg
PRTitle: "[DO NOT MERGE] Nightly Azure SDK for C++ CI validation"
PRBody: This is a long-lived draft PR should not be merged. Its branch is be updated periodically to validate the Azure SDK for C++ against the vcpkg CI.
ScriptDirectory: $(System.DefaultWorkingDirectory)/eng/common/scripts
SkipCheckingForChanges: true
OpenAsDraft: true
108 changes: 65 additions & 43 deletions eng/pipelines/templates/stages/archetype-cpp-release.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
parameters:
Artifacts: []
ArtifactName: 'not-specified'
ServiceDirectory: not-specified
TestPipeline: false

stages:
Expand Down Expand Up @@ -122,52 +123,18 @@ stages:
Write-Host "##vso[task.setvariable variable=PrTitle]$title"
displayName: Set PR title

- task: Powershell@2
inputs:
filePath: eng/scripts/Initialize-VcpkgRelease.ps1
arguments: >-
-GitHubRepo "$(Build.Repository.Name)"
-PackageSpecPath $(Pipeline.Workspace)/packages/${{artifact.Name}}/package-info.json
-SourceDirectory $(Pipeline.Workspace)/packages/${{artifact.Name}}/vcpkg
-Verbose
pwsh: true
displayName: Initialize vcpkg release

# There are potential race conditions if this script runs
# in parallel against the same branch name. Release only
# one package at a time.
- pwsh: |
Write-Host "git clone https://github.com/azure-sdk/vcpkg $(Pipeline.Workspace)/vcpkg"
git clone https://github.com/azure-sdk/vcpkg $(Pipeline.Workspace)/vcpkg
if ($LASTEXITCODE -ne 0) {
Write-Error "Unable to check out vcpkg fork repo"
exit $LASTEXITCODE
}
Write-Host "##vso[task.setvariable variable=VcpkgWorkspace]$(Pipeline.Workspace)/vcpkg"
displayName: Clone vcpkg from upstream
- template: /eng/pipelines/templates/steps/vcpkg-clone.yml
parameters:
RepoOwner: azure-sdk
PRBranchName: $(PrBranchName)

# Check out the PR branch if it's already in remote.
# Ignore failures.
- pwsh: |
$ErrorActionPreference = "Continue"
git checkout "origin/$(PrBranchName)" 2>&1 | Out-Null
$LASTEXITCODE = 0 # This ignores any error from git checkout
git status
displayName: Checkout Previous PRBranch if it exist.
workingDirectory: $(VcpkgWorkspace)

- task: Powershell@2
inputs:
pwsh: true
targetType: filePath
filePath: eng/scripts/Update-VcpkgPort.ps1
arguments: >-
-ReleaseArtifactSourceDirectory "$(Pipeline.Workspace)/packages/${{ artifact.Name }}"
-PortDestinationDirectory 'ports/${{ artifact.VcpkgPortName }}'
-VcpkgPortName '${{ artifact.VcpkgPortName }}'
-GitCommitParameters '-c user.name="azure-sdk" -c user.email="azuresdk@microsoft.com"'
workingDirectory: $(VcpkgWorkspace)
displayName: Update vcpkg port
- template: /eng/pipelines/templates/steps/vcpkg-publish.yml
parameters:
ArtifactName: ${{ artifact.Name }}
VcpkgPortName: ${{ artifact.VcpkgPortName }}

# Set $(HasChanges) to $true so that
# create-pull-request.yml completes the push and PR
Expand Down Expand Up @@ -214,4 +181,59 @@ stages:
PRBranchName: increment-package-version-${{ parameters.ServiceDirectory }}-$(Build.BuildId)
CommitMsg: "Increment package version after release of ${{ artifact.name }}"
PRTitle: "Increment version for ${{ parameters.ServiceDirectory }} releases"
CloseAfterOpenForTesting: '${{ parameters.TestPipeline }}'
CloseAfterOpenForTesting: '${{ parameters.TestPipeline }}'

- ${{if eq(variables['System.TeamProject'], 'internal') }}:
- stage: Integration
dependsOn: Build
condition: or(eq(variables['PublishNightlyVcpkg'], 'true'), eq(variables['Build.Reason'],'Schedule'))
danieljurek marked this conversation as resolved.
Show resolved Hide resolved
jobs:
- job: PublishNightlyVcpkg
displayName: Publish to vcpkg nightly branch

pool:
name: azsdk-pool-mms-win-2019-general
vmImage: MMS2019

steps:
- checkout: self
- download: current
artifact: packages

- template: /eng/pipelines/templates/steps/generate-nightly-branch-name.yml

- template: /eng/pipelines/templates/steps/vcpkg-clone.yml
parameters:
RepoOwner: azure-sdk
PRBranchName: $(NightlyBranchName)

# Placing artifacts in the same branch in sequence reduces the
# potential for race conditions. If this job does fail to push
# because another job has already pushed to the same branch it
# can be re-run.
- ${{ each artifact in parameters.Artifacts }}:
- template: /eng/pipelines/templates/steps/vcpkg-publish.yml
parameters:
ArtifactName: ${{ artifact.Name }}
VcpkgPortName: ${{ artifact.VcpkgPortName }}
DisplayNameExtension: ${{ artifact.Name }}
TestRelease: true
TestReleaseRef: $(Build.SourceVersion)

- pwsh: |
Write-Host "git reset --soft HEAD~${{ length(parameters.Artifacts) }}"
git reset --soft HEAD~${{ length(parameters.Artifacts) }}
danieljurek marked this conversation as resolved.
Show resolved Hide resolved

Write-Host "git status"
git status
displayName: Reset commits to enable pushing a single commit
workingDirectory: $(Pipeline.Workspace)/vcpkg

- template: /eng/common/pipelines/templates/steps/git-push-changes.yml
parameters:
BaseRepoBranch: $(NightlyBranchName)
CommitMsg: Nightly vcpkg ports for ${{ parameters.ServiceDirectory }}
TargetRepoOwner: azure-sdk
TargetRepoName: vcpkg
WorkingDirectory: $(Pipeline.Workspace)/vcpkg
ScriptDirectory: $(System.DefaultWorkingDirectory)/eng/common/scripts
10 changes: 10 additions & 0 deletions eng/pipelines/templates/steps/generate-nightly-branch-name.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
steps:
danieljurek marked this conversation as resolved.
Show resolved Hide resolved
- pwsh: |
$branchName = if ($env:NIGHTLYBRANCHOVERRIDE) {
$env:NIGHTLYBRANCHOVERRIDE
} else {
$formattedDate = Get-Date -Format 'yyyy-MM-dd'
"nightly/$formattedDate"
}
danieljurek marked this conversation as resolved.
Show resolved Hide resolved
Write-Host "##vso[task.setvariable variable=NightlyBranchName]$branchName"
displayName: Set branch name in fork repo
25 changes: 25 additions & 0 deletions eng/pipelines/templates/steps/vcpkg-clone.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
parameters:
Workspace: $(Pipeline.Workspace)
RepoOwner: not-set
RepoName: vcpkg
PRBranchName: not-set

steps:
- pwsh: |
Write-Host "git clone https://github.com/${{ parameters.RepoOwner }}/${{ parameters.RepoName }} ${{ parameters.Workspace }}/vcpkg"
git clone https://github.com/${{ parameters.RepoOwner }}/${{ parameters.RepoName }} ${{ parameters.Workspace }}/vcpkg
if ($LASTEXITCODE -ne 0) {
Write-Error "Unable to check out vcpkg fork repo"
exit $LASTEXITCODE
}
displayName: Clone vcpkg from upstream

# Check out the PR branch if it's already in remote.
# Ignore failures.
- pwsh: |
$ErrorActionPreference = "Continue"
git checkout "origin/${{ parameters.PRBranchName }}" 2>&1 | Out-Null
$LASTEXITCODE = 0 # This ignores any error from git checkout
git status
displayName: Checkout Previous PRBranch if it exists
workingDirectory: ${{ parameters.Workspace }}/vcpkg
35 changes: 35 additions & 0 deletions eng/pipelines/templates/steps/vcpkg-publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
parameters:
GitHubRepo: $(Build.Repository.Name)
Workspace: $(Pipeline.Workspace)
ArtifactName: not-set
VcpkgPortName: not-set
DisplayNameExtension:
TestReleaseRef:
TestRelease: false

steps:
- task: Powershell@2
inputs:
filePath: eng/scripts/Initialize-VcpkgRelease.ps1
arguments: >-
-GitHubRepo "${{ parameters.GitHubRepo }}"
-PackageSpecPath ${{ parameters.Workspace }}/packages/${{ parameters.ArtifactName }}/package-info.json
-SourceDirectory ${{ parameters.Workspace }}/packages/${{ parameters.ArtifactName }}/vcpkg
-TestReleaseRef '${{ parameters.TestReleaseRef }}'
-Verbose
pwsh: true
displayName: Initialize vcpkg release ${{ parameters.DisplayNameExtension }}

- task: Powershell@2
inputs:
pwsh: true
targetType: filePath
filePath: eng/scripts/Update-VcpkgPort.ps1
arguments: >-
-ReleaseArtifactSourceDirectory "${{ parameters.Workspace }}/packages/${{ parameters.ArtifactName }}"
-PortDestinationDirectory 'ports/${{ parameters.VcpkgPortName }}'
-VcpkgPortName '${{ parameters.VcpkgPortName }}'
-GitCommitParameters '-c user.name="azure-sdk" -c user.email="azuresdk@microsoft.com"'
-TestRelease:$${{ parameters.TestRelease }}
workingDirectory: ${{ parameters.Workspace }}/vcpkg
displayName: Update vcpkg port ${{ parameters.DisplayNameExtension }}
27 changes: 22 additions & 5 deletions eng/scripts/Initialize-VcpkgRelease.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,10 @@ Location of the relevant package-info.json file
.PARAMETER GitHubRepo
Name of the GitHub repo (of the form Azure/azure-sdk-for-cpp)

.PARAMETER TestReleaseRef
If supplied update the portfile.cmake file's REF and SHA512 with values
associated with the given ref.

#>

param (
Expand All @@ -25,7 +29,9 @@ param (

[Parameter(Mandatory = $true)]
[ValidateNotNullOrEmpty()]
[string] $GitHubRepo
[string] $GitHubRepo,

[string] $TestReleaseRef
)

# If there's nothing in the "port" folder to upload set SkipVcpkgUpdate to true
Expand All @@ -36,8 +42,14 @@ if (!(Get-ChildItem -Path "$SourceDirectory/port/CONTROL")) {
exit
}

$packageSpec = Get-Content -Raw -Path $PackageSpecPath | ConvertFrom-Json
$tarGzUri = "https://github.com/$GitHubRepo/archive/$($packageSpec.packageName).tar.gz"
$tarGzUri = if (-not $TestReleaseRef) {
danieljurek marked this conversation as resolved.
Show resolved Hide resolved
Write-Verbose "Initializing Production Release"
$packageSpec = Get-Content -Raw -Path $PackageSpecPath | ConvertFrom-Json
"https://github.com/$GitHubRepo/archive/$($packageSpec.packageName).tar.gz"
} else {
Write-Verbose "Initializing Test Release"
"https://github.com/$GitHubRepo/archive/$TestReleaseRef.tar.gz"
}

Write-Host "Downloading tarball to compute hash from $tarGzUri"
$localTarGzPath = New-TemporaryFile
Expand All @@ -53,7 +65,12 @@ $portfileLocation = "$SourceDirectory/port/portfile.cmake"
# recommended in vcpkg documentation
# Before: " SHA512 1"
# After: " SHA512 f6cf1c16c52"
$newContent = Get-Content -Raw -Path $portfileLocation `
| ForEach-Object { $_ -replace '(SHA512\s+)1', "`${1}$sha512" }
$portFileContent = Get-Content -Raw -Path $portfileLocation
$newContent = $portFileContent -replace '(SHA512\s+)1', "`${1}$sha512"

if ($TestReleaseRef) {
Write-Verbose "Overriding REF with test release ref: $TestReleaseRef"
$newContent = $newContent -replace '(?m)^(\s+)REF azure.*$', "`${1}REF $TestReleaseRef"
}

$newContent | Set-Content $portfileLocation -NoNewLine
Loading