Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CircleCI to GHA migration #7154

Merged
merged 155 commits into from
Oct 10, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
155 commits
Select commit Hold shift + click to select a range
266c412
Disable CircleCI
gokhangulbiz Aug 7, 2023
50dd50f
Flaky test debugging on GHA
gokhangulbiz Aug 7, 2023
f5c680f
Reorder required params.
gokhangulbiz Aug 7, 2023
b3adf9c
Reorganize env variables and inputs.
gokhangulbiz Aug 7, 2023
48a987e
Reorganize upload artifact folder structure
gokhangulbiz Aug 16, 2023
2c9375b
Initial commit for citus testing on GHA
gokhangulbiz Aug 16, 2023
488271f
Fix image suffix.
gokhangulbiz Aug 16, 2023
13b62fe
Fix file path
gokhangulbiz Aug 16, 2023
1d21f50
Fix parameter binding
gokhangulbiz Aug 16, 2023
6f883f3
Use root user for container images
gokhangulbiz Aug 16, 2023
a5a65cc
Fix container image suffix
gokhangulbiz Aug 16, 2023
80db3f1
naming
gokhangulbiz Aug 16, 2023
58da0c5
Artifact upload optimization
gokhangulbiz Aug 16, 2023
204df4b
download artifact path fix
gokhangulbiz Aug 16, 2023
acc18ec
download artifact path fix
gokhangulbiz Aug 16, 2023
6005499
Fix invalid attribute
gokhangulbiz Aug 16, 2023
fbe7675
Fix downloaded artifacts folder
gokhangulbiz Aug 16, 2023
0ed6ea1
Minor fixes
gokhangulbiz Aug 16, 2023
a52e4a0
Fix artifacts path
gokhangulbiz Aug 17, 2023
9c47dba
Matrix customization for failtester
gokhangulbiz Aug 17, 2023
91e3229
Stack trace and upload covarage run always
gokhangulbiz Aug 17, 2023
01e2596
Fix for check-sql-snapshots and check-style issues
gokhangulbiz Aug 17, 2023
c1ff845
Run container image with root user
gokhangulbiz Aug 17, 2023
812077e
Change default user to circleci
gokhangulbiz Aug 17, 2023
cf62647
Safe directory configuration
gokhangulbiz Aug 17, 2023
6127fe3
Fix newline issues
gokhangulbiz Aug 17, 2023
0ba0c68
Set user to root for sql snapshots
gokhangulbiz Aug 17, 2023
f4bc979
Fetch all branches to be able to run git diff..
gokhangulbiz Aug 17, 2023
e5b5581
Merge remote-tracking branch 'origin/main' into circleci-gha-migration
gokhangulbiz Aug 17, 2023
97a8a31
Include check-failure in matrix
gokhangulbiz Aug 17, 2023
0fd2e9b
Reorganize failtester matrix
gokhangulbiz Aug 17, 2023
0e3a292
Use env variable for PG_MAJOR
gokhangulbiz Aug 17, 2023
20f8baa
Do not override PG_MAJOR
gokhangulbiz Aug 17, 2023
c77306d
Try to evaluate env variable case-sensitivity
gokhangulbiz Aug 17, 2023
4dd0a8a
Test
gokhangulbiz Aug 17, 2023
5afb9a1
Add PG_MAJOR as Github Env Var
gokhangulbiz Aug 17, 2023
f14938a
Add PG_MAJOR as Github Env Var
gokhangulbiz Aug 17, 2023
fb937db
Add PG_MAJOR as Github Env Var
gokhangulbiz Aug 17, 2023
1bc08df
Naming
gokhangulbiz Aug 17, 2023
f028827
Env var fix
gokhangulbiz Aug 17, 2023
c0a1998
Update matrix
gokhangulbiz Aug 17, 2023
c9c2d87
Update matrix
gokhangulbiz Aug 17, 2023
59dbf06
cdc and tap tests
gokhangulbiz Aug 17, 2023
5626f58
Fix log processing
gokhangulbiz Aug 17, 2023
f655f6d
Set timeout
gokhangulbiz Aug 17, 2023
451d6d8
fix step summary
gokhangulbiz Aug 17, 2023
1363824
fix wildcard path
gokhangulbiz Aug 17, 2023
903196c
minor fixes
gokhangulbiz Aug 17, 2023
13ceea3
narrow down upload path
gokhangulbiz Aug 17, 2023
f67f468
better wildcard usage
gokhangulbiz Aug 17, 2023
609fbc8
Diff markdown
gokhangulbiz Aug 18, 2023
088d70a
Cleanup
gokhangulbiz Aug 18, 2023
7d7f282
Fix double quote
gokhangulbiz Aug 18, 2023
3a40fce
Naming refactorings
gokhangulbiz Aug 18, 2023
16d03c9
Cleanup
gokhangulbiz Aug 18, 2023
c1904c4
Cleanup
gokhangulbiz Aug 18, 2023
dd088d3
Refactoring
gokhangulbiz Aug 18, 2023
48ec1b8
test-arbitrary-configs
gokhangulbiz Aug 18, 2023
b7e6eff
check-query-generator and pgupgrade
gokhangulbiz Aug 18, 2023
5347a51
fix
gokhangulbiz Aug 18, 2023
9419b4b
fix parallelization
gokhangulbiz Aug 18, 2023
eb73c20
Parallelism
gokhangulbiz Aug 21, 2023
072d7be
Remove workflow env variables.
gokhangulbiz Aug 21, 2023
a9c4bb5
Remove empty new line
gokhangulbiz Aug 21, 2023
16a55cb
test-citus-upgrade
gokhangulbiz Aug 21, 2023
5c4852d
upload-coverage
gokhangulbiz Aug 21, 2023
698a5f0
ch_benchmark
gokhangulbiz Aug 21, 2023
114574b
tpcc_benchmark
gokhangulbiz Aug 21, 2023
76196d5
test-flakyness
gokhangulbiz Aug 21, 2023
d2d7907
Fix test-arbitrary-config
gokhangulbiz Aug 21, 2023
0a63f31
fix test-flakyness
gokhangulbiz Aug 21, 2023
c46eeb2
Fix test-citus-upgrade
gokhangulbiz Aug 21, 2023
2f65a2c
Fix test-arbitrary-configs
gokhangulbiz Aug 21, 2023
de7d896
Enable test-citus
gokhangulbiz Aug 21, 2023
a80186d
Fix install_extension
gokhangulbiz Aug 21, 2023
eb6cb48
Merge remote-tracking branch 'origin/main' into circleci-gha-migration
gokhangulbiz Aug 21, 2023
291924f
Merge branch 'main' of https://github.com/citusdata/citus
gokhangulbiz Aug 21, 2023
d877f2d
Fix install extension
gokhangulbiz Aug 21, 2023
b691fc0
Fix install extension
gokhangulbiz Aug 21, 2023
ac17f07
Fix install extension
gokhangulbiz Aug 21, 2023
5f0f5cd
Fix install extension
gokhangulbiz Aug 21, 2023
ef20671
Fix install extension
gokhangulbiz Aug 21, 2023
035c6da
minor fixes
gokhangulbiz Aug 21, 2023
0af5f40
Minor updates
gokhangulbiz Aug 21, 2023
4aab137
fixe upgrade tests
gokhangulbiz Aug 21, 2023
27b47a2
Path fix
gokhangulbiz Aug 21, 2023
f4a358e
Fix path and permissions
gokhangulbiz Aug 21, 2023
e3689c5
Fix test-arbitrary-configs
gokhangulbiz Aug 21, 2023
95c9d26
skip_installation for test-citus-upgrade
gokhangulbiz Aug 21, 2023
abe4c42
fix condition
gokhangulbiz Aug 21, 2023
2ed1e76
Cleanup
gokhangulbiz Aug 21, 2023
5fe3ae7
16core vm
gokhangulbiz Aug 21, 2023
7779d89
Fix condition check
gokhangulbiz Aug 21, 2023
601a5f0
32core vm for build
gokhangulbiz Aug 21, 2023
2a6dbc6
32core vm for test-citus
gokhangulbiz Aug 21, 2023
e39a421
16core
gokhangulbiz Aug 21, 2023
1774385
revert 16core vm
gokhangulbiz Aug 22, 2023
e0fab47
Disable test-arbitrary-configs till we have 16core VMs
gokhangulbiz Aug 22, 2023
262aff2
Improve diff output
gokhangulbiz Aug 22, 2023
ae588a3
Adds concurrency
gokhangulbiz Aug 22, 2023
7b1c662
Fix for diffs output
gokhangulbiz Aug 22, 2023
b22ecb0
typo fix
gokhangulbiz Aug 22, 2023
784169c
flakyness test
gokhangulbiz Aug 22, 2023
d33088a
test-flakyness container image fix
gokhangulbiz Aug 23, 2023
2398b2f
Skip_test_flakyness added.
gokhangulbiz Aug 23, 2023
a6ec872
Move workflow inputs to repository variables
gokhangulbiz Aug 23, 2023
bb9bdf7
Cleanup
gokhangulbiz Aug 23, 2023
509c173
Fix test-flakyness
gokhangulbiz Aug 23, 2023
ad18d5d
Always save logs and results
gokhangulbiz Aug 23, 2023
e7d75f0
Fix container image name
gokhangulbiz Aug 23, 2023
3a76759
Add concurrency for flaky
gokhangulbiz Aug 23, 2023
dd8dfb9
Improve run name for build and test
gokhangulbiz Aug 23, 2023
cfa58e9
fix if condition
gokhangulbiz Aug 23, 2023
fa38db2
Preserve newlines
gokhangulbiz Aug 23, 2023
ca8aa3f
Fix condition
gokhangulbiz Aug 23, 2023
16f360a
better naming
gokhangulbiz Aug 23, 2023
f92c0fa
Revert "flakyness test"
gokhangulbiz Aug 24, 2023
fdc35c4
Provide path for codeclimate results
gokhangulbiz Aug 24, 2023
818bd9c
Disable test-citus temporarily
gokhangulbiz Aug 24, 2023
04c9d07
Fix upload coverage
gokhangulbiz Aug 24, 2023
a73da25
fix upload coverage
gokhangulbiz Aug 24, 2023
d5f1747
fix upload coverage
gokhangulbiz Aug 24, 2023
88c92cc
codecov and codeclimate
gokhangulbiz Aug 24, 2023
1c39c8b
Revert "Disable test-citus temporarily"
gokhangulbiz Aug 24, 2023
b291b09
Enable PR triggers
gokhangulbiz Aug 24, 2023
a3a6df2
Additional configuration for codecov
gokhangulbiz Aug 24, 2023
f961ed8
Remove run-name for build task
gokhangulbiz Aug 24, 2023
5c5375e
Fix secrets access in a composite action
gokhangulbiz Aug 25, 2023
752eaff
Better naming work build and test
gokhangulbiz Aug 25, 2023
142a631
enable gcov
gokhangulbiz Aug 25, 2023
82c97a0
Renaming
gokhangulbiz Aug 28, 2023
b46ac3f
Merge pull request #1 from gokhangulbiz/circleci-gha-migration-pr-tri…
gokhangulbiz Aug 28, 2023
c272bb5
Merge branch 'main', remote-tracking branch 'origin' into circleci-gh…
gokhangulbiz Aug 28, 2023
2493890
Merge branch 'main' into circleci-gha-migration
gokhangulbiz Sep 5, 2023
7a5325f
Merge branch 'main' into circleci-gha-migration
gokhangulbiz Sep 11, 2023
2700fb9
Merge branch 'main' into circleci-gha-migration
gokhangulbiz Sep 20, 2023
daa327d
Add PG16 to matrix.
gokhangulbiz Sep 20, 2023
9d41b76
Use pg14 for citus tests.
gokhangulbiz Sep 20, 2023
ca96868
Merge branch 'main' into circleci-gha-migration
gokhangulbiz Sep 25, 2023
e94a700
Remove duplicated action."
gokhangulbiz Sep 25, 2023
b498956
modify echo flag.
gokhangulbiz Sep 25, 2023
adae8b6
Merge branch 'circleci-gha-migration' of https://github.com/citusdata…
gokhangulbiz Sep 25, 2023
ce9da5e
Use ubuntu-22.04
gokhangulbiz Sep 25, 2023
5c3c9a7
1ES Hosted pool runners for build and arbitrary config tests
gokhangulbiz Sep 26, 2023
fbe81bc
Preserve new lines for regression outputs
gokhangulbiz Oct 3, 2023
6577272
debug - open 57637 port
gokhangulbiz Oct 3, 2023
3ec61f9
Debug - use public nameserver to have similar network setup with Circ…
gokhangulbiz Oct 3, 2023
4fad9b6
Merge branch 'main' into circleci-gha-migration
gokhangulbiz Oct 3, 2023
25486bb
Add more comments on custom DNS settings.
gokhangulbiz Oct 3, 2023
f5b7d5b
Use github hosted runners for build jobs.
gokhangulbiz Oct 4, 2023
2563376
Provide a workaround for test splitting in arbitrary configs tests.
gokhangulbiz Oct 6, 2023
8b14839
Improve namings and outputs
gokhangulbiz Oct 6, 2023
80b7909
Enable CircleCI
gokhangulbiz Oct 6, 2023
0d3fa0b
Merge branch 'main' into circleci-gha-migration
gokhangulbiz Oct 6, 2023
ae3478f
Merge branch 'main' into circleci-gha-migration
hanefi Oct 10, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -698,7 +698,6 @@ jobs:
workflows:
version: 2
flaky_test_debugging:
when: << pipeline.parameters.flaky_test >>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we want to revert the changes to this file before merge, right?

jobs:
- build:
name: build-flaky-15
Expand All @@ -714,8 +713,6 @@ workflows:
runs: << pipeline.parameters.flaky_test_runs_per_job >>

build_and_test:
when:
not: << pipeline.parameters.flaky_test >>
jobs:
- build:
name: build-14
Expand Down
23 changes: 23 additions & 0 deletions .github/actions/parallelization/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
name: 'Parallelization matrix'
inputs:
count:
required: false
default: 32
outputs:
json:
value: ${{ steps.generate_matrix.outputs.json }}
runs:
using: "composite"
steps:
- name: Generate parallelization matrix
id: generate_matrix
shell: bash
run: |-
json_array="{\"include\": ["
for ((i = 1; i <= ${{ inputs.count }}; i++)); do
json_array+="{\"id\":\"$i\"},"
done
json_array=${json_array%,}
json_array+=" ]}"
echo "json=$json_array" >> "$GITHUB_OUTPUT"
echo "json=$json_array"
38 changes: 38 additions & 0 deletions .github/actions/save_logs_and_results/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
name: save_logs_and_results
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we not have a human-readable name instead?

Suggested change
name: save_logs_and_results
name: Save logs and results

I assumed quotation marks are not necessary, but I am not so sure about that.

inputs:
folder:
required: false
default: "log"
runs:
using: composite
steps:
- uses: actions/upload-artifact@v3.1.1
name: Upload logs
with:
name: ${{ inputs.folder }}
if-no-files-found: ignore
path: |
src/test/**/proxy.output
src/test/**/results/
src/test/**/tmp_check/master/log
src/test/**/tmp_check/worker.57638/log
src/test/**/tmp_check/worker.57637/log
src/test/**/*.diffs
src/test/**/out/ddls.sql
src/test/**/out/queries.sql
src/test/**/logfile_*
/tmp/pg_upgrade_newData_logs
- name: Publish regression.diffs
run: |-
diffs="$(find src/test/regress -name "*.diffs" -exec cat {} \;)"
if ! [ -z "$diffs" ]; then
echo '```diff' >> $GITHUB_STEP_SUMMARY
echo -E "$diffs" >> $GITHUB_STEP_SUMMARY
echo '```' >> $GITHUB_STEP_SUMMARY
echo -E $diffs
fi
shell: bash
- name: Print stack traces
run: "./ci/print_stack_trace.sh"
if: failure()
shell: bash
35 changes: 35 additions & 0 deletions .github/actions/setup_extension/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: setup_extension
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we not have a human-readable name here?

inputs:
pg_major:
required: false
skip_installation:
required: false
default: false
type: boolean
runs:
using: composite
steps:
- name: Expose $PG_MAJOR to Github Env
run: |-
if [ -z "${{ inputs.pg_major }}" ]; then
echo "PG_MAJOR=${PG_MAJOR}" >> $GITHUB_ENV
else
echo "PG_MAJOR=${{ inputs.pg_major }}" >> $GITHUB_ENV
fi
shell: bash
- uses: actions/download-artifact@v3.0.1
with:
name: build-${{ env.PG_MAJOR }}
- name: Install Extension
if: ${{ inputs.skip_installation == 'false' }}
run: tar xfv "install-$PG_MAJOR.tar" --directory /
shell: bash
- name: Configure
run: |-
chown -R circleci .
git config --global --add safe.directory ${GITHUB_WORKSPACE}
gosu circleci ./configure --without-pg-version-check
shell: bash
- name: Enable core dumps
run: ulimit -c unlimited
shell: bash
27 changes: 27 additions & 0 deletions .github/actions/upload_coverage/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
name: coverage
inputs:
flags:
required: false
codecov_token:
required: true
runs:
using: composite
steps:
- uses: codecov/codecov-action@v3
with:
flags: ${{ inputs.flags }}
token: ${{ inputs.codecov_token }}
verbose: true
gcov: true
- name: Create codeclimate coverage
run: |-
lcov --directory . --capture --output-file lcov.info
lcov --remove lcov.info -o lcov.info '/usr/*'
sed "s=^SF:$PWD/=SF:=g" -i lcov.info # relative pats are required by codeclimate
mkdir -p /tmp/codeclimate
cc-test-reporter format-coverage -t lcov -o /tmp/codeclimate/${{ inputs.flags }}.json lcov.info
shell: bash
- uses: actions/upload-artifact@v3.1.1
with:
path: "/tmp/codeclimate/*.json"
name: codeclimate
Loading
Loading