Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TT-1832 Add slack notification for nightly run with detailed flaky test summary #15446

Merged
merged 44 commits into from
Dec 4, 2024
Merged

TT-1832 Add slack notification for nightly run with detailed flaky test summary #15446

merged 44 commits into from
Dec 4, 2024
+86 −16

Conversation

lukaszcl
Copy link
Collaborator

@lukaszcl lukaszcl commented Nov 28, 2024 •
edited
Loading

This PR introduces Slack notifications for nightly flaky test detection runs. The Slack message includes the following information:

  • Total Failed Tests: Displays the total number of tests that failed during the run.
  • Failed to Non-Failed Ratio: Provides the ratio of failed tests to non-failed tests in percentage form.

Also, this PR increases retention-days for flaky test results to 90 days (requested by @AKhozya).

@lukaszcl lukaszcl requested review from a team as code owners November 28, 2024 09:05
@lukaszcl lukaszcl marked this pull request as draft November 28, 2024 09:14
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 3, 2024 •
edited
Loading

AER Report: CI Core ran successfully ✅

aer_workflow , commit

AER Report: Operator UI CI ran successfully ✅

aer_workflow , commit

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 3, 2024

Flakeguard Summary

Setting Value
Project github.com/smartcontractkit/chainlink/v2
Max Pass Ratio 100.00%
Test Run Count 3
Race Detection false
Excluded Tests TestChainComponents

Found Flaky Tests ❌

Category Total
Tests 329
Panicked Tests 0
Raced Tests 0
Flaky Tests 2
Flaky Test Ratio 0.60%
Runs 987
Passes 981
Failures 6
Skips 0
Pass Ratio 99.39%
Name Pass Ratio Panicked? Timed Out? Race? Runs Successes Failures Skips Package Package Panicked? Avg Duration Code Owners
TestResolver_CronSpec 0.00% false false false 3 0 3 0 github.com/smartcontractkit/chainlink/v2/core/web/resolver false 0s Unknown
TestResolver_CronSpec/cron_spec_success 0.00% false false false 3 0 3 0 github.com/smartcontractkit/chainlink/v2/core/web/resolver false 40ms Unknown

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 3, 2024

Flakeguard Summary

Setting Value
Project github.com/smartcontractkit/chainlink/v2
Max Pass Ratio 100.00%
Test Run Count 1
Race Detection false
Excluded Tests TestChainComponents

Found Flaky Tests ❌

Name Pass Ratio Panicked? Timed Out? Race? Runs Successes Failures Skips Package Package Panicked? Avg Duration Code Owners
TestResolver_CronSpec 0.00% false false false 1 0 1 0 github.com/smartcontractkit/chainlink/v2/core/web/resolver false 0s @smartcontractkit/deployment-automation, @smartcontractkit/foundations, @smartcontractkit/core
TestResolver_CronSpec/cron_spec_success 0.00% false false false 1 0 1 0 github.com/smartcontractkit/chainlink/v2/core/web/resolver false 40ms @smartcontractkit/deployment-automation, @smartcontractkit/foundations, @smartcontractkit/core

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 3, 2024

Flakeguard Summary

Setting Value
Project github.com/smartcontractkit/chainlink/v2
Max Pass Ratio 100.00%
Test Run Count 1
Race Detection false
Excluded Tests TestChainComponents

Found Flaky Tests ❌

Category Total
Tests 329
Panicked Tests 0
Raced Tests 0
Flaky Tests 2
Flaky Test Ratio 0.60%
Runs 329
Passes 327
Failures 2
Skips 0
Pass Ratio 99.39%
Name Pass Ratio Panicked? Timed Out? Race? Runs Successes Failures Skips Package Package Panicked? Avg Duration Code Owners
TestResolver_CronSpec 0.00% false false false 1 0 1 0 github.com/smartcontractkit/chainlink/v2/core/web/resolver false 0s @smartcontractkit/deployment-automation, @smartcontractkit/foundations, @smartcontractkit/core
TestResolver_CronSpec/cron_spec_success 0.00% false false false 1 0 1 0 github.com/smartcontractkit/chainlink/v2/core/web/resolver false 40ms @smartcontractkit/deployment-automation, @smartcontractkit/foundations, @smartcontractkit/core

@lukaszcl lukaszcl changed the base branch from develop to fix-nightly-flakeguard-job December 3, 2024 16:08
Base automatically changed from fix-nightly-flakeguard-job to develop December 3, 2024 16:28
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 4, 2024

Flakeguard Summary

Setting Value
Project github.com/smartcontractkit/chainlink/v2
Max Pass Ratio 100.00%
Test Run Count 3
Race Detection false
Excluded Tests TestChainComponents

Found Flaky Tests ❌

Category Total
Tests 329
Panicked Tests 0
Raced Tests 0
Flaky Tests 2
Flaky Test Ratio 0.60%
Runs 987
Passes 981
Failures 6
Skips 0
Pass Ratio 99.39%
Name Pass Ratio Panicked? Timed Out? Race? Runs Successes Failures Skips Package Package Panicked? Avg Duration Code Owners
TestResolver_CronSpec 0.00% false false false 3 0 3 0 github.com/smartcontractkit/chainlink/v2/core/web/resolver false 0s @smartcontractkit/deployment-automation, @smartcontractkit/foundations, @smartcontractkit/core
TestResolver_CronSpec/cron_spec_success 0.00% false false false 3 0 3 0 github.com/smartcontractkit/chainlink/v2/core/web/resolver false 40ms @smartcontractkit/deployment-automation, @smartcontractkit/foundations, @smartcontractkit/core

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 4, 2024

Flakeguard Summary

Setting Value
Project github.com/smartcontractkit/chainlink/v2
Max Pass Ratio 100.00%
Test Run Count 3
Race Detection false
Excluded Tests TestChainComponents

Found Flaky Tests ❌

Category Total
Tests 329
Panicked Tests 0
Raced Tests 0
Flaky Tests 2
Flaky Test Ratio 0.60%
Runs 987
Passes 981
Failures 6
Skips 0
Pass Ratio 99.39%
Name Pass Ratio Panicked? Timed Out? Race? Runs Successes Failures Skips Package Package Panicked? Avg Duration Code Owners
TestResolver_CronSpec 0.00% false false false 3 0 3 0 github.com/smartcontractkit/chainlink/v2/core/web/resolver false 0s @smartcontractkit/deployment-automation, @smartcontractkit/foundations, @smartcontractkit/core
TestResolver_CronSpec/cron_spec_success 0.00% false false false 3 0 3 0 github.com/smartcontractkit/chainlink/v2/core/web/resolver false 43.333333ms @smartcontractkit/deployment-automation, @smartcontractkit/foundations, @smartcontractkit/core

@lukaszcl lukaszcl marked this pull request as ready for review December 4, 2024 13:43
@lukaszcl lukaszcl added this pull request to the merge queue Dec 4, 2024
Merged via the queue into develop with commit 52676f1 Dec 4, 2024
96 checks passed
@lukaszcl lukaszcl deleted the TT-1832-Implement-Slack-Notification-for-Nightly-Run-with-Detailed-Flaky-Test-Summary branch December 4, 2024 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gheorghestrimtu gheorghestrimtu gheorghestrimtu approved these changes

@skudasov skudasov skudasov approved these changes

@kalverra kalverra kalverra approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@lukaszcl @kalverra @skudasov @gheorghestrimtu