Skip LLO integration test that flakes #15311

samsondav · 2024-11-19T18:41:32Z

Requires

Supports

pavel-raykov · 2024-11-19T18:46:54Z

What is the goal of this PR? I.e., how removing the "interdependency" would help?

samsondav · 2024-11-19T18:55:59Z

@pavel-raykov not sure yet, I am going to try a few things to narrow down the flake

github-actions · 2024-11-19T19:10:35Z

Flaky Test Detector for `github.com/smartcontractkit/chainlink` project has failed ❌

Ran new or updated tests between develop and 666343e (try_fix_flakey_llo_integration_test).

View Flaky Detector Details | Compare Changes

Flaky Tests

Ran 49 unique tests. Below are the tests identified as flaky, with a pass ratio lower than the 99% threshold:

TestPackage                                                              TestName                                                                                                                                                                                                                                                                                                                                                                                 PassRatio  RunCount   Skipped
---------                                                                ---------                                                                                                                                                                                                                                                                                                                                                                                ---------  ---------  ---------
github.com/smartcontractkit/chainlink/v2/core/services/ocr2/plugins/llo  TestIntegration_LLO/using_legacy_verifier_configuration_contract,_produces_reports_in_v0.3_format/receives_at_least_one_report_per_channel_from_each_oracle_when_EAs_are_at_100%_reliability/test_on-chain_verification_-_node_63656363313530376463316464643732393539353163323930383838663039356164623930343464316237336436393665366466303635643638336264346663                          67%        3          false
github.com/smartcontractkit/chainlink/v2/core/services/ocr2/plugins/llo  TestIntegration_LLO/using_legacy_verifier_configuration_contract,_produces_reports_in_v0.3_format/receives_at_least_one_report_per_channel_from_each_oracle_when_EAs_are_at_100%_reliability/test_on-chain_verification_-_node_64616462643138346132643532366631656264643563303666646164393335396232323837353962346437663739643636363839666132353461616438353436#01/destination_verifier  67%        3          false
github.com/smartcontractkit/chainlink/v2/core/services/ocr2/plugins/llo  TestIntegration_LLO/using_legacy_verifier_configuration_contract,_produces_reports_in_v0.3_format/receives_at_least_one_report_per_channel_from_each_oracle_when_EAs_are_at_100%_reliability/test_on-chain_verification_-_node_36623739633537653661303935323339323832633034383138653936313132663366303361343030316261393761353634633233383532613366316561356663#01                       67%        3          false
github.com/smartcontractkit/chainlink/v2/core/services/ocr2/plugins/llo  TestIntegration_LLO/using_legacy_verifier_configuration_contract,_produces_reports_in_v0.3_format/receives_at_least_one_report_per_channel_from_each_oracle_when_EAs_are_at_100%_reliability/test_on-chain_verification_-_node_33623661323762636365623661343264363261336138643032613666306437333635333231353737316465323433613633616330343861313862353964613239                          67%        3          false
github.com/smartcontractkit/chainlink/v2/core/services/ocr2/plugins/llo  TestIntegration_LLO/using_legacy_verifier_configuration_contract,_produces_reports_in_v0.3_format/receives_at_least_one_report_per_channel_from_each_oracle_when_EAs_are_at_100%_reliability/test_on-chain_verification_-_node_36623739633537653661303935323339323832633034383138653936313132663366303361343030316261393761353634633233383532613366316561356663/destination_verifier     67%        3          false
github.com/smartcontractkit/chainlink/v2/core/services/ocr2/plugins/llo  TestIntegration_LLO/using_legacy_verifier_configuration_contract,_produces_reports_in_v0.3_format/receives_at_least_one_report_per_channel_from_each_oracle_when_EAs_are_at_100%_reliability/test_on-chain_verification_-_node_33623661323762636365623661343264363261336138643032613666306437333635333231353737316465323433613633616330343861313862353964613239#01/destination_verifier  67%        3          false
github.com/smartcontractkit/chainlink/v2/core/services/ocr2/plugins/llo  TestIntegration_LLO/using_legacy_verifier_configuration_contract,_produces_reports_in_v0.3_format/receives_at_least_one_report_per_channel_from_each_oracle_when_EAs_are_at_100%_reliability/test_on-chain_verification_-_node_33623661323762636365623661343264363261336138643032613666306437333635333231353737316465323433613633616330343861313862353964613239/destination_verifier     67%        3          false
github.com/smartcontractkit/chainlink/v2/core/services/ocr2/plugins/llo  TestIntegration_LLO/using_legacy_verifier_configuration_contract,_produces_reports_in_v0.3_format/receives_at_least_one_report_per_channel_from_each_oracle_when_EAs_are_at_100%_reliability/test_on-chain_verification_-_node_33623661323762636365623661343264363261336138643032613666306437333635333231353737316465323433613633616330343861313862353964613239#01                       67%        3          false
github.com/smartcontractkit/chainlink/v2/core/services/ocr2/plugins/llo  TestIntegration_LLO/using_legacy_verifier_configuration_contract,_produces_reports_in_v0.3_format/receives_at_least_one_report_per_channel_from_each_oracle_when_EAs_are_at_100%_reliability/test_on-chain_verification_-_node_63656363313530376463316464643732393539353163323930383838663039356164623930343464316237336436393665366466303635643638336264346663#01                       67%        3          false
github.com/smartcontractkit/chainlink/v2/core/services/ocr2/plugins/llo  TestIntegration_LLO/using_legacy_verifier_configuration_contract,_produces_reports_in_v0.3_format/receives_at_least_one_report_per_channel_from_each_oracle_when_EAs_are_at_100%_reliability/test_on-chain_verification_-_node_64616462643138346132643532366631656264643563303666646164393335396232323837353962346437663739643636363839666132353461616438353436                          67%        3          false
github.com/smartcontractkit/chainlink/v2/core/services/ocr2/plugins/llo  TestIntegration_LLO/using_legacy_verifier_configuration_contract,_produces_reports_in_v0.3_format/receives_at_least_one_report_per_channel_from_each_oracle_when_EAs_are_at_100%_reliability/test_on-chain_verification_-_node_64616462643138346132643532366631656264643563303666646164393335396232323837353962346437663739643636363839666132353461616438353436#01                       67%        3          false
github.com/smartcontractkit/chainlink/v2/core/services/ocr2/plugins/llo  TestIntegration_LLO/using_legacy_verifier_configuration_contract,_produces_reports_in_v0.3_format                                                                                                                                                                                                                                                                                        67%        3          false
github.com/smartcontractkit/chainlink/v2/core/services/ocr2/plugins/llo  TestIntegration_LLO/using_legacy_verifier_configuration_contract,_produces_reports_in_v0.3_format/receives_at_least_one_report_per_channel_from_each_oracle_when_EAs_are_at_100%_reliability/test_on-chain_verification_-_node_63656363313530376463316464643732393539353163323930383838663039356164623930343464316237336436393665366466303635643638336264346663#01/destination_verifier  67%        3          false
github.com/smartcontractkit/chainlink/v2/core/services/ocr2/plugins/llo  TestIntegration_LLO                                                                                                                                                                                                                                                                                                                                                                      67%        3          false
github.com/smartcontractkit/chainlink/v2/core/services/ocr2/plugins/llo  TestIntegration_LLO/using_legacy_verifier_configuration_contract,_produces_reports_in_v0.3_format/receives_at_least_one_report_per_channel_from_each_oracle_when_EAs_are_at_100%_reliability/test_on-chain_verification_-_node_64616462643138346132643532366631656264643563303666646164393335396232323837353962346437663739643636363839666132353461616438353436/destination_verifier     67%        3          false
github.com/smartcontractkit/chainlink/v2/core/services/ocr2/plugins/llo  TestIntegration_LLO/using_legacy_verifier_configuration_contract,_produces_reports_in_v0.3_format/receives_at_least_one_report_per_channel_from_each_oracle_when_EAs_are_at_100%_reliability/test_on-chain_verification_-_node_36623739633537653661303935323339323832633034383138653936313132663366303361343030316261393761353634633233383532613366316561356663#01/destination_verifier  67%        3          false
github.com/smartcontractkit/chainlink/v2/core/services/ocr2/plugins/llo  TestIntegration_LLO/using_legacy_verifier_configuration_contract,_produces_reports_in_v0.3_format/receives_at_least_one_report_per_channel_from_each_oracle_when_EAs_are_at_100%_reliability/test_on-chain_verification_-_node_63656363313530376463316464643732393539353163323930383838663039356164623930343464316237336436393665366466303635643638336264346663/destination_verifier     67%        3          false
github.com/smartcontractkit/chainlink/v2/core/services/ocr2/plugins/llo  TestIntegration_LLO/using_legacy_verifier_configuration_contract,_produces_reports_in_v0.3_format/receives_at_least_one_report_per_channel_from_each_oracle_when_EAs_are_at_100%_reliability/test_on-chain_verification_-_node_36623739633537653661303935323339323832633034383138653936313132663366303361343030316261393761353634633233383532613366316561356663                          67%        3          false
github.com/smartcontractkit/chainlink/v2/core/services/ocr2/plugins/llo  TestIntegration_LLO/using_legacy_verifier_configuration_contract,_produces_reports_in_v0.3_format/receives_at_least_one_report_per_channel_from_each_oracle_when_EAs_are_at_100%_reliability                                                                                                                                                                                             67%        3          false

github-actions · 2024-11-19T19:11:40Z

AER Report: CI Core ran successfully ✅

aer_workflow , commit

AER Report: Operator UI CI ran successfully ✅

aer_workflow , commit

github-actions · 2024-11-19T20:29:20Z

I see you updated files related to core. Please run pnpm changeset in the root directory to add a changeset as well as in the text include at least one of the following tags:

#added For any new functionality added.
#breaking_change For any functionality that requires manual action for the node to boot.
#bugfix For bug fixes.
#changed For any change to the existing functionality.
#db_update For any feature that introduces updates to database schema.
#deprecation_notice For any upcoming deprecation functionality.
#internal For changesets that need to be excluded from the final changelog.
#nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
#removed For any functionality/config that is removed.
#updated For any functionality that is updated.
#wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

🎖️ No JIRA issue number found in: PR title, commit message, or branch name. Please include the issue ID in one of these.

pavel-raykov · 2024-11-20T12:11:11Z

core/services/ocr2/plugins/llo/integration_test.go

@@ -342,11 +342,6 @@ func TestIntegration_LLO(t *testing.T) {
 	multiplier := decimal.New(1, 18)
 	expirationWindow := time.Hour / time.Second

-	reqs := make(chan request, 100000)
-	serverKey := csakey.MustNewV2XXXTestingOnly(big.NewInt(-1))
-	serverPubKey := serverKey.PublicKey


the t.Run(...) tests do not call t.Parallel() , so I am not sure how moving this to individual tests would help.

Would it make sense to keep this PR only about disabling tests and not modifying them?

No, because if a previous test fails it causes a subsequent test to fail for no good reason if you re-use the same server

Thanks, this makes total sense. In this case, would it be possible to put this stanza inside of startMercuryServer instead of duplicating it?

samsondav requested review from a team as code owners November 19, 2024 18:41

samsondav requested a review from pavel-raykov November 19, 2024 18:41

samsondav force-pushed the try_fix_flakey_llo_integration_test branch from 3194dd7 to 666343e Compare November 19, 2024 18:55

samsondav changed the title ~~Split out mercury servers to remove test interdependency~~ Attempt to fix LLO integration test flakes Nov 19, 2024

Split out mercury servers to remove test interdependency

f44ab8f

samsondav force-pushed the try_fix_flakey_llo_integration_test branch from 666343e to f44ab8f Compare November 19, 2024 20:06

Disable the offending test

842df78

samsondav force-pushed the try_fix_flakey_llo_integration_test branch from 8a46b87 to 842df78 Compare November 19, 2024 20:16

samsondav changed the title ~~Attempt to fix LLO integration test flakes~~ Skip LLO integration test that flakes Nov 19, 2024

samsondav enabled auto-merge November 19, 2024 20:18

krehermann previously approved these changes Nov 19, 2024

View reviewed changes

jmank88 previously approved these changes Nov 19, 2024

View reviewed changes

Clean up loggin a tiny bit

b34a6a8

samsondav dismissed stale reviews from jmank88 and krehermann via b34a6a8 November 19, 2024 20:27

samsondav requested a review from a team as a code owner November 19, 2024 20:27

samsondav requested review from krehermann and jmank88 November 19, 2024 20:42

patrickhuie19 approved these changes Nov 19, 2024

View reviewed changes

pavel-raykov reviewed Nov 20, 2024

View reviewed changes

samsondav requested a review from pavel-raykov November 20, 2024 13:35

jmank88 approved these changes Nov 20, 2024

View reviewed changes

samsondav added this pull request to the merge queue Nov 20, 2024

Merged via the queue into develop with commit a9c9ddf Nov 20, 2024
145 of 146 checks passed

samsondav deleted the try_fix_flakey_llo_integration_test branch November 20, 2024 18:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Skip LLO integration test that flakes #15311

Skip LLO integration test that flakes #15311

samsondav commented Nov 19, 2024

pavel-raykov commented Nov 19, 2024

samsondav commented Nov 19, 2024

github-actions bot commented Nov 19, 2024

github-actions bot commented Nov 19, 2024 •

edited

Loading

github-actions bot commented Nov 19, 2024

pavel-raykov Nov 20, 2024

samsondav Nov 20, 2024

pavel-raykov Nov 20, 2024

Skip LLO integration test that flakes #15311

Skip LLO integration test that flakes #15311

Conversation

samsondav commented Nov 19, 2024

Requires

Supports

pavel-raykov commented Nov 19, 2024

samsondav commented Nov 19, 2024

github-actions bot commented Nov 19, 2024

Flaky Test Detector for github.com/smartcontractkit/chainlink project has failed ❌

Flaky Tests

github-actions bot commented Nov 19, 2024 • edited Loading

AER Report: CI Core ran successfully ✅

AER Report: Operator UI CI ran successfully ✅

github-actions bot commented Nov 19, 2024

pavel-raykov Nov 20, 2024

Choose a reason for hiding this comment

samsondav Nov 20, 2024

Choose a reason for hiding this comment

pavel-raykov Nov 20, 2024

Choose a reason for hiding this comment

Flaky Test Detector for `github.com/smartcontractkit/chainlink` project has failed ❌

github-actions bot commented Nov 19, 2024 •

edited

Loading