Update MultiNode with latest changes #15058

DylanTinianov · 2024-10-31T18:58:37Z

Updates MultiNode with the latest changes made during the Solana integration.
Ticket: https://smartcontract-it.atlassian.net/browse/BCFR-1069

New TransactionSender implementation uses generic RESULT and moves the error classification to the RPC client.

Also fixes the following issues:

Node lifecycle OutOfSync issue
Transaction Sender SendTransaction context issue
MultiNode DoAll context issue

Soak Test

`CommitHash` `ced19c2` (latest)
Namespace: `soak-ocr-v2-arbitrum-sepolia-d3677`

Grafana Logs

github-actions · 2024-10-31T19:32:10Z

AER Report: CI Core ran successfully ✅

aer_workflow , commit

AER Report: Operator UI CI ran successfully ✅

aer_workflow , commit

cl-sonarqube-production · 2024-10-31T19:53:14Z

Quality Gate passed

Issues
3 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
95.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

… into BCFR-1069-fix-multinode-issues

github-actions · 2024-11-07T17:35:12Z

Flaky Test Detector for `./go.mod` project has failed ❌

Ran new or updated tests between develop and 7b20152 (BCFR-1069-fix-multinode-issues).

View Flaky Detector Details | Compare Changes

Failed Tests

Ran 462 tests in total for all affected test packages. Below are the tests identified as flaky, with a pass ratio lower than the 100% threshold:

TestPackage                                             TestName                                                                                                                                 PassRatio  RunCount   Skipped
---------                                               ---------                                                                                                                                ---------  ---------  ---------
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_unreachableLoop                                                                                                   0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_outOfSyncLoop/adds_finalized_block_is_not_increasing_flag,_if_there_is_no_new_finalized_heads_for_too_long        0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_invalidChainIDLoop/on_failed_chainID_call_becomes_unreachable                                                     0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestMultiNode_Dial/Closes_started_nodes_on_failure                                                                                       0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_start                                                                                                             0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_start/on_isSyncing_transitions_to_syncing                                                                         0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestMultiNode_Dial/Fails_without_nodes                                                                                                   0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestMultiNode_Dial/Fails_with_wrong_node's_chainID                                                                                       0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestMultiNode_Dial/Fails_with_wrong_send_only_node's_chainID                                                                             0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_outOfSyncLoop/if_initial_dial_fails,_transitions_to_unreachable                                                   0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_outOfSyncLoop/becomes_alive_if_it_receives_a_newer_head                                                           0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_unreachableLoop/on_chain_ID_mismatch_transitions_to_invalidChainID                                                0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_aliveLoop/stays_alive_while_below_pollFailureThreshold_and_resets_counter_on_success                              0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_invalidChainIDLoop/on_invalid_dial_becomes_unreachable                                                            0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestMultiNode_CheckLease/NodeStates_returns_proper_states                                                                                0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_start/if_fails_on_initial_dial,_becomes_unreachable                                                               0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_aliveLoop/when_behind_but_SyncThreshold=0,_stay_alive                                                             0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_aliveLoop/If_finality_tag_is_not_enabled_updates_finalized_block_metric_using_finality_depth_and_latest_head      0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestStartSendOnlyNode/Can_recover_from_chainID_mismatch                                                                                  0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_invalidChainIDLoop/on_chainID_mismatch_keeps_trying                                                               0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_start/on_successful_verification_without_isSyncing_becomes_alive                                                  0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestMultiNode_Dial                                                                                                                       0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_outOfSyncLoop/becomes_alive_on_new_finalized_block                                                                0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_start/on_successful_verification_becomes_alive                                                                    0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_aliveLoop/If_finalized_heads_channel_is_closed,_transitions_to_unreachable                                        0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestStartSendOnlyNode/Default_ChainID(0)_produces_warn_and_skips_checks                                                                  0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestStartSendOnlyNode/Start_with_Random_ChainID                                                                                          0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_unreachableLoop/on_failed_redial,_keeps_trying                                                                    0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestStartSendOnlyNode                                                                                                                    0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_aliveLoop/if_initial_subscribe_fails,_transitions_to_unreachable                                                  0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_aliveLoop/with_threshold_poll_failures,_transitions_to_unreachable                                                0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_invalidChainIDLoop/on_successful_verification_becomes_alive                                                       0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_unreachableLoop/on_successful_verification_without_isSyncing_becomes_alive                                        0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestStartSendOnlyNode/becomes_unusable_if_initial_dial_fails                                                                             0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_invalidChainIDLoop                                                                                                0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_start/if_chainID_verification_fails,_becomes_unreachable                                                          0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_aliveLoop/when_no_new_finalized_heads_received_for_threshold,_transitions_to_out_of_sync                          0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_outOfSyncLoop/on_subscription_termination_becomes_unreachable                                                     0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_outOfSyncLoop/on_subscription_termination_becomes_unreachable#01                                                  0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_invalidChainIDLoop/on_successful_verification_without_isSyncing_becomes_alive                                     0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_aliveLoop/when_behind_more_than_SyncThreshold_but_we_are_the_last_live_node,_forcibly_stays_alive                 0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_start/on_chain_ID_mismatch_transitions_to_invalidChainID                                                          0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestMultiNode_CheckLease/Misconfigured_lease_check_period_won't_start                                                                    0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_unreachableLoop/returns_on_closed                                                                                 0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_unreachableLoop/on_successful_verification_becomes_alive                                                          0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_aliveLoop/if_remote_RPC_connection_is_closed_transitions_to_unreachable                                           0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_outOfSyncLoop/if_chainID_does_not_match,_transitions_to_invalidChainID                                            0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_SyncingLoop/on_IsSyncing_-_keeps_trying                                                                           0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_aliveLoop/Stays_alive_and_waits_for_signal                                                                        0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_aliveLoop/when_no_new_heads_received_for_threshold_but_we_are_the_last_live_node,_forcibly_stays_alive            0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_unreachableLoop/on_syncing_status_check_failure,_keeps_trying                                                     0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_aliveLoop/Logs_warning_if_latest_finalized_block_is_not_valid                                                     0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_outOfSyncLoop/becomes_unreachable_if_head_channel_is_closed#01                                                    0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_SyncingLoop/on_successful_verification_becomes_alive                                                              0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestMultiNode_Dial/Fails_if_node_fails                                                                                                   0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_aliveLoop/If_fails_to_subscribe_to_latest_finalized_blocks,_transitions_to_unreachable                            0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_outOfSyncLoop/if_fails_to_subscribe,_becomes_unreachable                                                          0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_outOfSyncLoop/Stays_out-of-sync_if_received_new_head,_but_lags_behind_pool                                        0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_SyncingLoop                                                                                                       0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_outOfSyncLoop/if_syncing,_transitions_to_syncing                                                                  0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_outOfSyncLoop/if_fails_to_subscribe_to_finalized,_becomes_unreachable                                             0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_invalidChainIDLoop/returns_on_closed                                                                              0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_aliveLoop/If_finalized_subscription_returns_an_error,_transitions_to_unreachable                                  0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_SyncingLoop/on_failed_Syncing_check_-_becomes_unreachable                                                         0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestStartSendOnlyNode/Can_recover_from_chainID_verification_failure                                                                      0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_aliveLoop/when_behind_more_than_SyncThreshold,_transitions_to_out_of_sync                                         0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_outOfSyncLoop/becomes_unreachable_if_head_channel_is_closed                                                       0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestMultiNode_CheckLease/Round_robin_disables_lease_check                                                                                0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestMultiNode_CheckLease/Lease_check_updates_active_node                                                                                 0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_aliveLoop/when_no_new_heads_received_for_threshold,_transitions_to_out_of_sync                                    0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_outOfSyncLoop/if_fails_to_fetch_syncing_status,_transitions_to_unreachable                                        0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_aliveLoop/with_threshold_poll_failures,_but_we_are_the_last_node_alive,_forcibly_keeps_it_alive                   0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_aliveLoop/On_new_finalized_block_updates_corresponding_metric                                                     0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_outOfSyncLoop/becomes_alive_if_there_is_no_other_nodes                                                            0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_unreachableLoop/on_failed_chainID_verification,_keep_trying                                                       0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_start/if_syncing_verification_fails,_becomes_unreachable                                                          0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_unreachableLoop/on_syncing,_transitions_to_syncing_state                                                          0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestMultiNode_CheckLease                                                                                                                 0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_outOfSyncLoop                                                                                                     0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_aliveLoop/when_no_new_finalized_heads_received_for_threshold_but_we_are_the_last_live_node,_forcibly_stays_alive  0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_aliveLoop                                                                                                         0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestMultiNode_Dial/Fails_on_send_only_node_failure                                                                                       0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_outOfSyncLoop/on_old_blocks_stays_outOfSync_and_returns_on_close                                                  0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_aliveLoop/rpc_closed_head_channel                                                                                 0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestUnit_NodeLifecycle_outOfSyncLoop/if_fail_to_get_chainID,_transitions_to_unreachable                                                  0%         1          false
github.com/smartcontractkit/chainlink/v2/common/client  TestMultiNode_Dial/Starts_successfully_with_healthy_nodes                                                                                0%         1          false

common/client/node_lifecycle.go

core/chains/evm/client/rpc_client.go

common/client/transaction_sender_test.go

core/chains/evm/client/rpc_client.go

common/client/transaction_sender_test.go

… into BCFR-1069-fix-multinode-issues

Update MultiNode to latest

e649c93

DylanTinianov self-assigned this Oct 31, 2024

DylanTinianov requested a review from dhaidashenko October 31, 2024 18:58

DylanTinianov added 2 commits October 31, 2024 15:07

changeset

492e57f

lint

adc9c1d

DylanTinianov marked this pull request as ready for review October 31, 2024 19:23

DylanTinianov requested review from a team as code owners October 31, 2024 19:23

DylanTinianov requested a review from Atrax1 October 31, 2024 19:23

DylanTinianov changed the title ~~Update MultiNode to latest~~ Update MultiNode with latest changes Oct 31, 2024

jmank88 previously approved these changes Oct 31, 2024

View reviewed changes

defer reportWg

630ea8d

DylanTinianov dismissed jmank88’s stale review via 630ea8d November 5, 2024 14:36

Merge branch 'develop' of https://github.com/smartcontractkit/chainlink…

d41096e

… into BCFR-1069-fix-multinode-issues

DylanTinianov requested a review from jmank88 November 5, 2024 15:09

Use latest changes

7b20152

DylanTinianov added 2 commits November 7, 2024 12:46

Update node_lifecycle.go

cd49ba2

lint

c77b3b5

patrick-dowell reviewed Nov 9, 2024

View reviewed changes

common/client/node_lifecycle.go Show resolved Hide resolved

DylanTinianov requested a review from patrick-dowell November 12, 2024 15:05

dhaidashenko reviewed Nov 12, 2024

View reviewed changes

DylanTinianov added 4 commits November 13, 2024 11:17

Remove TxError

fffd833

Use cfg clientErrors

9ae86dc

Update transaction_sender_test.go

94e77fc

Use require

8a5d75c

DylanTinianov requested a review from dhaidashenko November 13, 2024 18:08

Merge branch 'develop' of https://github.com/smartcontractkit/chainlink…

ced19c2

… into BCFR-1069-fix-multinode-issues

DylanTinianov requested a review from a team as a code owner November 15, 2024 17:21

DylanTinianov force-pushed the BCFR-1069-fix-multinode-issues branch from 210d85f to ced19c2 Compare November 15, 2024 18:28

Merge branch 'develop' of https://github.com/smartcontractkit/chainlink…

345323c

… into BCFR-1069-fix-multinode-issues

jmank88 approved these changes Nov 20, 2024

View reviewed changes

dhaidashenko approved these changes Nov 20, 2024

View reviewed changes

jmank88 added this pull request to the merge queue Nov 20, 2024

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Nov 20, 2024

jmank88 added this pull request to the merge queue Nov 20, 2024

Merged via the queue into develop with commit d4d1456 Nov 20, 2024
143 checks passed

jmank88 deleted the BCFR-1069-fix-multinode-issues branch November 20, 2024 19:23

This was referenced Nov 20, 2024

[DO NOT MERGE] Changeset Release Preview - v2.19.0 #13148

Draft

[DO NOT MERGE] Changeset Release Preview - v2.19.0 galactic-beyond/chainlink#1

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update MultiNode with latest changes #15058

Update MultiNode with latest changes #15058

DylanTinianov commented Oct 31, 2024 •

edited

Loading

github-actions bot commented Oct 31, 2024 •

edited

Loading

cl-sonarqube-production bot commented Oct 31, 2024

github-actions bot commented Nov 7, 2024

Update MultiNode with latest changes #15058

Update MultiNode with latest changes #15058

Conversation

DylanTinianov commented Oct 31, 2024 • edited Loading

Soak Test

github-actions bot commented Oct 31, 2024 • edited Loading

AER Report: CI Core ran successfully ✅

AER Report: Operator UI CI ran successfully ✅

cl-sonarqube-production bot commented Oct 31, 2024

Quality Gate passed

github-actions bot commented Nov 7, 2024

Flaky Test Detector for ./go.mod project has failed ❌

Failed Tests

DylanTinianov commented Oct 31, 2024 •

edited

Loading

github-actions bot commented Oct 31, 2024 •

edited

Loading

Flaky Test Detector for `./go.mod` project has failed ❌