Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve recoverer efficiency #13779

Closed
wants to merge 1 commit into from
Closed

Improve recoverer efficiency #13779

wants to merge 1 commit into from

Conversation

ferglor
Copy link
Collaborator

@ferglor ferglor commented Jul 8, 2024

In this PR, we're making some changes to how the log recoverer works:

  • Instead of storing payloads in slices, we're storing payloads in maps, indexable by workID
  • We're storing map keys as slices, which can be sorted as currently expected
  • We also track the number payloads per upkeep using a map
  • We recover up to 10 filters sequentially, rather than in parallel, as to not flood the log poller

With these changes, we do not see an increase in the number of payloads processed, but the time it takes to process the same number of payloads as the version of the recoverer in develop, is much lower:

Function name recoverFilter recoverFilter GetRecoveryProposals GetRecoveryProposals recover recover
Code versions Old New Old New Old New
Average time (last 1000 calls) 0.9638761953 0.01344082459 1.304877573 0.000848521156 1.371147103 0.1414009405
Total time (last 1000 calls) 963.8761953 13.44082459 1304.877573 0.848521156 1371.147103 141.4009405
Max time (last 1000 calls) 27.78149292 0.401462046 29.50123379 0.002757988 28.84100264 0.577488912
x Improvement 1 71.71 🥇 1 1537.83 🥇 1 9.7 🥇

Develop version test run: https://chainlink-core.slack.com/archives/C03KJ5S7KEK/p1720652788875089
This PR test run: https://chainlink-core.slack.com/archives/C03KJ5S7KEK/p1720645152371079

PR to add timings to base develop calls: #13790

@ferglor ferglor requested a review from a team as a code owner July 8, 2024 14:59
Copy link
Contributor

github-actions bot commented Jul 8, 2024

I see you updated files related to core. Please run pnpm changeset in the root directory to add a changeset as well as in the text include at least one of the following tags:

  • #added For any new functionality added.
  • #breaking_change For any functionality that requires manual action for the node to boot.
  • #bugfix For bug fixes.
  • #changed For any change to the existing functionality.
  • #db_update For any feature that introduces updates to database schema.
  • #deprecation_notice For any upcoming deprecation functionality.
  • #internal For changesets that need to be excluded from the final changelog.
  • #nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
  • #removed For any functionality/config that is removed.
  • #updated For any functionality that is updated.
  • #wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

@ferglor ferglor force-pushed the feature/AUTO-10074-2 branch from bcedb62 to bb824ae Compare July 9, 2024 11:05
@ferglor ferglor force-pushed the feature/AUTO-10074-2 branch from 208f976 to c33c800 Compare July 9, 2024 12:37
@ferglor ferglor force-pushed the feature/AUTO-10074-2 branch from c33c800 to 5e6a172 Compare July 9, 2024 15:08
@ferglor ferglor force-pushed the feature/AUTO-10074-2 branch from 2e1a96a to 9074868 Compare July 10, 2024 00:02
@ferglor ferglor force-pushed the feature/AUTO-10074-2 branch from 9074868 to 8c69b7b Compare July 10, 2024 20:38
@ferglor ferglor requested a review from a team as a code owner July 10, 2024 20:38
@ferglor ferglor changed the base branch from develop to spike/load-test-gen-fix July 10, 2024 20:39
@ferglor ferglor requested review from a team and RensR as code owners July 10, 2024 20:39
@ferglor ferglor requested a review from a team July 10, 2024 20:39
@ferglor ferglor requested a review from a team as a code owner July 10, 2024 20:39
@ferglor ferglor changed the base branch from spike/load-test-gen-fix to spike/load-gen-volume-fix July 10, 2024 20:39
WIP track recoverables with maps

Track time

Add logs

Timestamp recover

Bump recovery batch size
@ferglor ferglor force-pushed the feature/AUTO-10074-2 branch from 619e5e4 to bd08023 Compare July 19, 2024 19:29
@ferglor ferglor requested review from a team and samsondav as code owners July 19, 2024 19:29
@ferglor ferglor changed the base branch from spike/load-gen-volume-fix to develop July 19, 2024 19:30
@ferglor ferglor removed request for a team, samsondav and RensR July 19, 2024 19:38
@cl-sonarqube-production
Copy link

Quality Gate failed Quality Gate failed

Failed conditions
25 New Major Issues (required ≤ 5)

See analysis details on SonarQube

Catch issues before they fail your Quality Gate with our IDE extension SonarLint SonarLint

@ferglor ferglor changed the title Reduce recoverer workload Improve recoverer efficiency Jul 19, 2024
@ferglor ferglor closed this Jul 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant