Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate redemptions with the monitoring system #11 #3664

Open
3 of 7 tasks
lukasz-zimnoch opened this issue Jul 3, 2023 · 0 comments
Open
3 of 7 tasks

Integrate redemptions with the monitoring system #11 #3664

lukasz-zimnoch opened this issue Jul 3, 2023 · 0 comments

Comments

@lukasz-zimnoch
Copy link
Member

lukasz-zimnoch commented Jul 3, 2023

We need to integrate redemptions with the monitoring system described in the monitoring and telemetry document This work needs to be done in two places.

Monitoring of redemption-related chain events

We need to enhance tbtc-v2-monitoring tool and add the following items:

Tasks

Monitoring of the wallet maintainer

We need to monitor our instance of the wallet maintainer. That means we need to add the following items:

  • Alert: Maintainer down
  • Alert: Maintainer failed to submit a redemption proposal
  • Alert: Maintainer lost connection with their Ethereum node
  • Alert: Maintainer lost connection with their Bitcoin node

Monitoring of the SPV maintainer

We need to monitor our instance of the SPV maintainer. That means we need to add the following items:

  • Alert: Maintainer failed to submit SPV proof
  • Alert: Maintainer detected the difficulty mismatch
  • Alert: Maintainer lost connection with their Ethereum node
  • Alert: Maintainer lost connection with their Bitcoin node
r-czajkowski added a commit to keep-network/tbtc-v2 that referenced this issue Aug 2, 2023
Refs: keep-network/keep-core#3664

Here we expose the `Bridge.getRedemptionRequestedEvents` method that
allows fetching past redemption requested events from the `Bridge`
contract.
pdyraga added a commit to keep-network/tbtc-v2 that referenced this issue Aug 3, 2023
Refs: keep-network/keep-core#3664
Depends on: #677

This changeset adds redemptions to the monitoring system. Specifically,
two new system events are supported now:

### Redemption requested

An **informational system event** indicating that a new redemption was
requested from the on-chain Bridge contract. This event is directly sent
to Discord as a notification that does not require any action.

### Large redemption requested

A **warning system event** indicating that a large redemption was
requested from the on-chain Bridge contract. This event is sent to
Sentry hub and should get team’s attention. The default action is making
sure that the redemption is not a result of a malicious action, and if
not, that the redemption is handled correctly by the system.
pdyraga added a commit to keep-network/tbtc-v2 that referenced this issue Aug 3, 2023
Refs: keep-network/keep-core#3664
Depends on: #678

This changeset adds one new system event to the monitoring system:

### Stale redemption

A **warning system event** indicating that a redemption request became
stale, i.e. was not handled within the expected time. This event is sent
to Sentry hub and should get team’s attention. The default action is
investigating the cause of the extended processing time as this alert
may be an early sign of a malfunctioning wallet or may indicate a
problem with the maintainer bot.
@lukasz-zimnoch lukasz-zimnoch removed their assignment Nov 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants
@lukasz-zimnoch and others