Error using Lodestar as a failover

[wemeetagain]

From discord (https://discord.com/channels/593655374469660673/743858262864167062/1090175408340733962):

I have a setup where a teku validator uses Lodestar as a failover.

however when it tries to connect, Lodestar returns this error

[Chozowarrior]

also tested on Goerli with same results

[Chozowarrior]

The other way around. Teku as failover for Lodestar VC does work but does display some errors in the logging:

[dapplion]

The other way around. Teku as failover for Lodestar VC does work but does display some errors in the logging:

What version of Teku are you using? execution_optimistic should be implemented by all clients now

[Chozowarrior]

The other way around. Teku as failover for Lodestar VC does work but does display some errors in the logging:

What version of Teku are you using? execution_optimistic should be implemented by all clients now

yes the otherway around issue has been resolved but the main issue (Lodestar as failover for Teku has not been resolved yet).

[dapplion]

@Chozowarrior Could you share the version of the Teku beacon node that the Lodestar validator client of your screen capture? Then I can check with Teku devs to see if they are complying with the beacon API spec

[Chozowarrior]

@Chozowarrior Could you share the version of the Teku beacon node that the Lodestar validator client of your screen capture? Then I can check with Teku devs to see if they are complying with the beacon API spec

You mean validator client right? I was using Teku 23.3.1 as a validator at the time. I was also in contact with the Teku team regarding this issue. since there was also a different issue on their side because after failing to connect to Lodestar it would not successfully attest after connecting to a different failover. That was this issue in case it is of any interest Consensys/teku#7016 I was in contact with Stefan Bratanov regarding this.

[nflaig]

@Chozowarrior Are you still running Lodestar as a failover node with Teku VC? The errors you reported should be fixed now by #5722 and #5784 but those should not have caused actual issues (just noise).

I am running Lodestar and Teku in a 1:1 setup which works great with the latest releases. Would be interesting to know if this issue is resolved by now. If you deploy from unstable branch the noisy eventstream errors should be gone too.

[Chozowarrior]

@nflaig yes I am still using Lodestar as a failover for some validators and as a primary for others as well. It is working for last month or so indeed so that is great. The behavior together with Teku only shows some errors that are maybe a bit overdone. The Teku dev told me the default behaviour is that the Teku-VC sends the attestations to the primary node plus all failover nodes to maximize the propogation. However this results in error messages on the Lodestar when the attestation has already been gossiped (see below). The behaviour in Nimbus which is a 2nd failover is different as this just blindly attests without the check that Lodestar seems to have.

However this issue is can indeed be closed but wonder whether the below is still interesting to look into as this error look messy, maybe info would be better here.

jul 21 23:01:25 ssay-NUC8i5BEH lodestar[1557]: Jul-21 23:01:25.336[api] error: Error on submitPoolAttestations [0] slot=6929105, index=40 code=ATTESTATION_ERROR_ATTESTATION_ALREADY_KNOWN, targetEpoch=216534, validatorIndex=XXXX

[nflaig]

Thanks for the detailed answer. Great to hear that it works for you as well. Indeed, the error might be a bit annoying but as far as I know this does not cause any issues.

@tuyennhv wdyt? could we log this error with that specific code to debug instead?

[twoeths]

@nflaig I think we should implement ignoreIfKnown flag, it's to be true for api validation but false for gossip validation

lodestar/packages/beacon-node/src/chain/validation/attestation.ts

Line 79 in 3257345

const prioritizeBls = true;

even when we use Lodestar as the primary node, submitted attestations can reach our node by gossip before the api since vc can published to fallback node already

[nflaig]

Closing as errors mentioned in this issue have been resolved #5323 (comment).

The other error mentioned in #5323 (comment) is tracked in a different issue.

• Ignore known attestations submitted through API #6021