invalid payload because bad withdrawals root

[ews]

Hi,

This started happening recently, any idea what could it be and if I may have some misconfiguration?

lighthousebeacon  | Jul 2XXXXX  WARN Builder returned invalid payload        parent_hash: 0x081ebd07f5be91bbcae6537ca2ea4ae800778ef9ef831c72edb3e1bb1f1e9655, relay_block_hash: 0xfaeaed2303dedc6d1a733a26b6986f5b7dc64db61eedbfd2c57e63a839dd6e20, reason: payload withdrawals root was 0xd8f7…c1e7 not 0x6b78…f590, info: using local payload, service: exec    

[aimxhaisse]

I can see this as well, I think one relayer/building is not providing proper blocks (get header forces to use a specific block root so the returned block should match it).

[aimxhaisse]

Equivalent log for Teku:

Unable to obtain a valid bid from builder. Falling back to local execution engine.

[aimxhaisse]

I've picked randomly 5 block proposals affected by this from today, all point to the titan relayer, I'm engaging with them and will post an update here.

[aimxhaisse]

Yup, they confirm and this will be fixed:

I think on mev-boost side it could also be improved: the parent hash could be checked against what was requested to filter out invalid headers.

[metachris]

Thanks for updating in this thread!

[aimxhaisse]

Just a quick update: still investigating / trying to understand what's going on, it's not clear yet.

[aimxhaisse]

It was root-caused on Gattaca's side and they rolled out a fix ; a post-mortem will likely come.

Still watching on our side but so far we did not see it again (we had it about 10-20 times a day before).

[adaszko]

I've picked randomly 5 block proposals affected by this from today, all point to the titan relayer, I'm engaging with them and will post an update here.

I removed Titan as per your observation from the relay list and now got this issue with Ultra Sound Money too:

Jul 25 17:34:35 vouch-teku-0 vouch[6572]: {"level":"trace","service":"blockrelay","impl":"standard","slot":9592071,"provider":"https://boost-relay.flashbots.net/","value":"36998626614129998","delta":"1844344095457129","selected":false,"time":"2024-07-25T17:34:35Z","message":"Auction participant"}
Jul 25 17:34:35 vouch-teku-0 vouch[6572]: {"level":"trace","service":"blockrelay","impl":"standard","slot":9592071,"provider":"https://aestus.live/","value":"33410142859292318","delta":"5432827850294809","selected":false,"time":"2024-07-25T17:34:35Z","message":"Auction participant"}
Jul 25 17:34:35 vouch-teku-0 vouch[6572]: {"level":"trace","service":"blockrelay","impl":"standard","slot":9592071,"provider":"https://bloxroute.max-profit.blxrbdn.com/","value":"37921768946206115","delta":"921201763381012","selected":false,"time":"2024-07-25T17:34:35Z","message":"Auction participant"}
Jul 25 17:34:35 vouch-teku-0 vouch[6572]: {"level":"trace","service":"blockrelay","impl":"standard","slot":9592071,"provider":"https://bloxroute.regulated.blxrbdn.com/","value":"37920346827122186","delta":"922623882464941","selected":false,"time":"2024-07-25T17:34:35Z","message":"Auction participant"}
Jul 25 17:34:35 vouch-teku-0 vouch[6572]: {"level":"trace","service":"blockrelay","impl":"standard","slot":9592071,"provider":"https://relay.ultrasound.money/","value":"38842970709587127","delta":"0","selected":true,"time":"2024-07-25T17:34:35Z","message":"Auction participant"}
Jul 25 17:34:35 vouch-teku-0 vouch[6572]: {"level":"trace","service":"blockrelay","impl":"standard","slot":9592071,"provider":"https://agnostic-relay.net/","value":"37193703340057695","delta":"1649267369529432","selected":false,"time":"2024-07-25T17:34:35Z","message":"Auction participant"}
Jul 25 17:34:35 vouch-teku-0 vouch[6572]: {"level":"trace","service":"blockrelay","impl":"standard","slot":9592071,"provider":"https://relay.edennetwork.io/","value":"25049854379178757","delta":"13793116330408370","selected":false,"time":"2024-07-25T17:34:35Z","message":"Auction participant"}
Jul 25 17:34:35 vouch-teku-0 vouch[6572]: {"level":"trace","service":"beaconblockproposer","impl":"standard","header":{"version":"deneb","data":{"message":{"header":{"parent_hash":"0x4dbc147ad0c0a8ef4c3742ab94b25ef8b93d593951484cf124bb3c08230c45b4","fee_recipient":"0x4838B106FCe9647Bdf1E7877BF73cE8B0BAD5f97","state_root":"0x25bcbb9e278b7c6ceceece6177849ebcffa637f0212a264ad5b08fe7e04be6d3","receipts_root":"0x0f73c0d71a689>
Jul 25 17:34:35 vouch-teku-0 vouch[6572]: {"level":"trace","service":"client","impl":"http","id":"6aa5f889","address":"lighthouse-beacon-node-nethermind-0.eu-west-1.compute.stakefish:39000","endpoint":"/eth/v1/validator/blinded_blocks/9592071?randao_reveal=0x976c548c025e20aba3f4f108c29e1b58f00f24c281d3227acb234e6b6a462f9534e8cf042fb534a790e39b8fe6d472e011cafa735cdd427e8b3c694f3da47ae18cce68bc36ecacb7625536f2ae76c8ed81394>
Jul 25 17:34:36 vouch-teku-0 vouch[6572]: {"level":"trace","service":"beaconblockproposer","impl":"standard","time":"2024-07-25T17:34:36Z","message":"Pre-obtained proposal"}
Jul 25 17:34:36 vouch-teku-0 vouch[6572]: {"level":"trace","service":"beaconblockproposer","impl":"standard","proposal":{"Version":"deneb","Bellatrix":null,"Capella":null,"Deneb":{"slot":"9592071","proposer_index":"503872","parent_root":"0xb1d0a7b64bcf95d82388ae51fbb48b889b4529c2bfbd1ef23c0b0213dfbd538e","state_root":"0xc2922792165b259d84eea866b55509bc4ab7acd4d0c9e38dcffc76e618da7c55","body":{"randao_reveal":"0x976c548c0>
Jul 25 17:34:36 vouch-teku-0 vouch[6572]: 9d496b","0x97faa19cb8df83a79c245e8f4f79ef4e1acd58047c0dea7c0d8f12cdb1601749af504014c07f146713beeb5b01959de4","0xb88bbc90abce6fc393a75356d254de61db55342aa880d612dd2a28406248a594086e3372eee78bcda37fbbd04bd07e0f","0xb17b1df60d50fec07d03f39f207f8de9fced01f01715533adbbbdeb286bfd2b0e3e41d70bcc638383c191e4a7dd0739f","0x8da5c26fd24bf6d1ba805b4832667a64ef07b440125be34838895489c1bc05e5b784>
Jul 25 17:34:36 vouch-teku-0 vouch[6572]: {"level":"debug","service":"beaconblockproposer","impl":"standard","slot":9592071,"proposal_transactions_root":"0xcf4c97712bc72d9f917df099495e87d8d9f606f4e3fb836a37cedbafb20f755c","auction_transactions_root":"0x425e6f58de78af80ecd70a163c83ed76f719e5c6aa310ba9fed6267c3bf51111","time":"2024-07-25T17:34:36Z","message":"Transactions root mismatch"}
Jul 25 17:34:36 vouch-teku-0 vouch[6572]: {"level":"error","service":"beaconblockproposer","impl":"standard","slot":9592071,"error":"transactions root mismatch","time":"2024-07-25T17:34:36Z","message":"Failed to obtain blinded proposal"}
Jul 25 17:34:36 vouch-teku-0 vouch[6572]: {"level":"warn","service":"beaconblockproposer","impl":"standard","slot":9592071,"time":"2024-07-25T17:34:36Z","message":"Failed to propose with auction; attempting to propose without auction"}

[aimxhaisse]

I've picked randomly 5 block proposals affected by this from today, all point to the titan relayer, I'm engaging with them and will post an update here.

I removed Titan as per your observation from the relay list and now got this issue with Ultra Sound Money too:

Jul 25 17:34:35 vouch-teku-0 vouch[6572]: {"level":"trace","service":"blockrelay","impl":"standard","slot":9592071,"provider":"https://boost-relay.flashbots.net/","value":"36998626614129998","delta":"1844344095457129","selected":false,"time":"2024-07-25T17:34:35Z","message":"Auction participant"}
Jul 25 17:34:35 vouch-teku-0 vouch[6572]: {"level":"trace","service":"blockrelay","impl":"standard","slot":9592071,"provider":"https://aestus.live/","value":"33410142859292318","delta":"5432827850294809","selected":false,"time":"2024-07-25T17:34:35Z","message":"Auction participant"}
Jul 25 17:34:35 vouch-teku-0 vouch[6572]: {"level":"trace","service":"blockrelay","impl":"standard","slot":9592071,"provider":"https://bloxroute.max-profit.blxrbdn.com/","value":"37921768946206115","delta":"921201763381012","selected":false,"time":"2024-07-25T17:34:35Z","message":"Auction participant"}
Jul 25 17:34:35 vouch-teku-0 vouch[6572]: {"level":"trace","service":"blockrelay","impl":"standard","slot":9592071,"provider":"https://bloxroute.regulated.blxrbdn.com/","value":"37920346827122186","delta":"922623882464941","selected":false,"time":"2024-07-25T17:34:35Z","message":"Auction participant"}
Jul 25 17:34:35 vouch-teku-0 vouch[6572]: {"level":"trace","service":"blockrelay","impl":"standard","slot":9592071,"provider":"https://relay.ultrasound.money/","value":"38842970709587127","delta":"0","selected":true,"time":"2024-07-25T17:34:35Z","message":"Auction participant"}
Jul 25 17:34:35 vouch-teku-0 vouch[6572]: {"level":"trace","service":"blockrelay","impl":"standard","slot":9592071,"provider":"https://agnostic-relay.net/","value":"37193703340057695","delta":"1649267369529432","selected":false,"time":"2024-07-25T17:34:35Z","message":"Auction participant"}
Jul 25 17:34:35 vouch-teku-0 vouch[6572]: {"level":"trace","service":"blockrelay","impl":"standard","slot":9592071,"provider":"https://relay.edennetwork.io/","value":"25049854379178757","delta":"13793116330408370","selected":false,"time":"2024-07-25T17:34:35Z","message":"Auction participant"}
Jul 25 17:34:35 vouch-teku-0 vouch[6572]: {"level":"trace","service":"beaconblockproposer","impl":"standard","header":{"version":"deneb","data":{"message":{"header":{"parent_hash":"0x4dbc147ad0c0a8ef4c3742ab94b25ef8b93d593951484cf124bb3c08230c45b4","fee_recipient":"0x4838B106FCe9647Bdf1E7877BF73cE8B0BAD5f97","state_root":"0x25bcbb9e278b7c6ceceece6177849ebcffa637f0212a264ad5b08fe7e04be6d3","receipts_root":"0x0f73c0d71a689>
Jul 25 17:34:35 vouch-teku-0 vouch[6572]: {"level":"trace","service":"client","impl":"http","id":"6aa5f889","address":"lighthouse-beacon-node-nethermind-0.eu-west-1.compute.stakefish:39000","endpoint":"/eth/v1/validator/blinded_blocks/9592071?randao_reveal=0x976c548c025e20aba3f4f108c29e1b58f00f24c281d3227acb234e6b6a462f9534e8cf042fb534a790e39b8fe6d472e011cafa735cdd427e8b3c694f3da47ae18cce68bc36ecacb7625536f2ae76c8ed81394>
Jul 25 17:34:36 vouch-teku-0 vouch[6572]: {"level":"trace","service":"beaconblockproposer","impl":"standard","time":"2024-07-25T17:34:36Z","message":"Pre-obtained proposal"}
Jul 25 17:34:36 vouch-teku-0 vouch[6572]: {"level":"trace","service":"beaconblockproposer","impl":"standard","proposal":{"Version":"deneb","Bellatrix":null,"Capella":null,"Deneb":{"slot":"9592071","proposer_index":"503872","parent_root":"0xb1d0a7b64bcf95d82388ae51fbb48b889b4529c2bfbd1ef23c0b0213dfbd538e","state_root":"0xc2922792165b259d84eea866b55509bc4ab7acd4d0c9e38dcffc76e618da7c55","body":{"randao_reveal":"0x976c548c0>
Jul 25 17:34:36 vouch-teku-0 vouch[6572]: 9d496b","0x97faa19cb8df83a79c245e8f4f79ef4e1acd58047c0dea7c0d8f12cdb1601749af504014c07f146713beeb5b01959de4","0xb88bbc90abce6fc393a75356d254de61db55342aa880d612dd2a28406248a594086e3372eee78bcda37fbbd04bd07e0f","0xb17b1df60d50fec07d03f39f207f8de9fced01f01715533adbbbdeb286bfd2b0e3e41d70bcc638383c191e4a7dd0739f","0x8da5c26fd24bf6d1ba805b4832667a64ef07b440125be34838895489c1bc05e5b784>
Jul 25 17:34:36 vouch-teku-0 vouch[6572]: {"level":"debug","service":"beaconblockproposer","impl":"standard","slot":9592071,"proposal_transactions_root":"0xcf4c97712bc72d9f917df099495e87d8d9f606f4e3fb836a37cedbafb20f755c","auction_transactions_root":"0x425e6f58de78af80ecd70a163c83ed76f719e5c6aa310ba9fed6267c3bf51111","time":"2024-07-25T17:34:36Z","message":"Transactions root mismatch"}
Jul 25 17:34:36 vouch-teku-0 vouch[6572]: {"level":"error","service":"beaconblockproposer","impl":"standard","slot":9592071,"error":"transactions root mismatch","time":"2024-07-25T17:34:36Z","message":"Failed to obtain blinded proposal"}
Jul 25 17:34:36 vouch-teku-0 vouch[6572]: {"level":"warn","service":"beaconblockproposer","impl":"standard","slot":9592071,"time":"2024-07-25T17:34:36Z","message":"Failed to propose with auction; attempting to propose without auction"}

Hello,

Interesting! I think your issue is slightly different (the Titan issue was for withdrawal roots, in your case it seems to be transaction roots) ; I'm not sure what could be the issue. I haven't seen it via ultrasound or other relayers so far (we don't use vouch though), it may be related to something specific to your setup.

[gd-0]

Hey all,
Just wanted to give an update from Titan Relay regarding this issue.

On July 24, Kiln alerted us to this issue. We discovered our relay was improperly handling Optimistic V2 bids, specifically due to invalid withdrawals roots. This led to validators being forced to use the locally produced block after the consensus client rejected their invalid header for a subset of blocks.

We deployed a fix on July 25 at 10:30 UTC that ensures proper validation. Additional technical details can be found here: gattaca-com/helix#29

We've been monitoring the situation closely with Kiln's assistance, and have not detected any further issues since the fix was implemented. We’ll shortly share a detailed post-mortem with additional analysis!

[metachris]

Thanks for the update. Would appreciate you linking the post mortem here too 🙏

[gd-0]

Here's a more detailed incident report: https://titanbuilder.substack.com/p/titan-relay-incident-report-withdrawals