Missing Block

[Piradoxlanieve]

🐞 Bug Report

Description

After waiting more than 276 days to propose a block, the day came and I find that it is missing.
I have all the correct attestations, was it a mevboost failure?

🔥 Error

validador service

May 19 20:05:12 nodeeth prysm.sh[183923]: time="2023-05-19 20:05:12" level=info msg="Attestation schedule" attesterDutiesAtSlot=1 prefix=validator pubKeys=[0x8da251b37c20] slot =6474650 slotInEpoch=26 timeTillDuty=5m11s totalAttestersInEpoch=1
May 19 20:05:12 nodeeth prysm.sh[183923]: time="2023-05-19 20:05:12" level=info msg="Submitted builder validator registration settings for custom builders" prefix=validator
May 19 20:09:28 nodeeth prysm.sh[183923]: time="2023-05-19 20:09:28" level=error msg="Failed to propose block" blockSlot=6474645 error="rpc error: code = Unknown desc = could not submit blinded block: error posting the SignedBlindedBeaconBlockCapella to the builder api: unsupported error code: 502: did not receive 200 response from API" prefix=validator pubKey=0x8da251b37c20
May 19 20:10:27 nodeeth prysm.sh[183923]: time="2023-05-19 20:10:27" level=info msg="Submitted new attestations" AggregatorIndices=[] AttesterIndices=[25698] BeaconBlockRoot= 0xca01c02b3226 CommitteeIndex=51 Slot=6474650 SourceEpoch=202331 SourceRoot=0xc9a5331d4df7 TargetEpoch=202332 TargetRoot=0x26dc702dd6a9 prefix=validator
May 19 20:11:23 nodoeth prysm.sh[183923]: time="2023-05-19 20:11:23" level=info msg="Previous epoch voting summary" correctlyVotedHead=true correctlyVotedSource=true correctlyVotedTarget=true epoch =202331 inactivityScore=0 newBalance=32.008798377 oldBalance=32.008785952 percentChange="0.00004%" percentChangeSinceStart="0.02602%" prefix=validator pubKey=0x8da251b37c20 startBalance=32.00047 2946
May 19 20:11:23 nodoeth prysm.sh[183923]: time="2023-05-19 20:11:23" level=info msg="Previous epoch aggregated voting summary" attestationInclusionPct="100%" averageInactivityScore=0 correctlyVotedHeadPct="100%" correctlyVotedSourcePct="100%" correctlyVotedTargetPct="100%" epoch=202331 prefix=validator
May 19 20:11:23 nodoeth prysm.sh[183923]: time="2023-05-19 20:11:23" level=info msg="Vote summary since launch" attestationsInclusionPct="100%" correctlyVotedHeadPct="98 %" correctlyVotedSourcePct="100%" correctlyVotedTargetPct="100%" numberOfEpochs=1693 pctChangeCombinedBalance="0.02598%" prefix=validator
May 19 20:11:36 nodeeth prysm.sh[183923]: time="2023-05-19 20:11:36" level=info msg="Attestation schedule" attesterDutiesAtSlot=1 prefix=validator pubKeys=[0x8da251b37c20] slot =6474684 slotInEpoch=28 timeTillDuty=5m35s totalAttestersInEpoch=1
May 19 20:11:36 nodeeth prysm.sh[183923]: time="2023-05-19 20:11:36" level=info msg="Submitted builder validator registration settings for custom builders" prefix=validator

Mevboost Service.

19 20:05:12 nodoeth mev-boost[183917]: time="2023-05-19T20:05:12+02:00" level=info msg="http: POST /eth/v1/builder/validators 200" duration=0.047472 method=POST path=/eth/v1/builder/validators status=200 version=v1.5.0
may 19 20:05:58 nodoeth mev-boost[183917]: time="2023-05-19T20:05:58+02:00" level=info msg="http: GET /eth/v1/builder/status 200" duration=0.321673 method=GET path=/eth/v1/builder/status status=200 version=v1.5.0
may 19 20:06:58 nodoeth mev-boost[183917]: time="2023-05-19T20:06:58+02:00" level=info msg="http: GET /eth/v1/builder/status 200" duration=0.324014 method=GET path=/eth/v1/builder/status status=200 version=v1.5.0
may 19 20:07:58 nodoeth mev-boost[183917]: time="2023-05-19T20:07:58+02:00" level=info msg="http: GET /eth/v1/builder/status 200" duration=0.321195 method=GET path=/eth/v1/builder/status status=200 version=v1.5.0
may 19 20:08:58 nodoeth mev-boost[183917]: time="2023-05-19T20:08:58+02:00" level=info msg="http: GET /eth/v1/builder/status 200" duration=0.321533 method=GET path=/eth/v1/builder/status status=200 version=v1.5.0
may 19 20:09:25 nodoeth mev-boost[183917]: time="2023-05-19T20:09:25+02:00" level=warning msg="error making request to relay" error="HTTP error response: 400 / {\"code\":400,\"message\":\"sent too late - 2945 ms into slot\"}\n" method=getHeader parentHash=0x35a8730ba8cdc74d34196b31b8c365aeca3f0f6db3947728a645f24525e0dcfc pubkey=0x8da251b37c2080d8dbe2c80c565feb5b3ec18236223141a7fa3cd74415deb23f4287f91abd2cab20ac6030cd05a07375 slot=6474645 url="https://relay.edennetwork.io/eth/v1/builder/header/6474645/0x35a8730ba8cdc74d34196b31b8c365aeca3f0f6db3947728a645f24525e0dcfc/0x8da251b37c2080d8dbe2c80c565feb5b3ec18236223141a7fa3cd74415deb23f4287f91abd2cab20ac6030cd05a07375" version=v1.5.0
may 19 20:09:26 nodoeth mev-boost[183917]: time="2023-05-19T20:09:26+02:00" level=warning msg="error making request to relay" error="Get \"https://mainnet-relay.securerpc.com/eth/v1/builder/header/6474645/0x35a8730ba8cdc74d34196b31b8c365aeca3f0f6db3947728a645f24525e0dcfc/0x8da251b37c2080d8dbe2c80c565feb5b3ec18236223141a7fa3cd74415deb23f4287f91abd2cab20ac6030cd05a07375\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" method=getHeader parentHash=0x35a8730ba8cdc74d34196b31b8c365aeca3f0f6db3947728a645f24525e0dcfc pubkey=0x8da251b37c2080d8dbe2c80c565feb5b3ec18236223141a7fa3cd74415deb23f4287f91abd2cab20ac6030cd05a07375 slot=6474645 url="https://mainnet-relay.securerpc.com/eth/v1/builder/header/6474645/0x35a8730ba8cdc74d34196b31b8c365aeca3f0f6db3947728a645f24525e0dcfc/0x8da251b37c2080d8dbe2c80c565feb5b3ec18236223141a7fa3cd74415deb23f4287f91abd2cab20ac6030cd05a07375" version=v1.5.0
may 19 20:09:26 nodoeth mev-boost[183917]: time="2023-05-19T20:09:26+02:00" level=info msg="best bid" blockHash=0x34f783bb85eafcf75feb07f31ccae7e799f22d949c1531aa7d8b6b9cd22ffc69 blockNumber=17295218 method=getHeader parentHash=0x35a8730ba8cdc74d34196b31b8c365aeca3f0f6db3947728a645f24525e0dcfc pubkey=0x8da251b37c2080d8dbe2c80c565feb5b3ec18236223141a7fa3cd74415deb23f4287f91abd2cab20ac6030cd05a07375 relays="https://0xac6e77dfe25ecd6110b8e780608cce0dab71fdd5ebea22a16c0205200f2f8e2e3ad3b71d3499c54ad14d6c21b41a37ae@boost-relay.flashbots.net" slot=6474645 txRoot=0xbe88105972e6b5049f3f7b161a7e031291e25ec5e45ca55824d055a759919a15 value=0.173702278839660800 version=v1.5.0
may 19 20:09:26 nodoeth mev-boost[183917]: time="2023-05-19T20:09:26+02:00" level=info msg="http: GET /eth/v1/builder/header/6474645/0x35a8730ba8cdc74d34196b31b8c365aeca3f0f6db3947728a645f24525e0dcfc/0x8da251b37c2080d8dbe2c80c565feb5b3ec18236223141a7fa3cd74415deb23f4287f91abd2cab20ac6030cd05a07375 200" duration=0.953862 method=GET path=/eth/v1/builder/header/6474645/0x35a8730ba8cdc74d34196b31b8c365aeca3f0f6db3947728a645f24525e0dcfc/0x8da251b37c2080d8dbe2c80c565feb5b3ec18236223141a7fa3cd74415deb23f4287f91abd2cab20ac6030cd05a07375 status=200 version=v1.5.0
may 19 20:09:27 nodoeth mev-boost[183917]: time="2023-05-19T20:09:27+02:00" level=warning msg="error making request to relay, retrying" blockHash=0x34f783bb85eafcf75feb07f31ccae7e799f22d949c1531aa7d8b6b9cd22ffc69 error="HTTP error response: 400 / {\"code\":400,\"message\":\"sent too late - 4220 ms into slot\"}\n" method=getPayload parentHash=0x35a8730ba8cdc74d34196b31b8c365aeca3f0f6db3947728a645f24525e0dcfc slot=6474645 url="https://boost-relay.flashbots.net/eth/v1/builder/blinded_blocks" version=v1.5.0
may 19 20:09:27 nodoeth mev-boost[183917]: time="2023-05-19T20:09:27+02:00" level=warning msg="error making request to relay, retrying" blockHash=0x34f783bb85eafcf75feb07f31ccae7e799f22d949c1531aa7d8b6b9cd22ffc69 error="HTTP error response: 400 / {\"code\":400,\"message\":\"sent too late - 4605 ms into slot\"}\n" method=getPayload parentHash=0x35a8730ba8cdc74d34196b31b8c365aeca3f0f6db3947728a645f24525e0dcfc slot=6474645 url="https://boost-relay.flashbots.net/eth/v1/builder/blinded_blocks" version=v1.5.0
may 19 20:09:28 nodoeth mev-boost[183917]: time="2023-05-19T20:09:28+02:00" level=warning msg="error making request to relay, retrying" blockHash=0x34f783bb85eafcf75feb07f31ccae7e799f22d949c1531aa7d8b6b9cd22ffc69 error="HTTP error response: 400 / {\"code\":400,\"message\":\"sent too late - 4968 ms into slot\"}\n" method=getPayload parentHash=0x35a8730ba8cdc74d34196b31b8c365aeca3f0f6db3947728a645f24525e0dcfc slot=6474645 url="https://boost-relay.flashbots.net/eth/v1/builder/blinded_blocks" version=v1.5.0
may 19 20:09:28 nodoeth mev-boost[183917]: time="2023-05-19T20:09:28+02:00" level=warning msg="error making request to relay, retrying" blockHash=0x34f783bb85eafcf75feb07f31ccae7e799f22d949c1531aa7d8b6b9cd22ffc69 error="HTTP error response: 400 / {\"code\":400,\"message\":\"sent too late - 5356 ms into slot\"}\n" method=getPayload parentHash=0x35a8730ba8cdc74d34196b31b8c365aeca3f0f6db3947728a645f24525e0dcfc slot=6474645 url="https://boost-relay.flashbots.net/eth/v1/builder/blinded_blocks" version=v1.5.0
may 19 20:09:28 nodoeth mev-boost[183917]: time="2023-05-19T20:09:28+02:00" level=warning msg="error making request to relay, retrying" blockHash=0x34f783bb85eafcf75feb07f31ccae7e799f22d949c1531aa7d8b6b9cd22ffc69 error="HTTP error response: 400 / {\"code\":400,\"message\":\"sent too late - 5740 ms into slot\"}\n" method=getPayload parentHash=0x35a8730ba8cdc74d34196b31b8c365aeca3f0f6db3947728a645f24525e0dcfc slot=6474645 url="https://boost-relay.flashbots.net/eth/v1/builder/blinded_blocks" version=v1.5.0
may 19 20:09:28 nodoeth mev-boost[183917]: time="2023-05-19T20:09:28+02:00" level=error msg="error making request to relay" blockHash=0x34f783bb85eafcf75feb07f31ccae7e799f22d949c1531aa7d8b6b9cd22ffc69 error="max retries exceeded" method=getPayload parentHash=0x35a8730ba8cdc74d34196b31b8c365aeca3f0f6db3947728a645f24525e0dcfc slot=6474645 url="https://boost-relay.flashbots.net/eth/v1/builder/blinded_blocks" version=v1.5.0
may 19 20:09:28 nodoeth mev-boost[183917]: time="2023-05-19T20:09:28+02:00" level=error msg="no payload received from relay!" blockHash=0x34f783bb85eafcf75feb07f31ccae7e799f22d949c1531aa7d8b6b9cd22ffc69 method=getPayload parentHash=0x35a8730ba8cdc74d34196b31b8c365aeca3f0f6db3947728a645f24525e0dcfc relays="https://0xac6e77dfe25ecd6110b8e780608cce0dab71fdd5ebea22a16c0205200f2f8e2e3ad3b71d3499c54ad14d6c21b41a37ae@boost-relay.flashbots.net" slot=6474645 version=v1.5.0
may 19 20:09:28 nodoeth mev-boost[183917]: time="2023-05-19T20:09:28+02:00" level=info msg="http: POST /eth/v1/builder/blinded_blocks 502" duration=1.986169 method=POST path=/eth/v1/builder/blinded_blocks status=502 version=v1.5.0

Beacon-chain

may 19 20:09:12 nodoeth prysm.sh[183977]: time="2023-05-19 20:09:12" level=info msg="Synced new block" block=0x053a6c04... epoch=202332 finalizedEpoch=202330 finalizedRoot=0x1867aa22... prefix=blockchain slot=6474644
may 19 20:09:12 nodoeth prysm.sh[183977]: time="2023-05-19 20:09:12" level=info msg="Finished applying state transition" attestations=68 payloadHash=0x35a8730ba8cd prefix=blockchain slot=6474644 syncBitsCount=506 txCount=153
may 19 20:09:25 nodoeth prysm.sh[183977]: time="2023-05-19 20:09:25" level=warning msg="removing invalid exit from pool" error="non-active validator cannot exit"
may 19 20:09:25 nodoeth prysm.sh[183977]: time="2023-05-19 20:09:25" level=warning msg="removing invalid exit from pool" error="non-active validator cannot exit"
may 19 20:09:25 nodoeth prysm.sh[183977]: time="2023-05-19 20:09:25" level=warning msg="removing invalid exit from pool" error="non-active validator cannot exit"
may 19 20:09:25 nodoeth prysm.sh[183977]: time="2023-05-19 20:09:25" level=warning msg="removing invalid exit from pool" error="non-active validator cannot exit"
may 19 20:09:25 nodoeth prysm.sh[183977]: time="2023-05-19 20:09:25" level=warning msg="removing invalid exit from pool" error="non-active validator cannot exit"
may 19 20:09:25 nodoeth prysm.sh[183977]: time="2023-05-19 20:09:25" level=warning msg="removing invalid exit from pool" error="non-active validator cannot exit"
may 19 20:09:25 nodoeth prysm.sh[183977]: time="2023-05-19 20:09:25" level=warning msg="removing invalid exit from pool" error="non-active validator cannot exit"
may 19 20:09:25 nodoeth prysm.sh[183977]: time="2023-05-19 20:09:25" level=warning msg="removing invalid exit from pool" error="non-active validator cannot exit"
may 19 20:09:25 nodoeth prysm.sh[183977]: time="2023-05-19 20:09:25" level=warning msg="removing invalid exit from pool" error="non-active validator cannot exit"
may 19 20:09:25 nodoeth prysm.sh[183977]: time="2023-05-19 20:09:25" level=warning msg="removing invalid exit from pool" error="non-active validator cannot exit"
may 19 20:09:25 nodoeth prysm.sh[183977]: time="2023-05-19 20:09:25" level=warning msg="removing invalid exit from pool" error="non-active validator cannot exit"
may 19 20:09:25 nodoeth prysm.sh[183977]: time="2023-05-19 20:09:25" level=warning msg="removing invalid exit from pool" error="non-active validator cannot exit"
may 19 20:09:25 nodoeth prysm.sh[183977]: time="2023-05-19 20:09:25" level=warning msg="removing invalid exit from pool" error="non-active validator cannot exit"
may 19 20:09:25 nodoeth prysm.sh[183977]: time="2023-05-19 20:09:25" level=warning msg="removing invalid exit from pool" error="non-active validator cannot exit"
may 19 20:09:25 nodoeth prysm.sh[183977]: time="2023-05-19 20:09:25" level=warning msg="removing invalid exit from pool" error="non-active validator cannot exit"
may 19 20:09:25 nodoeth prysm.sh[183977]: time="2023-05-19 20:09:25" level=warning msg="removing invalid exit from pool" error="non-active validator cannot exit"
may 19 20:09:25 nodoeth prysm.sh[183977]: time="2023-05-19 20:09:25" level=warning msg="removing invalid exit from pool" error="non-active validator cannot exit"
may 19 20:09:26 nodoeth prysm.sh[183977]: time="2023-05-19 20:09:26" level=info msg="Received header with bid" blockHash=0x34f783bb85eafcf75feb07f31ccae7e799f22d949c1531aa7d8b6b9cd22ffc69 builderPubKey=0xac6e77dfe25ecd6110b8e780608cce0dab71fdd5ebea22a16c0205200f2f8e2e3ad3b71d3499c54ad14d6c21b41a37ae prefix="rpc/validator" value=173702278839660800
may 19 20:09:26 nodoeth prysm.sh[183977]: time="2023-05-19 20:09:26" level=warning msg="removing invalid BLSToExecutionChange from pool" error="withdrawal credential prefix is not a BLS prefix"
may 19 20:09:26 nodoeth prysm.sh[183977]: time="2023-05-19 20:09:26" level=warning msg="removing invalid BLSToExecutionChange from pool" error="withdrawal credential prefix is not a BLS prefix"
may 19 20:09:26 nodoeth prysm.sh[183977]: time="2023-05-19 20:09:26" level=warning msg="removing invalid BLSToExecutionChange from pool" error="withdrawal credential prefix is not a BLS prefix"
may 19 20:09:37 nodoeth prysm.sh[183977]: time="2023-05-19 20:09:37" level=info msg="Synced new block" block=0x30aa6424... epoch=202332 finalizedEpoch=202330 finalizedRoot=0x1867aa22... prefix=blockchain slot=6474646

Geth Logs

ay 19 20:08:00 nodoeth geth[184240]: INFO [05-19|20:08:00.818] Imported new potential chain segment     number=17,295,211 hash=06da67..41b24c blocks=1 txs=142  mgas=12.252  elapsed=88.146ms     mgasps=139.000 dirty=2.60GiB
may 19 20:08:00 nodoeth geth[184240]: INFO [05-19|20:08:00.953] Chain head was updated                   number=17,295,211 hash=06da67..41b24c root=bcc6c7..d83c6d elapsed=2.795264ms
may 19 20:08:01 nodoeth geth[184240]: WARN [05-19|20:08:01.370] Peer delivering stale transactions       peer=fff87666c0eab105ca83f3acc223a73b19a6f87398e4305448e8767f5c3c24cf rejected=35
may 19 20:08:06 nodoeth geth[184240]: WARN [05-19|20:08:06.947] Peer delivering stale transactions       peer=fff87666c0eab105ca83f3acc223a73b19a6f87398e4305448e8767f5c3c24cf rejected=51
may 19 20:08:13 nodoeth geth[184240]: INFO [05-19|20:08:13.120] Imported new potential chain segment     number=17,295,212 hash=a91faa..ed9db9 blocks=1 txs=162  mgas=15.697  elapsed=157.675ms    mgasps=99.555  dirty=2.60GiB
may 19 20:08:13 nodoeth geth[184240]: INFO [05-19|20:08:13.305] Chain head was updated                   number=17,295,212 hash=a91faa..ed9db9 root=7ae7bb..b82d71 elapsed=4.131935ms
may 19 20:08:15 nodoeth geth[184240]: WARN [05-19|20:08:15.371] Peer delivering stale transactions       peer=b42e4d4e6f87b2c562080ac7bcf1f2bce7b2767c469933a0e08f58d8feee1752 rejected=128
may 19 20:08:15 nodoeth geth[184240]: WARN [05-19|20:08:15.579] Peer delivering stale transactions       peer=b42e4d4e6f87b2c562080ac7bcf1f2bce7b2767c469933a0e08f58d8feee1752 rejected=48
may 19 20:08:16 nodoeth geth[184240]: WARN [05-19|20:08:16.615] Peer delivering stale transactions       peer=b42e4d4e6f87b2c562080ac7bcf1f2bce7b2767c469933a0e08f58d8feee1752 rejected=128
may 19 20:08:16 nodoeth geth[184240]: WARN [05-19|20:08:16.846] Peer delivering stale transactions       peer=b42e4d4e6f87b2c562080ac7bcf1f2bce7b2767c469933a0e08f58d8feee1752 rejected=127
may 19 20:08:17 nodoeth geth[184240]: WARN [05-19|20:08:17.414] Peer delivering stale transactions       peer=b42e4d4e6f87b2c562080ac7bcf1f2bce7b2767c469933a0e08f58d8feee1752 rejected=127
may 19 20:08:17 nodoeth geth[184240]: WARN [05-19|20:08:17.649] Peer delivering stale transactions       peer=b42e4d4e6f87b2c562080ac7bcf1f2bce7b2767c469933a0e08f58d8feee1752 rejected=128
may 19 20:08:18 nodoeth geth[184240]: WARN [05-19|20:08:18.059] Peer delivering stale transactions       peer=b42e4d4e6f87b2c562080ac7bcf1f2bce7b2767c469933a0e08f58d8feee1752 rejected=128
may 19 20:08:18 nodoeth geth[184240]: WARN [05-19|20:08:18.288] Peer delivering stale transactions       peer=b42e4d4e6f87b2c562080ac7bcf1f2bce7b2767c469933a0e08f58d8feee1752 rejected=111
may 19 20:08:24 nodoeth geth[184240]: INFO [05-19|20:08:24.355] Imported new potential chain segment     number=17,295,213 hash=a72db9..63dae3 blocks=1 txs=173  mgas=17.233  elapsed=117.574ms    mgasps=146.568 dirty=2.60GiB
may 19 20:08:24 nodoeth geth[184240]: INFO [05-19|20:08:24.512] Chain head was updated                   number=17,295,213 hash=a72db9..63dae3 root=875000..ba6abb elapsed=3.669564ms
may 19 20:08:35 nodoeth geth[184240]: WARN [05-19|20:08:35.777] Peer delivering stale transactions       peer=b42e4d4e6f87b2c562080ac7bcf1f2bce7b2767c469933a0e08f58d8feee1752 rejected=105
may 19 20:08:36 nodoeth geth[184240]: INFO [05-19|20:08:36.981] Imported new potential chain segment     number=17,295,214 hash=fa7c86..7c6557 blocks=1 txs=131  mgas=9.616   elapsed=67.453ms     mgasps=142.556 dirty=2.60GiB
may 19 20:08:37 nodoeth geth[184240]: INFO [05-19|20:08:37.122] Chain head was updated                   number=17,295,214 hash=fa7c86..7c6557 root=bb9677..340b75 elapsed=2.526411ms
may 19 20:08:40 nodoeth geth[184240]: WARN [05-19|20:08:40.664] Peer delivering stale transactions       peer=fff87666c0eab105ca83f3acc223a73b19a6f87398e4305448e8767f5c3c24cf rejected=33
may 19 20:08:40 nodoeth geth[184240]: WARN [05-19|20:08:40.958] Peer delivering stale transactions       peer=b42e4d4e6f87b2c562080ac7bcf1f2bce7b2767c469933a0e08f58d8feee1752 rejected=58
may 19 20:08:44 nodoeth geth[184240]: WARN [05-19|20:08:44.226] Peer delivering stale transactions       peer=b42e4d4e6f87b2c562080ac7bcf1f2bce7b2767c469933a0e08f58d8feee1752 rejected=45
may 19 20:08:45 nodoeth geth[184240]: WARN [05-19|20:08:45.117] Peer delivering stale transactions       peer=b42e4d4e6f87b2c562080ac7bcf1f2bce7b2767c469933a0e08f58d8feee1752 rejected=45
may 19 20:08:49 nodoeth geth[184240]: WARN [05-19|20:08:49.035] Peer delivering stale transactions       peer=b42e4d4e6f87b2c562080ac7bcf1f2bce7b2767c469933a0e08f58d8feee1752 rejected=70
may 19 20:08:49 nodoeth geth[184240]: INFO [05-19|20:08:49.059] Imported new potential chain segment     number=17,295,215 hash=26ad31..008f87 blocks=1 txs=153  mgas=18.399  elapsed=119.311ms    mgasps=154.209 dirty=2.60GiB
may 19 20:08:49 nodoeth geth[184240]: INFO [05-19|20:08:49.223] Chain head was updated                   number=17,295,215 hash=26ad31..008f87 root=842d6a..d6f009 elapsed=3.416046ms
may 19 20:08:51 nodoeth geth[184240]: WARN [05-19|20:08:51.869] Peer delivering stale transactions       peer=b42e4d4e6f87b2c562080ac7bcf1f2bce7b2767c469933a0e08f58d8feee1752 rejected=43
may 19 20:08:56 nodoeth geth[184240]: WARN [05-19|20:08:56.847] Peer delivering stale transactions       peer=a8dad3ba5faa65cf71c92c36bb0e1af76e23fe979cb4f80e8112ae00995737d3 rejected=128
may 19 20:08:57 nodoeth geth[184240]: WARN [05-19|20:08:57.067] Peer delivering stale transactions       peer=a8dad3ba5faa65cf71c92c36bb0e1af76e23fe979cb4f80e8112ae00995737d3 rejected=110
may 19 20:09:00 nodoeth geth[184240]: INFO [05-19|20:09:00.626] Imported new potential chain segment     number=17,295,216 hash=086feb..0100a0 blocks=1 txs=130  mgas=12.333  elapsed=82.381ms     mgasps=149.705 dirty=2.60GiB
may 19 20:09:00 nodoeth geth[184240]: INFO [05-19|20:09:00.777] Chain head was updated                   number=17,295,216 hash=086feb..0100a0 root=a9eea5..805e2f elapsed=3.146578ms
may 19 20:09:03 nodoeth geth[184240]: WARN [05-19|20:09:03.535] Peer delivering stale transactions       peer=fff87666c0eab105ca83f3acc223a73b19a6f87398e4305448e8767f5c3c24cf rejected=44
may 19 20:09:12 nodoeth geth[184240]: INFO [05-19|20:09:12.101] Imported new potential chain segment     number=17,295,217 hash=35a873..e0dcfc blocks=1 txs=153  mgas=13.292  elapsed=83.895ms     mgasps=158.436 dirty=2.60GiB
may 19 20:09:12 nodoeth geth[184240]: INFO [05-19|20:09:12.253] Chain head was updated                   number=17,295,217 hash=35a873..e0dcfc root=f64df6..1d1592 elapsed=2.437351ms
may 19 20:09:12 nodoeth geth[184240]: INFO [05-19|20:09:12.339] Starting work on payload                 id=0xc6b040f54523959e
may 19 20:09:12 nodoeth geth[184240]: INFO [05-19|20:09:12.386] Updated payload                          id=0xc6b040f54523959e number=17,295,218 hash=1c7f03..69c5f9 txs=22   gas=1,768,633 fees=0.002011825214 root=3ccb2d..9f4a0f elapsed=42.192ms
may 19 20:09:14 nodoeth geth[184240]: INFO [05-19|20:09:14.452] Updated payload                          id=0xc6b040f54523959e number=17,295,218 hash=df2b13..3937d1 txs=45   gas=3,735,657 fees=0.009484780709 root=7884e3..7d78be elapsed=64.842ms
may 19 20:09:16 nodoeth geth[184240]: INFO [05-19|20:09:16.507] Updated payload                          id=0xc6b040f54523959e number=17,295,218 hash=a2887b..4daf09 txs=65   gas=5,643,559 fees=0.03005591156  root=e82e7a..973a1f elapsed=54.525ms
may 19 20:09:18 nodoeth geth[184240]: INFO [05-19|20:09:18.570] Updated payload                          id=0xc6b040f54523959e number=17,295,218 hash=52ec64..7bda89 txs=90   gas=10,642,198 fees=0.03459753653  root=8b2b8d..2d0934 elapsed=62.731ms
may 19 20:09:20 nodoeth geth[184240]: INFO [05-19|20:09:20.648] Updated payload                          id=0xc6b040f54523959e number=17,295,218 hash=3c8722..85fc23 txs=109  gas=13,798,039 fees=0.0366603927   root=6115dd..e9538c elapsed=77.923ms
may 19 20:09:22 nodoeth geth[184240]: INFO [05-19|20:09:22.741] Updated payload                          id=0xc6b040f54523959e number=17,295,218 hash=880d41..748857 txs=129  gas=14,818,694 fees=0.03748302136  root=dfde94..8be998 elapsed=93.057ms
may 19 20:09:23 nodoeth geth[184240]: WARN [05-19|20:09:23.400] Peer delivering stale transactions       peer=b42e4d4e6f87b2c562080ac7bcf1f2bce7b2767c469933a0e08f58d8feee1752 rejected=43
may 19 20:09:24 nodoeth geth[184240]: INFO [05-19|20:09:24.345] Stopping work on payload                 id=0xc6b040f54523959e reason=timeout
may 19 20:09:36 nodoeth geth[184240]: INFO [05-19|20:09:36.726] Imported new potential chain segment     number=17,295,218 hash=852a51..593f4c blocks=1 txs=287  mgas=29.426  elapsed=182.096ms    mgasps=161.596 dirty=2.60GiB
may 19 20:09:36 nodoeth geth[184240]: INFO [05-19|20:09:36.895] Chain head was updated                   number=17,295,218 hash=852a51..593f4c root=c02f63..c23844 elapsed=5.708923ms
may 19 20:09:43 nodoeth geth[184240]: WARN [05-19|20:09:43.334] Peer delivering stale transactions       peer=b42e4d4e6f87b2c562080ac7bcf1f2bce7b2767c469933a0e08f58d8feee1752 rejected=39
may 19 20:09:45 nodoeth geth[184240]: WARN [05-19|20:09:45.774] Peer delivering stale transactions       peer=b42e4d4e6f87b2c562080ac7bcf1f2bce7b2767c469933a0e08f58d8feee1752 rejected=33
may 19 20:09:46 nodoeth geth[184240]: WARN [05-19|20:09:46.635] Peer delivering stale transactions       peer=b42e4d4e6f87b2c562080ac7bcf1f2bce7b2767c469933a0e08f58d8feee1752 rejected=33
may 19 20:09:49 nodoeth geth[184240]: INFO [05-19|20:09:49.742] Imported new potential chain segment     number=17,295,219 hash=1f7999..b80884 blocks=1 txs=123  mgas=13.443  elapsed=153.266ms    mgasps=87.710  dirty=2.60GiB
may 19 20:09:49 nodoeth geth[184240]: INFO [05-19|20:09:49.899] Chain head was updated                   number=17,295,219 hash=1f7999..b80884 root=df260b..74a0b0 elapsed=4.799834ms
may 19 20:09:51 nodoeth geth[184240]: WARN [05-19|20:09:51.741] Peer delivering stale transactions       peer=b42e4d4e6f87b2c562080ac7bcf1f2bce7b2767c469933a0e08f58d8feee1752 rejected=45
may 19 20:10:00 nodoeth geth[184240]: INFO [05-19|20:10:00.562] Imported new potential chain segment     number=17,295,220 hash=c6bdae..231964 blocks=1 txs=113  mgas=10.781  elapsed=90.970ms     mgasps=118.511 dirty=2.60GiB
may 19 20:10:00 nodoeth geth[184240]: INFO [05-19|20:10:00.699] Chain head was updated                   number=17,295,220 hash=c6bdae..231964 root=eb1916..89b8e5 elapsed=2.255168ms
may 19 20:10:11 nodoeth geth[184240]: INFO [05-19|20:10:11.951] Imported new potential chain segment     number=17,295,221 hash=19f668..d87835 blocks=1 txs=136  mgas=10.389  elapsed=78.398ms     mgasps=132.516 dirty=2.60GiB
may 19 20:10:12 nodoeth geth[184240]: INFO [05-19|20:10:12.097] Chain head was updated                   number=17,295,221 hash=19f668..d87835 root=05b2b3..13fe5b elapsed=2.803758ms
may 19 20:10:25 nodoeth geth[184240]: INFO [05-19|20:10:25.191] Imported new potential chain segment     number=17,295,222 hash=a37666..ba741a blocks=1 txs=137  mgas=10.190  elapsed=75.355ms     mgasps=135.232 dirty=2.60GiB
may 19 20:10:25 nodoeth geth[184240]: INFO [05-19|20:10:25.355] Chain head was updated                   number=17,295,222 hash=a37666..ba741a root=04e1c6..dbaf56 elapsed=2.41794ms
may 19 20:10:37 nodoeth geth[184240]: INFO [05-19|20:10:37.078] Imported new potential chain segment     number=17,295,223 hash=97c56e..b948af blocks=1 txs=142  mgas=17.209  elapsed=168.855ms    mgasps=101.914 dirty=2.60GiB
may 19 20:10:37 nodoeth geth[184240]: INFO [05-19|20:10:37.211] Chain head was updated                   number=17,295,223 hash=97c56e..b948af root=064498..e5b67c elapsed=3.142928ms
may 19 20:10:49 nodoeth geth[184240]: INFO [05-19|20:10:49.599] Imported new potential chain segment     number=17,295,224 hash=dc5f60..71003d blocks=1 txs=129  mgas=13.362  elapsed=109.913ms    mgasps=121.572 dirty=2.60GiB
may 19 20:10:49 nodoeth geth[184240]: INFO [05-19|20:10:49.741] Chain head was updated                   number=17,295,224 hash=dc5f60..71003d root=40555c..4775ad elapsed=10.148663ms
may 19 20:11:00 nodoeth geth[184240]: INFO [05-19|20:11:00.918] Imported new potential chain segment     number=17,295,225 hash=9e649b..7dd770 blocks=1 txs=160  mgas=21.812  elapsed=234.352ms    mgasps=93.073  dirty=2.60GiB
may 19 20:11:01 nodoeth geth[184240]: INFO [05-19|20:11:01.083] Chain head was updated                   number=17,295,225 hash=9e649b..7dd770 root=60fa8d..8e0a4c elapsed=3.152425ms
may 19 20:11:12 nodoeth geth[184240]: INFO [05-19|20:11:12.969] Imported new potential chain segment     number=17,295,226 hash=29d0c3..78dc37 blocks=1 txs=211  mgas=24.844  elapsed=151.867ms    mgasps=163.587 dirty=2.60GiB
may 19 20:11:13 nodoeth geth[184240]: INFO [05-19|20:11:13.089] Chain head was updated                   number=17,295,226 hash=29d0c3..78dc37 root=fb19bf..5036ed elapsed=4.204245ms
may 19 20:11:25 nodoeth geth[184240]: INFO [05-19|20:11:25.121] Imported new potential chain segment     number=17,295,227 hash=d10ab3..eff218 blocks=1 txs=166  mgas=20.587  elapsed=263.161ms    mgasps=78.229  dirty=2.60GiB
may 19 20:11:25 nodoeth geth[184240]: INFO [05-19|20:11:25.298] Chain head was updated                   number=17,295,227 hash=d10ab3..eff218 root=b9ff95..ad39fb elapsed=4.512699ms
may 19 20:11:37 nodoeth geth[184240]: INFO [05-19|20:11:37.947] Imported new potential chain segment     number=17,295,228 hash=90a378..38a137 blocks=1 txs=132  mgas=25.603  elapsed=244.055ms    mgasps=104.905 dirty=2.60GiB
may 19 20:11:38 nodoeth geth[184240]: INFO [05-19|20:11:38.101] Chain head was updated                   number=17,295,228 hash=90a378..38a137 root=9c76e9..0539cc elapsed=3.848899ms
may 19 20:11:48 nodoeth geth[184240]: INFO [05-19|20:11:48.496] Imported new potential chain segment     number=17,295,229 hash=b3450f..aa6fe2 blocks=1 txs=110  mgas=10.408  elapsed=78.063ms     mgasps=133.329 dirty=2.60GiB
may 19 20:11:48 nodoeth geth[184240]: INFO [05-19|20:11:48.643] Chain head was updated                   number=17,295,229 hash=b3450f..aa6fe2 root=ce4869..c24c01 elapsed=2.110569ms
may 19 20:12:01 nodoeth geth[184240]: INFO [05-19|20:12:01.651] Imported new potential chain segment     number=17,295,230 hash=91aaa2..edf32c blocks=1 txs=110  mgas=8.877   elapsed=67.454ms     mgasps=131.593 dirty=2.60GiB
may 19 20:12:01 nodoeth geth[184240]: INFO [05-19|20:12:01.801] Chain head was updated                   number=17,295,230 hash=91aaa2..edf32c root=3e9810..b0c2c5 elapsed=2.113819ms
may 19 20:12:13 nodoeth geth[184240]: INFO [05-19|20:12:13.056] Imported new potential chain segment     number=17,295,231 hash=ce371d..097575 blocks=1 txs=145  mgas=15.459  elapsed=182.749ms    mgasps=84.593  dirty=2.60GiB
may 19 20:12:13 nodoeth geth[184240]: INFO [05-19|20:12:13.213] Chain head was updated                   number=17,295,231 hash=ce371d..097575 root=e45a15..6c52bb elapsed=2.291632ms
may 19 20:12:24 nodoeth geth[184240]: INFO [05-19|20:12:24.715] Imported new potential chain segment     number=17,295,232 hash=03ea28..cd1727 blocks=1 txs=114  mgas=15.788  elapsed=166.908ms    mgasps=94.593  dirty=2.60GiB
may 19 20:12:24 nodoeth geth[184240]: INFO [05-19|20:12:24.885] Chain head was updated                   number=17,295,232 hash=03ea28..cd1727 root=30c3fb..9b21d3 elapsed=2.483233ms
may 19 20:12:37 nodoeth geth[184240]: INFO [05-19|20:12:37.203] Imported new potential chain segment     number=17,295,233 hash=788e86..186b28 blocks=1 txs=130  mgas=13.098  elapsed=100.875ms    mgasps=129.841 dirty=2.60GiB
may 19 20:12:37 nodoeth geth[184240]: INFO [05-19|20:12:37.390] Chain head was updated                   number=17,295,233 hash=788e86..186b28 root=c9b254..560891 elapsed=2.717392ms
may 19 20:12:49 nodoeth geth[184240]: INFO [05-19|20:12:49.149] Imported new potential chain segment     number=17,295,234 hash=4b4465..ad5587 blocks=1 txs=158  mgas=21.244  elapsed=216.679ms    mgasps=98.043  dirty=2.60GiB
may 19 20:12:49 nodoeth geth[184240]: INFO [05-19|20:12:49.310] Chain head was updated                   number=17,295,234 hash=4b4465..ad5587 root=266ec4..bbe8b8 elapsed=3.361245ms
may 19 20:13:00 nodoeth geth[184240]: INFO [05-19|20:13:00.783] Imported new potential chain segment     number=17,295,235 hash=ceaf24..7168e6 blocks=1 txs=108  mgas=19.255  elapsed=194.476ms    mgasps=99.010  dirty=2.60GiB
may 19 20:13:00 nodoeth geth[184240]: INFO [05-19|20:13:00.937] Chain head was updated                   number=17,295,235 hash=ceaf24..7168e6 root=1f7f4d..e20229 elapsed=2.51057ms
may 19 20:13:12 nodoeth geth[184240]: INFO [05-19|20:13:12.150] Imported new potential chain segment     number=17,295,236 hash=b8f638..fb2657 blocks=1 txs=120  mgas=12.513  elapsed=87.354ms     mgasps=143.239 dirty=2.60GiB
may 19 20:13:12 nodoeth geth[184240]: INFO [05-19|20:13:12.311] Chain head was updated                   number=17,295,236 hash=b8f638..fb2657 root=1fe80d..60227c elapsed=2.785043ms
may 19 20:13:25 nodoeth geth[184240]: INFO [05-19|20:13:25.069] Imported new potential chain segment     number=17,295,237 hash=0651e7..728a90 blocks=1 txs=117  mgas=17.687  elapsed=105.894ms    mgasps=167.022 dirty=2.60GiB
may 19 20:13:25 nodoeth geth[184240]: INFO [05-19|20:13:25.241] Chain head was updated                   number=17,295,237 hash=0651e7..728a90 root=fcdc9f..398b5a elapsed=4.604258ms

Mevboost config

[Unit]
Description=mev-boost ethereum mainnet
Wants=network-online.target
After=network-online.target

[Service]
User=mevboost
Group=mevboost
Type=simple
Restart=always
RestartSec=5
ExecStart=/usr/local/bin/mev-boost \
  -mainnet \
  -min-bid 0.05 \
  -relay-check \
  -relay https://0xa15b52576bcbf1072f4a011c0f99f9fb6c66f3e1ff321f11f461d15e31b1cb359caa092c71bbded0bae5b5ea401aab7e@aestus.live \
  -relay https://0xa7ab7a996c8584251c8f925da3170bdfd6ebc75d50f5ddc4050a6fdc77f2a3b5fce2cc750d0865e05d7228af97d69561@agnostic-relay.net \
  -relay https://0x9000009807ed12c1f08bf4e81c6da3ba8e3fc3d953898ce0102433094e5f22f21102ec057841fcb81978ed1ea0fa8246@builder-relay-mainnet.blocknative.com \
  -relay https://0x8b5d2e73e2a3a55c6c87b8b6eb92e0149a125c852751db1422fa951e42a09b82c142c3ea98d0d9930b056a3bc9896b8f@bloxroute.max-profit.blxrbdn.com \
  -relay https://0xb3ee7afcf27f1f1259ac1787876318c6584ee353097a50ed84f51a1f21a323b3736f271a895c7ce918c038e4265918be@relay.edennetwork.io \
  -relay https://0xac6e77dfe25ecd6110b8e780608cce0dab71fdd5ebea22a16c0205200f2f8e2e3ad3b71d3499c54ad14d6c21b41a37ae@boost-relay.flashbots.net \
  -relay https://0x98650451ba02064f7b000f5768cf0cf4d4e492317d82871bdc87ef841a0743f69f0f1eea11168503240ac35d101c9135@mainnet-relay.securerpc.com \
  -relay https://0xa1559ace749633b997cb3fdacffb890aeebdb0f5a3b6aaa7eeeaf1a38af0a8fe88b9e4b1f61f236d2e64d95733327a62@relay.ultrasound.money

  [Install]
WantedBy=multi-user.target

🌍 Your Environment

Ubuntu 20.04
Geth
Version: 1.11.6-stable
Git Commit: ea9e62ca3db5c33aa7438ebf39c189afd53c6bf8
Architecture: amd64
Go Version: go1.20.3

@nodoeth:~# prysm version
Latest Prysm version is v4.0.4.

Mev-Boost 1.5.0

[metachris]

It looks like a case of poor network connection of your validator, at least in this timeframe, but maybe even generally.

Slot 6,474,645 was scheduled to start at timestamp 1,684,519,763, which is Fri May 19 2023 18:09:23 GMT+0000

The getHeader response is only received at second 26, which is already 3 seconds into the slot. And then the getPayload request hit the relay 4220ms into the slot, which is 200ms after the cutoff time (4s).

[Piradoxlanieve]

@metachris
my speed is very stable

 Ookla Rapid Test

       Server: Virgin telco - Derio (id: 39782)
          ISP: Adamo Telecom Iberia S.A.
Idle latency: 13.86 ms (jitter: 0.08 ms, low: 13.78 ms, high: 14.06 ms)
     Download: 895.22 Mbps (data used: 1.2 GB)
                  24.44 ms (jitter: 1.78 ms, low: 14.08 ms, high: 55.29 ms)
       Upload: 537.30 Mbps (data used: 443.6 MB)
                  14.44 ms (jitter: 0.83 ms, low: 13.57 ms, high: 29.41 ms)
  Packet loss: Not available.

It seems that something similar has happened to more people from what I have seen on reddit, I have added a flag that the prysm team told me --build-block-parallel=true
and I have also removed 2 relays
HW: i5 6500T , 32Gb ram DDR4, 2TB SSD, 512 Gb Nvme and 1000/600 internet speed

[metachris]

I understand that this is frustrating. Sadly it looks like it was a one-off network issue between your proposer and the relays. Closing this issue now.

For reference, here's a PR that will improve logging in such situations: #508

[metachris]

opened an issue to discuss possible mitigations: #518

thanks for the detailed report! 🙏