[Tracking] Hive engine-cancun test suite

[mpaulucci]

These are the failing tests

make run-hive SIMULATION=ethereum/engine TEST_PATTERN="engine-cancun"
cat $(ls -t hive/workspace/logs/*-*.json | head -n1) | jq '.testCases[] | select(.summaryResult.pass == false) | .name'

• Invalid Missing Ancestor Syncing ReOrg, StateRoot, EmptyTxs=True, CanonicalReOrg=False, Invalid P9
• Invalid Missing Ancestor Syncing ReOrg, StateRoot, EmptyTxs=False, CanonicalReOrg=False, Invalid P9
• Invalid Missing Ancestor Syncing ReOrg, StateRoot, EmptyTxs=True, CanonicalReOrg=True, Invalid P9
• Invalid Missing Ancestor Syncing ReOrg, StateRoot, EmptyTxs=False, CanonicalReOrg=True, Invalid P9
• Invalid Missing Ancestor Syncing ReOrg, ReceiptsRoot, EmptyTxs=False, CanonicalReOrg=False, Invalid P8
• Invalid Missing Ancestor Syncing ReOrg, ReceiptsRoot, EmptyTxs=False, CanonicalReOrg=True, Invalid P8
• Invalid Missing Ancestor Syncing ReOrg, GasLimit, EmptyTxs=False, CanonicalReOrg=False, Invalid P8
• Invalid Missing Ancestor Syncing ReOrg, GasLimit, EmptyTxs=False, CanonicalReOrg=True, Invalid P8
• Invalid Missing Ancestor Syncing ReOrg, GasUsed, EmptyTxs=False, CanonicalReOrg=False, Invalid P8
• Invalid Missing Ancestor Syncing ReOrg, GasUsed, EmptyTxs=False, CanonicalReOrg=True, Invalid P8
• Invalid Missing Ancestor Syncing ReOrg, Timestamp, EmptyTxs=False, CanonicalReOrg=False, Invalid P8
• Invalid Missing Ancestor Syncing ReOrg, Timestamp, EmptyTxs=False, CanonicalReOrg=True, Invalid P8
• Invalid Missing Ancestor Syncing ReOrg, Incomplete Transactions, EmptyTxs=False, CanonicalReOrg=False, Invalid P9
• Invalid Missing Ancestor Syncing ReOrg, Incomplete Transactions, EmptyTxs=False, CanonicalReOrg=True, Invalid P9
• Invalid Missing Ancestor Syncing ReOrg, Transaction Signature, EmptyTxs=False, CanonicalReOrg=False, Invalid P9
• Invalid Missing Ancestor Syncing ReOrg, Transaction Signature, EmptyTxs=False, CanonicalReOrg=True, Invalid P9
• Invalid Missing Ancestor Syncing ReOrg, Transaction Nonce, EmptyTxs=False, CanonicalReOrg=False, Invalid P9
• Invalid Missing Ancestor Syncing ReOrg, Transaction Nonce, EmptyTxs=False, CanonicalReOrg=True, Invalid P9
• Invalid Missing Ancestor Syncing ReOrg, Transaction Gas, EmptyTxs=False, CanonicalReOrg=False, Invalid P9
• Invalid Missing Ancestor Syncing ReOrg, Transaction Gas, EmptyTxs=False, CanonicalReOrg=True, Invalid P9
• Invalid Missing Ancestor Syncing ReOrg, Transaction GasPrice, EmptyTxs=False, CanonicalReOrg=False, Invalid P9
• Invalid Missing Ancestor Syncing ReOrg, Transaction GasPrice, EmptyTxs=False, CanonicalReOrg=True, Invalid P9
• Invalid Missing Ancestor Syncing ReOrg, Transaction Value, EmptyTxs=False, CanonicalReOrg=False, Invalid P9
• Invalid Missing Ancestor Syncing ReOrg, Transaction Value, EmptyTxs=False, CanonicalReOrg=True, Invalid P9
• Sidechain Reorg
• Transaction Re-Org, Re-Org Out
• Transaction Re-Org, Re-Org to Different Block
• Transaction Re-Org, New Payload on Revert Back
• Re-Org Back into Canonical Chain, Depth=10, Execute Side Payload on Re-Org
• Fork ID: Genesis=0, Cancun=2, Shanghai=2
• Fork ID: Genesis=0, Cancun=2, Shanghai=2, BlocksBeforePeering=1
• Fork ID: Genesis=1, Cancun=1
• Fork ID: Genesis=1, Cancun=1, BlocksBeforePeering=1
• Fork ID: Genesis=1, Cancun=1, Shanghai=1
• Fork ID: Genesis=1, Cancun=1, Shanghai=1, BlocksBeforePeering=1
• Fork ID: Genesis=1, Cancun=2, Shanghai=1
• Fork ID: Genesis=1, Cancun=2, Shanghai=1, BlocksBeforePeering=1
• Pre-Merge Fork Number > 0
• Request Blob Pooled Transactions Single
• Request Blob Pooled Transactions Multiple
• Blob Transaction Ordering, Multiple Clients
• Invalid NewPayload, ReceiptsRoot, Syncing=False, EmptyTxs=False, DynFeeTxs=False
• Invalid NewPayload, ReceiptsRoot, Syncing=True, EmptyTxs=False, DynFeeTxs=False

[ElFantasma]

The script to obtain this list is not precise.
awk returns the line of the next test after a failing one.
Got my own list from the json resulting on the Daily Hive Coverage run
EDIT: it can be generated locally with:

make run-hive SIMULATION=ethereum/engine TEST_PATTERN="engine-cancun"
cat $(ls -t hive/workspace/logs/*-*.json | head -n1) | jq '.testCases.[] | select(.summaryResult.pass == false) | .name'

Blob Transaction Ordering, Multiple Clients (Cancun) (ethrex)
ForkchoiceUpdatedV3 Set Head to Shanghai Payload, Null Payload Attributes (Cancun) (ethrex)
ForkchoiceUpdatedV3 To Request Shanghai Payload, Null Beacon Root (Cancun) (ethrex)
ForkchoiceUpdatedV3 To Request Shanghai Payload, Non-Null Beacon Root (Cancun) (ethrex)
ForkchoiceUpdatedV2 To Request Shanghai Payload, Non-Null Beacon Root (Cancun) (ethrex)
ForkchoiceUpdatedV2 To Request Cancun Payload, Non-Null Beacon Root (Cancun) (ethrex)
ForkchoiceUpdatedV2 To Request Cancun Payload, Missing Beacon Root (Cancun) (ethrex)
GetPayloadV3 To Request Shanghai Payload (Cancun) (ethrex)
GetPayloadV2 To Request Cancun Payload (Cancun) (ethrex)
NewPayloadV3 Before Cancun, Nil Data Fields, Nil Versioned Hashes, Nil Beacon Root (Cancun) (ethrex)
NewPayloadV3 Before Cancun, Nil ExcessBlobGas, 0x00 BlobGasUsed, Nil Versioned Hashes, Nil Beacon Root (Cancun) (ethrex)
NewPayloadV3 Before Cancun, 0x00 ExcessBlobGas, Nil BlobGasUsed, Nil Versioned Hashes, Nil Beacon Root (Cancun) (ethrex)
NewPayloadV3 Before Cancun, Nil Data Fields, Empty Array Versioned Hashes, Nil Beacon Root (Cancun) (ethrex)
NewPayloadV3 Before Cancun, Nil Data Fields, Nil Versioned Hashes, Zero Beacon Root (Cancun) (ethrex)
NewPayloadV3 Before Cancun, 0x00 Data Fields, Empty Array Versioned Hashes, Zero Beacon Root (Cancun) (ethrex)
ForkchoiceUpdatedV2 then ForkchoiceUpdatedV3 Valid Payload Building Requests (Cancun) (ethrex)
Request Blob Pooled Transactions Single (Cancun) (ethrex)
Request Blob Pooled Transactions Multiple (Cancun) (ethrex)
Invalid NewPayload, ReceiptsRoot, Syncing=False, EmptyTxs=False, DynFeeTxs=False (Cancun) (ethrex)
Invalid NewPayload, ReceiptsRoot, Syncing=True, EmptyTxs=False, DynFeeTxs=False (Cancun) (ethrex)
Invalid Missing Ancestor Syncing ReOrg, StateRoot, EmptyTxs=True, CanonicalReOrg=False, Invalid P9 (Cancun) (ethrex)
Invalid Missing Ancestor Syncing ReOrg, StateRoot, EmptyTxs=False, CanonicalReOrg=False, Invalid P9 (Cancun) (ethrex)
Invalid Missing Ancestor Syncing ReOrg, StateRoot, EmptyTxs=True, CanonicalReOrg=True, Invalid P9 (Cancun) (ethrex)
Invalid Missing Ancestor Syncing ReOrg, StateRoot, EmptyTxs=False, CanonicalReOrg=True, Invalid P9 (Cancun) (ethrex)
Invalid Missing Ancestor Syncing ReOrg, ReceiptsRoot, EmptyTxs=False, CanonicalReOrg=False, Invalid P8 (Cancun) (ethrex)
Invalid Missing Ancestor Syncing ReOrg, ReceiptsRoot, EmptyTxs=False, CanonicalReOrg=True, Invalid P8 (Cancun) (ethrex)
Invalid Missing Ancestor Syncing ReOrg, GasLimit, EmptyTxs=False, CanonicalReOrg=False, Invalid P8 (Cancun) (ethrex)
Invalid Missing Ancestor Syncing ReOrg, GasLimit, EmptyTxs=False, CanonicalReOrg=True, Invalid P8 (Cancun) (ethrex)
Invalid Missing Ancestor Syncing ReOrg, GasUsed, EmptyTxs=False, CanonicalReOrg=False, Invalid P8 (Cancun) (ethrex)
Invalid Missing Ancestor Syncing ReOrg, GasUsed, EmptyTxs=False, CanonicalReOrg=True, Invalid P8 (Cancun) (ethrex)
Invalid Missing Ancestor Syncing ReOrg, Timestamp, EmptyTxs=False, CanonicalReOrg=False, Invalid P8 (Cancun) (ethrex)
Invalid Missing Ancestor Syncing ReOrg, Timestamp, EmptyTxs=False, CanonicalReOrg=True, Invalid P8 (Cancun) (ethrex)
Invalid Missing Ancestor Syncing ReOrg, Incomplete Transactions, EmptyTxs=False, CanonicalReOrg=False, Invalid P9 (Cancun) (ethrex)
Invalid Missing Ancestor Syncing ReOrg, Incomplete Transactions, EmptyTxs=False, CanonicalReOrg=True, Invalid P9 (Cancun) (ethrex)
Invalid Missing Ancestor Syncing ReOrg, Transaction Signature, EmptyTxs=False, CanonicalReOrg=False, Invalid P9 (Cancun) (ethrex)
Invalid Missing Ancestor Syncing ReOrg, Transaction Signature, EmptyTxs=False, CanonicalReOrg=True, Invalid P9 (Cancun) (ethrex)
Invalid Missing Ancestor Syncing ReOrg, Transaction Nonce, EmptyTxs=False, CanonicalReOrg=False, Invalid P9 (Cancun) (ethrex)
Invalid Missing Ancestor Syncing ReOrg, Transaction Nonce, EmptyTxs=False, CanonicalReOrg=True, Invalid P9 (Cancun) (ethrex)
Invalid Missing Ancestor Syncing ReOrg, Transaction Gas, EmptyTxs=False, CanonicalReOrg=False, Invalid P9 (Cancun) (ethrex)
Invalid Missing Ancestor Syncing ReOrg, Transaction Gas, EmptyTxs=False, CanonicalReOrg=True, Invalid P9 (Cancun) (ethrex)
Invalid Missing Ancestor Syncing ReOrg, Transaction GasPrice, EmptyTxs=False, CanonicalReOrg=False, Invalid P9 (Cancun) (ethrex)
Invalid Missing Ancestor Syncing ReOrg, Transaction GasPrice, EmptyTxs=False, CanonicalReOrg=True, Invalid P9 (Cancun) (ethrex)
Invalid Missing Ancestor Syncing ReOrg, Transaction Value, EmptyTxs=False, CanonicalReOrg=False, Invalid P9 (Cancun) (ethrex)
Invalid Missing Ancestor Syncing ReOrg, Transaction Value, EmptyTxs=False, CanonicalReOrg=True, Invalid P9 (Cancun) (ethrex)
Sidechain Reorg (Cancun) (ethrex)
Transaction Re-Org, Re-Org Out (Cancun) (ethrex)
Transaction Re-Org, Re-Org to Different Block (Cancun) (ethrex)
Transaction Re-Org, New Payload on Revert Back (Cancun) (ethrex)
Re-Org Back into Canonical Chain, Depth=10, Execute Side Payload on Re-Org (Cancun) (ethrex)
Fork ID: Genesis=0, Cancun=2 (Cancun) (ethrex)
Fork ID: Genesis=0, Cancun=2, BlocksBeforePeering=1 (Cancun) (ethrex)
Fork ID: Genesis=0, Cancun=2, Shanghai=1 (Cancun) (ethrex)
Fork ID: Genesis=0, Cancun=2, Shanghai=1, BlocksBeforePeering=1 (Cancun) (ethrex)
Fork ID: Genesis=0, Cancun=2, Shanghai=2 (Cancun) (ethrex)
Fork ID: Genesis=0, Cancun=2, Shanghai=2, BlocksBeforePeering=1 (Cancun) (ethrex)
Fork ID: Genesis=1, Cancun=1 (Cancun) (ethrex)
Fork ID: Genesis=1, Cancun=1, BlocksBeforePeering=1 (Cancun) (ethrex)
Fork ID: Genesis=1, Cancun=1, Shanghai=1 (Cancun) (ethrex)
Fork ID: Genesis=1, Cancun=1, Shanghai=1, BlocksBeforePeering=1 (Cancun) (ethrex)
Fork ID: Genesis=1, Cancun=2, Shanghai=1 (Cancun) (ethrex)
Fork ID: Genesis=1, Cancun=2, Shanghai=1, BlocksBeforePeering=1 (Cancun) (ethrex)
Pre-Merge Fork Number > 0 (Cancun) (ethrex)