Error with polkadot-v1.9.0 : Timestamp slot must match CurrentSlot

[5tan-enj]

Hi, we have upgraded our parachain to polkadot-v1.9.0 but see this error after upgrade,

runtime                      ERROR: panicked at /Users/ubuntu/.cargo/git/checkouts/polkadot-sdk-cff69157b985ed76/61801ce/substrate/frame/babe/src/lib.rs:907:9:
assertion `left == right` failed: Timestamp slot must match `CurrentSlot`
  left: Slot(0)
 right: Slot(285386249)
[12:46:04.628] WARN (block-builder): Failed to apply inherents {} Error: wasm `unreachable` instruction executed
    app: "chopsticks"
/Users/ubuntu/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/block-builder.js:174
            throw new Error('Failed to apply inherents');
                  ^

Error: Failed to apply inherents
    at initNewBlock (/Users/ubuntu/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/block-builder.js:174:19)
    at async buildBlock (/Users/ubuntu/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/block-builder.js:281:44)
    at async TxPool.buildBlock (/Users/ubuntu/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/txpool.js:340:43)
    at async TxPool.buildBlockIfNeeded (/Users/ubuntu/.npm/_npx/81ad9c881cb83600/node_modules/@acala-network/chopsticks-core/dist/cjs/blockchain/txpool.js:324:9)

The error looks like its caused by the relaychain not supporting the v1.9.0 changes, is this the case or is this a mistake on our end?

[ermalkaleci]

Steps to reproduce this please? Which parachain?

[clangenb]

I can reproduce the same error if I do a runtime upgrade from v1.7.0 to v1.10.0 on a local up-to-date clone of the kusama/bajun-parachain. Although in the case, the slot number mismatch is only one. Did maybe the execution order of the inherents change in this version bump?

[xlc]

it could be related to async backing but we need something allow us to reproduce the issue

[clangenb]

Take this PR: ajuna-network/bajun-parachain#21, build the wasm and run:

 npx @acala-network/chopsticks@latest --config ./chopsticks/bajun.yml  --wasm-override ./target/release/wbuild/bajun-runtime/bajun_runtime.compact.compressed.wasm

[xlc]

Thanks. I am able to reproduce it. Looks like we need to workaround some additional validation code introduced by async backing.

[ermalkaleci]

@clangenb have you tried latest build?

[clangenb]

jup, just rebuilt master, no change:

../chopsticks/packages/chopsticks/chopsticks.cjs --config ./chopsticks/bajun.yml --wasm-override ./target/release/wbuild/bajun-runtime/bajun_runtime.compact.compressed.wasm
[09:33:13.882] INFO: Bajun Kusama RPC listening on port 8000
    app: "chopsticks"
[09:33:54.255] INFO (block-builder): Bajun Kusama building #4,435,292
    app: "chopsticks"
    number: 4435292
    extrinsics: [
      "0x45028400d43593c715fdd31c61141abd…9cb226aa4794f26a480b00a0724e1809"
    ]
    umpCount: 0
runtime::frame-support        INFO: 🐥 New pallet "Migrations" detected in the runtime. The pallet has no defined storage version, so the on-chain version is being initialized to StorageVersion(0).
runtime::frame-support        INFO: 🐥 New pallet "TechnicalCommittee" detected in the runtime. Initializing the on-chain storage version to match the storage version defined in the pallet: StorageVersion(4)
runtime::frame-support        INFO: 🐥 New pallet "TechnicalCommitteeMembership" detected in the runtime. Initializing the on-chain storage version to match the storage version defined in the pallet: StorageVersion(4)
runtime::frame-support        INFO: 🐥 New pallet "Democracy" detected in the runtime. Initializing the on-chain storage version to match the storage version defined in the pallet: StorageVersion(1)
runtime::frame-support        INFO: 🐥 New pallet "AffiliatesAAA" detected in the runtime. Initializing the on-chain storage version to match the storage version defined in the pallet: StorageVersion(1)
runtime::frame-support        INFO: 🐥 New pallet "TournamentAAA" detected in the runtime. Initializing the on-chain storage version to match the storage version defined in the pallet: StorageVersion(1)
runtime::ajuna-awesome-avatars::migration  INFO: Updated GlobalConfig from v5 to v6
runtime::ajuna-awesome-avatars::migration  INFO: Updated 2 Season entries from v5 to v6
runtime::ajuna-awesome-avatars::migration  INFO: Upgraded storage to version StorageVersion(6)
runtime::frame-support       DEBUG: ✅ no migration for System
runtime::frame-support       DEBUG: ✅ no migration for ParachainSystem
runtime::frame-support       DEBUG: ✅ no migration for Timestamp
runtime::frame-support       DEBUG: ✅ no migration for ParachainInfo
runtime::frame-support       DEBUG: ✅ no migration for Multisig
runtime::frame-support       DEBUG: ✅ no migration for Utility
runtime::frame-support       DEBUG: ✅ no migration for Identity
runtime::frame-support       DEBUG: ✅ no migration for Proxy
runtime::frame-support       DEBUG: ✅ no migration for Scheduler
runtime::frame-support       DEBUG: ✅ no migration for Preimage
runtime::frame-support        INFO: ⚠️ Migrations declares internal migrations (which *might* execute). On-chain `StorageVersion(0)` vs in-code storage version `NoStorageVersionSet`
pallet_migrations            DEBUG: Onboarding 4 new MBM migrations
bajun_runtime                 INFO: MigrationStatusHandler started
runtime::frame-support       DEBUG: ✅ no migration for Balances
runtime::frame-support       DEBUG: ✅ no migration for TransactionPayment
runtime::frame-support       DEBUG: ✅ no migration for Vesting
runtime::frame-support       DEBUG: ✅ no migration for Authorship
runtime::frame-support       DEBUG: ✅ no migration for CollatorSelection
runtime::frame-support       DEBUG: ✅ no migration for Session
runtime::frame-support       DEBUG: ✅ no migration for Aura
runtime::frame-support       DEBUG: ✅ no migration for AuraExt
runtime::frame-support       DEBUG: ✅ no migration for XcmpQueue
runtime::frame-support       DEBUG: ✅ no migration for PolkadotXcm
runtime::frame-support       DEBUG: ✅ no migration for CumulusXcm
runtime::frame-support       DEBUG: ✅ no migration for MessageQueue
runtime::frame-support       DEBUG: ✅ no migration for Sudo
runtime::frame-support       DEBUG: ✅ no migration for Treasury
runtime::frame-support       DEBUG: ✅ no migration for Council
runtime::frame-support       DEBUG: ✅ no migration for CouncilMembership
runtime::frame-support       DEBUG: ✅ no migration for TechnicalCommittee
runtime::frame-support       DEBUG: ✅ no migration for TechnicalCommitteeMembership
runtime::frame-support       DEBUG: ✅ no migration for Democracy
runtime::frame-support       DEBUG: ✅ no migration for Randomness
runtime::frame-support       DEBUG: ✅ no migration for AwesomeAvatars
runtime::frame-support       DEBUG: ✅ no migration for Nft
runtime::frame-support       DEBUG: ✅ no migration for NftTransfer
runtime::frame-support       DEBUG: ✅ no migration for AffiliatesAAA
runtime::frame-support       DEBUG: ✅ no migration for TournamentAAA
runtime::system              TRACE: Used block weight: PerDispatchClass { normal: Weight { ref_time: 0, proof_size: 0 }, operational: Weight { ref_time: 0, proof_size: 0 }, mandatory: Weight { ref_time: 13084235827, proof_size: 123647 } }
runtime::system              TRACE: Used block length: 11
runtime                      ERROR: panicked at /home/clang/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cumulus-pallet-aura-ext-0.11.0/src/consensus_hook.rs:69:9:
assertion `left == right` failed: slot number mismatch
  left: Slot(142835867)
 right: Slot(142835866)
[09:33:56.714] WARN (block-builder): Failed to apply inherents {} Error: wasm `unreachable` instruction executed
    app: "chopsticks"
/home/clang/code/chopsticks/packages/core/dist/cjs/blockchain/block-builder.js:213
            throw new Error('Failed to apply inherents');
                  ^

Error: Failed to apply inherents
    at initNewBlock (/home/clang/code/chopsticks/packages/core/dist/cjs/blockchain/block-builder.js:213:19)
    at async buildBlock (/home/clang/code/chopsticks/packages/core/dist/cjs/blockchain/block-builder.js:320:44)
    at async TxPool.buildBlock (/home/clang/code/chopsticks/packages/core/dist/cjs/blockchain/txpool.js:340:43)
    at async TxPool.buildBlockIfNeeded (/home/clang/code/chopsticks/packages/core/dist/cjs/blockchain/txpool.js:324:9)

Node.js v21.7.3

[ermalkaleci]

it will be helpful if you can upload wasm and config

[clangenb]

Sure, I actually forgot that you don't even need the collator binary for this to work, which is why I thought that the repository is better :)

bajun.zip