Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ETH RPC issues #12205

Open
4 of 7 tasks
ashwinphatak opened this issue Jul 10, 2024 · 9 comments
Open
4 of 7 tasks

ETH RPC issues #12205

ashwinphatak opened this issue Jul 10, 2024 · 9 comments
Assignees
@aarshkshah1992
Labels
area/eth-api kind/bug Kind: Bug
Milestone
DX-Streamline

Comments

@ashwinphatak
Copy link

Checklist

  • This is not a security-related bug/issue. If it is, please follow please follow the security policy.
  • I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
  • I did not make any code changes to lotus.

Lotus component

  • lotus Ethereum RPC
  • lotus FVM - Lotus FVM interactions
  • FEVM tooling
  • Other

Lotus Version

Daemon:  1.27.0-dev+mainnet+git.20640c062+api1.5.0
Local: lotus version 1.27.0-dev+mainnet+git.20640c062

and later

Repro Steps

No response

Describe the Bug

  1. lotus gets stuck (bad block), restarting works temporarily till it hits the same issue
  2. snap synced head tracking nodes get slower over time
  3. archive node is slow (eth_getLogs especially)
  4. null block issue - eth_getLogs throws an error if either from or to are a null block (doesn't seem to happen with all nodes, needs more analysis)
  5. consistency issues at head (getLogs seems to be missing events when run at head #12078, but same might be the case for other RPC calls like eth_call)

Tooling

We're using etherjs to make RPC calls

Configuration Options

config.toml


[Fevm]
  # EnableEthRPC enables eth_ rpc, and enables storing a mapping of eth transaction hashes to filecoin message Cids.
  # This will also enable the RealTimeFilterAPI and HistoricFilterAPI by default, but they can be disabled by config options above.
  #
  # type: bool
  # env var: LOTUS_FEVM_ENABLEETHRPC
  #EnableEthRPC = false

  # EthTxHashMappingLifetimeDays the transaction hash lookup database will delete mappings that have been stored for more than x days
  # Set to 0 to keep all mappings
  #
  # type: int
  # env var: LOTUS_FEVM_ETHTXHASHMAPPINGLIFETIMEDAYS
  #EthTxHashMappingLifetimeDays = 0



run script:

export LOTUS_CHAINSTORE_ENABLESPLITSTORE="true"
export LOTUS_EVENTS_ENABLEACTOREVENTSAPI="true"
export LOTUS_INDEX_ENABLEMSGINDEX="true"
export LOTUS_FEVM_ENABLEETHRPC="true"
export LOTUS_FEVM_ENABLEETHHASHTOFILECOINCIDMAPPING="true"

./bin/lotus daemon
@ashwinphatak ashwinphatak added the kind/bug Kind: Bug label Jul 10, 2024
@nikugogoi
Copy link

  1. lotus gets stuck (bad block), restarting works temporarily till it hits the same issue

The nodes running are full nodes (snap synced).
New blocks are not created after a period of time.
We use eth_getBlockByNumber RPC API to get latest block (using latest blockTag)

curl -X POST -H "Content-Type: application/json" \
  --data '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["latest", false],"id":1}' \
  http://localhost:1234/rpc/v1

# Same block is returned when node is stuck
  1. snap synced head tracking nodes get slower over time

eth_getLogs requests get slower over time. RPC calls are generally done near head.
For example the following eth_getLogs request takes different times in returning response for 2 different Lotus nodes (version 1.27.1-rc2)

time curl -X POST -H "Content-Type: application/json" \
  --data '{"method":"eth_getLogs","params":[{
    "fromBlock": "0x3E31B0",
    "toBlock": "0x3E31BB",
    "address":["0xa02cbf1dc75058cc6f2f5b8e7a9087425f5248e3","0x0E4a2276Ac259CF226eEC6536f2b447Fc26F2D8a"],
    "topics":[["0x47cee97cb7acd717b3c0aa1435d004cd5b3c8c57d70dbceb4e4458bbd60e39d4","0x8c5be1e5ebec7d5bd14f71427d1e84f3dd0314c0f7b2291e5b200ac8c7c3b925"]]
  }],"id":1,"jsonrpc":"2.0"}' \
  http://localhost:1234/rpc/v1

{"id":1,"jsonrpc":"2.0","result":[]}

# Different response times are `5.966s` and `20.225s`
  1. archive node is slow (eth_getLogs especially)

eth_getLogs API response time with archive Lotus node (version 1.28.0-rc1) is ~3.5 mins

time curl -X POST -H "Content-Type: application/json" \
  --data '{"method":"eth_getLogs","params":[{
    "fromBlock": "0x3bc92e",
    "toBlock": "0x3bc92f",
    "address":["0xa02cbf1dc75058cc6f2f5b8e7a9087425f5248e3","0x0E4a2276Ac259CF226eEC6536f2b447Fc26F2D8a"],
    "topics":[["0x47cee97cb7acd717b3c0aa1435d004cd5b3c8c57d70dbceb4e4458bbd60e39d4","0x8c5be1e5ebec7d5bd14f71427d1e84f3dd0314c0f7b2291e5b200ac8c7c3b925"]]
  }],"id":1,"jsonrpc":"2.0"}' \
  http://localhost:1234/rpc/v1

{"id":1,"jsonrpc":"2.0","result":[]}
curl -X POST -H "Content-Type: application/json" --data    0.00s user 0.01s system 0% cpu 3:40.03 total
  1. null block issue - eth_getLogs throws an error if either from or to are a null block (doesn't seem to happen with all nodes, needs more analysis)

eth_getLogs API behaviour using null block 4075962, the following are observed:

  • With Lotus node version 1.28.0-rc4+mainnet+git.b5aab0121
    • When null block is set in fromBlock filter param 
      time curl -X POST -H "Content-Type: application/json" \
  --data '{"method":"eth_getLogs","params":[{
    "fromBlock": "0x3E31BA",
    "toBlock": "0x3E31BB"
  }],"id":1,"jsonrpc":"2.0"}' \
  http://localhost:1234/rpc/v1

{"error":{"code":1,"message":"event index does not have event for epoch 4075962"},"id":1,"jsonrpc":"2.0"}
curl -X POST -H "Content-Type: application/json" --data    0.00s user 0.00s system 48% cpu 0.014 total
    • When null block is set in toBlock filter param 
      time curl -X POST -H "Content-Type: application/json" \
  --data '{"method":"eth_getLogs","params":[{
    "fromBlock": "0x3E31B0",
    "toBlock": "0x3E31BA"
  }],"id":1,"jsonrpc":"2.0"}' \
  http://localhost:1234/rpc/v1

{"error":{"code":1,"message":"context deadline exceeded"},"id":1,"jsonrpc":"2.0"}
curl -X POST -H "Content-Type: application/json" --data    0.00s user 0.00s system 0% cpu 1:30.01 total
    • When block range in eth_getLogs filter has null block in between, there is no error 
      time curl -X POST -H "Content-Type: application/json" \
  --data '{"method":"eth_getLogs","params":[{
    "fromBlock": "0x3E31B0",
    "toBlock": "0x3E31BB"
  }],"id":1,"jsonrpc":"2.0"}' \
  http://localhost:1234/rpc/v1

{"id":1,"jsonrpc":"2.0","result":[...]}
curl -X POST -H "Content-Type: application/json" --data    0.00s user 0.01s system 0% cpu 18.743 total
  • With Lotus node version 1.27.1-rc2+mainnet+git.82e35e9b4 eth_getLogs works with null block in all above scenarios 
    time curl -X POST -H "Content-Type: application/json" \
  --data '{"method":"eth_getLogs","params":[{
    "fromBlock": "0x3E31BA",
    "toBlock": "0x3E31BB"
  }],"id":1,"jsonrpc":"2.0"}' \
  http://localhost:1234/rpc/v1

{"id":1,"jsonrpc":"2.0","result":[]}
curl -X POST -H "Content-Type: application/json" --data    0.00s user 0.01s system 0% cpu 12.339 total

Major ETH RPC APIs used in subgraphs/watchers are:

  • eth_getLogs
  • eth_call
  • eth_getBlockByNumber
  • eth_getBlockByHash
  • eth_getTransactionByHash
  • eth_getTransactionReceipt

@aarshkshah1992
Copy link
Contributor

@nikugogoi @ashwinphatak

Thank you for all the details.

lotus gets stuck (bad block), restarting works temporarily till it hits the same issue

  • What Lotus version is this happening for ? Is it a new problem or has this been a long running problem for you ?

@aarshkshah1992 aarshkshah1992 self-assigned this Jul 10, 2024
@aarshkshah1992 aarshkshah1992 mentioned this issue Jul 10, 2024
Merged
@ashwinphatak
Copy link
Author

One of the lotus snap synced nodes just got stuck (doesn't move past block 4079492):

lotus version 1.27.1-rc2+mainnet+git.82e35e9b4

curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["latest", false],"id":1}' http://localhost:1234/rpc/v1
{"id":1,"jsonrpc":"2.0","result":{"hash":"0xbcd10c8a3b5e47a167e87557aca649adf7667e8f9ccb90117819b01691d77526","parentHash":"0x030e47ad5b59fa5e1fa0ad5fcfc4e519e6ad3bd32d6ca0b85ebc8d45d77fdfcc","sha3Uncles":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","miner":"0x0000000000000000000000000000000000000000","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","transactionsRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","receiptsRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","logsBloom":"0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff","difficulty":"0x0","totalDifficulty":"0x0","number":"0x3e3f84","gasLimit":"0x104c533c00","gasUsed":"0x3f7d260ff","timestamp":"0x668fa9d8","extraData":"0x","mixHash":"0x0000000000000000000000000000000000000000000000000000000000000000","nonce":"0x0000000000000000","baseFeePerGas":"0x79","size":"0x0","transactions":["0xbd678be76547f52c63790caab13813416e04b110eed80f7d985da8938d1d4c19","0x555aa841135adbb6e02d2faf323ff1467373c112fdbce672aee76ca267a5c679","0x5be76b963b5abcc80583801464cce932309ae25f19062fa0079e36cc3b245875","0x2363485372c68025942f59f9c9dcc754ea29717fed7c3883c54d800179c52d42","0x96b35cdbc92483f90927baf6cdacb575746c19671c09aef0f9c8a892ae428236","0x16d4c5f322bb550c49980c2e7a910f5baab56172619aa2eb767084a992a49197","0x46f418789b13c7e991e3103ec638771f2e5d267745ee417ed734ff497fa43b6b","0x352b501db25e792175c4ad190b457d4e1add2166b90dde7fadb9a6e183c07594","0xb12bc422eba8c9d2b421e1d0890ab518083a2950596b29b61682dab35bc4c7c5","0x58de9a22d58d99787e659e774a077253199f171666af29f842b54395183300e2","0x7523e6927e752cd682497e15253470780457f180941965bde9c61f1b732744c3","0xa2f73b2b098b664cdb3726114cf2a37af1a1d62728664db05f2fd336590fec75","0xcacccc5553d4a8f27cf5a605813e8926b00dcf10c9cb620aa06b1c6cb44ea076","0x6d4e62da748aacb0c56daa5569e241151a94db8704f8f1ad120e1115e1201f83","0x529c2d8045b19cdbfaedee278d380c79d313960159a7d1e4aa5bc39fe8b7a178","0x8f116ef00d75cb2045a72a3141b7da4db1ec1cc354f58bf17411adb5460a2899","0x6ae473f7e8482300295946c81cfc23f47fd5ae6b49038ad2de6f17f439af585c","0x8d85e1adffec98a3bd43e8ca1a41ca5235995606692e93fbff861613888b3c32","0xd17b27cbfc1a5dd5d03d6e2dc894c14d4f153be9a2025476468827f357a9839f","0xdaeaf1e76b570e5c2b4509b01d25c459f951cf250a311033b9c6f4e7cea01890","0x1a665837e54534a8bff43f6c0d9214ac93cf822576469b5ecdfe1df530963a94","0x8ee5c82741672a0fbd0d5ea55a7b5a0eabcba3b27e619aa2c5a8f1e20b3c3693","0x1cafe8a03288c099dd905d63cfb0908f6d138eb289159ca18f24e1ec47a47401","0x21680dd45907b16172b76f25ab7551f2339e805e1a8122f09dafbb61e5bac301","0xe25baf5b639b617b29eb23e7f110e927f0d62ebddb77a1b6d413aed1a783994e","0x6e75fb131ac531efc470b3eff17b9a53e50efdcbb274d8e835590c122a17169c","0x58ff5374111dd74b6013ab5e46fdb9dd86ed47235142049912d9ae858a30bb2e","0x785d4b7bc3c0b0ee1cf2b8b2a2edb53909f58a4ee17d022fad476ffbd93b56d9","0x563ac73a7c6d64ad1792445d47d10cc3054b6cb10f6beb82643a21321e22f7f3","0x2deee28ff1420882cdf370180f09e0c72217f3bb0654ceb3e99869c43b37e6fd","0x977873181a152106edf82907d24531a67751d69bd7bf948e4fae9a91a496b56f","0x8bbfa3cb9050649d0b2a16dd387d14ef0c301781d2e5f924912e966296b0b1ad","0x56b1544f8ef601b52d0d115044926a9e2bfdbfdf35a03ff4b184aa1f64e5cf0c","0xeb63dc3974420920054c9f8845553980079a1c8e5692f9bfe870bb6567f2809e","0x8e9f249f42778d8feea04195ad1097512dbc06956d2208272f70f04ad73d788c","0x5cfbe6c00fdd9dad5dffc46e90d9d20e4763ca28ee2c86a5f9ba7d415bd63a46","0x459cee756a6cbc25190ca8e3238e14686e4c18343664de0d0f6fbd261f53a413","0x7ffe18c09aaea8fe99bf7817fe083042f17d4a4181eea2c87fa69b2f34366d83","0xd0b421971aa13e5ef28a62163797b476c312a863c0b55dc26d5e06aa5389f641","0xca19daf994c75df6da36e51e2c38d04e079e2206679756ec95840075435cd0cf","0xb5e969387820a65e119e6ebf800c347059ad5a1b02b820784ce68f8f27c4bd32","0x8746e31b0280bc6ea7537094132fdd60e9e2ff9f86ed1b93176dd7ed569d3813","0xeabcf7fc08db752e5bf228bbf340afbe6f380e8aa10fc3799d26e407b95fc6b7","0xfd519165e7b75293e4585bf28a9fe7bf5924d8ac64c71bcc6165919c375ce0d7","0xc3cc9bf4cf7628dd2492aa6a51d6bae4e4cd52c55e7df637509b640237844063","0xbde15e9eee955c2e02628ebcaa823651ac7439dc382987e049d4d0690f5a1fc7","0x3697ca76dc7fc3e2aaf30cf01652676f3a984f8d9415d9ce6a8379e815a60e79","0xd147f9278fb49a220d7e0476307bda996dec83fde52933be1a788c595c2a5baf","0xd2b70430ecf498ae90bf8a4a8c7193e2fd1ba6a240d15f4c6d3c299192062f4f","0x9eae584844b8c6f63ba335d91f2be8e25c0a6a60e883b2158c724436306abf50","0x8308e45944391a5c174bc399a4b9130768efc7eca0035bd7eab9f07d9bfde02f","0x54543fd91e72c2c8e83a4935df063224250e8de5bc25c674c39492a6128e623a","0xc66dc2a15ba5dde9715d8154db64fba85a11ab6e3b1a9925aa9016411fda642a","0xdd7a25643e25c4b4e5364d22258d5cd408da8329c5bbf1b14edbcf1a5a8aec93","0xcf9870a6cc4e4872f86d21e7b28f746a1a52bf207185691d379194dfb59bc41a","0x3653f876f3a262b11378c370f417c17c4a07fce01356e71a6f56f183744dc919","0x68ee1f506446ad9aaa1bbbe289f9bbcb0c55fc4d01111e68668f324e39d48dab","0x3a16ae085b6771c47d5ad06532a7e9ac9f22078cf5fa50d47a315966a5955986","0xb2eaec046ca8d9e97c0b0a6ac8bdaf54f1d87cd46911a12085ae1af067750ca6","0xe3d30566e7c597e694198039d1e43d60577c238c9ea371bfe4f9d544377c7e44","0xf16c8584663abe072eeba7b0a628fdfcb7f13f4bd7afc4bb0262a734020edaa5","0xa8c47fdfd2565a70a0e52fa8bacf7a7a335ac0c5751df1000bf491baf38badd9","0x5690fc8ea45276805561817e1f031ab418cc9885b5062f23506796fea35b44df","0x054f0068b3bd2df6b7c7030d0a795760c1817a98102f7489c4703c278694cbf2","0x397c9613146a58d2522228f3b926bc026d1c997ff452a0531c33057b6e53c831","0x39f3f784540bc7e652a4b5cc48adf464d34859f0dc15c2d24111fbb64328f620","0x001e3455568abeb023c3a83ba5c363535ebcc456ef3f152bc03ea9f1f6293436","0xfeefccf9d245bb0ab0ffe535a1600197aaa30203970ba3101cebc552c0af74fb","0x4448f99499784ed3c53aec9dc27c9822c6bf0415956a5e58bca958bbb8f4fa88","0x0ff36feddbdb31aaeb8dc5a15a035e98df8d969b92ad0eb05abb750649faa2db","0xd44a98c501842220e07d349c5c54f739e4099633ee1eb59553f8c20f012d8453","0xde9c03b3f0f96b895ae6e4454fdd327e61256840b7b023acf1ab5de66d88fffc","0x46f6d2cece4c04137a22acd74a596e25fbba492d96f115c4c13e839ec2a11e7d","0xa4f8c2402330ba701476da305f2d7a7a0453a7c0117c55465d5e7af4a9f610ed","0x079e51fd048d025eb0755a7cb9f681163ced98bd52711eec785ba02b3d5c2fbf","0x57feee990658cbcfe17cce3706e9feb9432a0932de0947d069c69cebbd165d88","0x997ec7709f498900c3cbc451251ab88ccdcbb100584100bb5bea268df0d6a617","0x1e950f3c8a98d2a03d2b981c25d700f2ca923b163b9e0a0322e279072ffd1ecb","0x7522271875d5859f0050f0dd29db56f93bc3b33e0a8e62e1269705bd762a0ac8","0xe0099932196f10382532e068a5e549a8d2d2412b218dcaea1434389047f9c5f0","0x5a17072ae784fe146262eebfcd71e55686005bf71216bb2b1a8892165740685c","0x48ee660c026abb4002d6e176a5d11a35fe1b0b99c1334807502c15eefd8f4c84","0x149f921be1a6c75ed1c558893e9c02d671339e4fe2bd5226b99c41dd5bb0d1fe","0x7f3b1a7ac02ac0651099cec9df558707a1cdb402da72ea5ed1f11bf388f14889","0x8d6043b2d97fca0886bb4bb0b47a80f66cc1a2190f04d53bca33086575961a56","0x4bb85b7362b58d5de669fdea8b74a51254ba008b9240808ed57260783e22a6ef","0x4faa44cf4ec340d78e1e306656a452d3736834f9266a2c0b629fa56f18a93501","0x7ba219eb3d254065fcd6bf8e8a610c20562564a29bf1f592aab9089cdc3574fd","0x0a7c6178c8f03824fcbee9dc9078bdd0a5e3dfbcd142afe756e439a1775da1b4","0xf35b0e8bfc88147e0a7d6db08bf7c8302cd04eb3f348b8f77ebe1f71157a73fe","0x1a81829dfd6f12d66799dee19cfe0b3359bffdda35780011be90c2719aa1fb2c","0x2d6f4fb8ddba83d871871a8d66a724a55fee49af29dff5a3052a13633d497f2d","0xcda184f71b432167a64a1901d9a5066e62102427393636e80e2e4f39d5d82294","0xdb886bb678a340d664551dc72c0e7a75c93e93be83720f088be5e712b115ed39","0x9cbbce10c9dffe0414670ea8b255f62039db4f1876b8fcf8ca022b858ffdc3ba","0xb57e30cf7db63e56d6777eb2a664258bd016086fe55bb099416254a7c92896e0","0x634ca9404164cfd4fb5bffb3f10e9d75d38fd131c74bbb69db78fbf60219df01","0xb014e8806196df57fa50bbfd3f0248f7b6fd95e1952d65c36659f0cfbe1b57b1","0x5f8e17a690b26d957ce33ff7ec4c46cea23de86a9e36d75dec8982dd51f8126e","0x6eab4db8a283d599a6ecba86494c6697d3d14ddb4bb791201117a8dcc1a48b43","0x7c8dadf3278012c3b02bb333228c8d5776fa2c01d126f8d05e65bd1617f4c2ea","0xca791214ab77095da93082c1d5fabf8ffbe4c1fea81d5151e17516bfab30f56b","0x659daa936df0680df327f741fa4b57e95577d6fb9927c6491e898ea9be422bd5","0xa366a3df447cfd02bd7ad924692a8e78a1b4cbf3f60c7d22b52e34ffe3ad1ca4","0xa6a3a20e66fa80e2bb947a364d0f4ea27fc11f6a2348708e62dd6a780aa37630","0x91a074ed0f98543c194754250a961b5202445a6d5b0da23d46820a445411986b","0x0db23f12af69689e0d037ebf65c2160702130478c9228441e445087eb51371c5","0x15fd6fc89051ae7df2b44b1b24c23859c8eb2b1c80a98bc71cad8f29ebce7094","0x1a6c78c53dc3e16cee129ce4304f2cfeddd82d05360334f6aafb352ceda51b94","0xe599d9fc8682148f87e323cb3b3712aa23550bc5eca15656b589d73603597ac4","0x396299be9f80b15d95cfedbc6a100fbecfa837d6de85d3c9a87022d04fc16feb"],"uncles":[]}}

Recent logs from the lotus terminal:

2024-07-11T10:50:00.666Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-119352060", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-11T10:50:00.673Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-119352061", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-11T10:50:00.685Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-119352062", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-11T10:50:00.700Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-119352063", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-11T10:50:00.783Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-119352064", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-11T10:50:00.810Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-119352065", "edge": "transient", "direction": "Outbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve outbound stream: resource limit exceeded"}
2024-07-11T10:50:00.821Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-119352066", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-11T10:50:00.893Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-119352067", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-11T10:50:00.909Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-119352068", "edge": "transient", "direction": "Outbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve outbound stream: resource limit exceeded"}
2024-07-11T10:50:00.909Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-119352069", "edge": "transient", "direction": "Outbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve outbound stream: resource limit exceeded"}
2024-07-11T10:50:00.991Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-119352070", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-11T10:50:01.059Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-119352071", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-11T10:50:01.073Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-119352072", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-11T10:50:01.125Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-119352073", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-11T10:50:01.264Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-119352074", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-11T10:50:01.287Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-119352075", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}

@rvagg rvagg mentioned this issue Jul 11, 2024
Closed
@ashwinphatak
Copy link
Author

Lotus node version 1.28.0-rc4+mainnet+git.b5aab0121 is pretty much unusable at this point as we keep running into the error in point 4 here: #12205 (comment)

Switching over to 1.27 to keep the watchers running.

@aarshkshah1992
Copy link
Contributor

@ashwinphatak We just got a fix rolled out for 4 at #12212
Let me get it in a RC for you.

@nikugogoi
Copy link

One of the upgraded Lotus nodes also got stuck

lotus --version
lotus version 1.28.0-rc5+mainnet+git.304e1681b

It is stuck at block 4090469

curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["latest", false],"id":1}' http://localhost:1234/rpc/v1

{"id":1,"jsonrpc":"2.0","result":{"hash":"0x2fcce5f961287e555f86b2f752173e1a50da6d70161a9e12d990f255bf7a317f","parentHash":"0xf5393626dd0973469d9f748dd251aae14574df3e9b7ee369c4518169c9845565","sha3Uncles":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","miner":"0x0000000000000000000000000000000000000000","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","transactionsRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","receiptsRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","logsBloom":"0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff","difficulty":"0x0","totalDifficulty":"0x0","number":"0x3e6a65","gasLimit":"0x6fc23ac00","gasUsed":"0x19047be48","timestamp":"0x6694b036","extraData":"0x","mixHash":"0x0000000000000000000000000000000000000000000000000000000000000000","nonce":"0x0000000000000000","baseFeePerGas":"0x64","size":"0x0","transactions":[...],"uncles":[]}}

Recent logs from the lotus terminal:

2024-07-15T06:59:29.114Z        INFO    canonical-log   swarm/swarm_listen.go:136       CANONICAL_PEER_STATUS: peer=12D3KooWLbKyxQaBreWj1yaJdoabkq91PX15aJCSXyXXmgjc2ska addr=/ip4/203.176.226.195/udp/49056/quic-v1 sample_rate=100 connection_status="established" dir="inbound"
2024-07-15T06:59:29.166Z        INFO    chain   chain/sync.go:212       received block outside of consensus range at height 3875401
2024-07-15T06:59:30.113Z        INFO    bs:sess session/session.go:462  No peers - broadcasting {"session": 43036, "want-count": 2}
2024-07-15T06:59:30.469Z        WARN    sub     sub/incoming.go:556     cannot authenticate message     {"err": "miner id does not map to peer that sent message", "errVerbose": "miner id does not map to peer that sent message:\n    github.com/filecoin-project/lotus/chain/sub.(*IndexerMessageValidator).authenticateMessage\n        /home/lotus/lotus/chain/sub/incoming.go:629", "peer": "12D3KooWEmxCWa2a2mCt9sDEjvp1iCupxkGETATH4TiK5ckmma4Z", "minerID": "f03098989"}
2024-07-15T06:59:30.587Z        WARN    sub     sub/incoming.go:556     cannot authenticate message     {"err": "miner id does not map to peer that sent message", "errVerbose": "miner id does not map to peer that sent message:\n    github.com/filecoin-project/lotus/chain/sub.(*IndexerMessageValidator).authenticateMessage\n        /home/lotus/lotus/chain/sub/incoming.go:629", "peer": "12D3KooWEmxCWa2a2mCt9sDEjvp1iCupxkGETATH4TiK5ckmma4Z", "minerID": "f03098989"}
2024-07-15T06:59:31.040Z        INFO    chain   chain/sync.go:212       received block outside of consensus range at height 4089214
2024-07-15T06:59:31.278Z        INFO    chain   chain/sync.go:212       received block outside of consensus range at height 3000
2024-07-15T06:59:31.285Z        INFO    chain   chain/sync.go:212       received block outside of consensus range at height 4040593
2024-07-15T06:59:31.368Z        INFO    chain   chain/sync.go:212       received block outside of consensus range at height 4052057
2024-07-15T06:59:31.371Z        INFO    chain   chain/sync.go:212       received block outside of consensus range at height 4084008
2024-07-15T06:59:31.450Z        INFO    chain   chain/sync.go:212       received block outside of consensus range at height 3477995
2024-07-15T06:59:33.014Z        INFO    net/identify    identify/id.go:460      failed negotiate identify protocol with peer    {"peer": "12D3KooWLm9mRUmojHge2QWA4woiujxEm2oZ2TD6mVtBXq4L9Fa9", "error": "stream reset"}
2024-07-15T06:59:33.015Z        WARN    net/identify    identify/id.go:434      failed to identify 12D3KooWLm9mRUmojHge2QWA4woiujxEm2oZ2TD6mVtBXq4L9Fa9: stream reset
2024-07-15T06:59:33.179Z        WARN    chainxchg       exchange/client.go:123  could not send request to peer 12D3KooWLm9mRUmojHge2QWA4woiujxEm2oZ2TD6mVtBXq4L9Fa9: failed to read chainxchg response: stream reset
2024-07-15T06:59:33.179Z        ERROR   hello   hello/hello.go:130      failed to fetch tipset from peer during hello: failed to doRequest:
    github.com/filecoin-project/lotus/chain/exchange.(*client).GetFullTipSet
        /home/lotus/lotus/chain/exchange/client.go:364
  - doRequest failed for single peer 12D3KooWLm9mRUmojHge2QWA4woiujxEm2oZ2TD6mVtBXq4L9Fa9:
    github.com/filecoin-project/lotus/chain/exchange.(*client).doRequest
        /home/lotus/lotus/chain/exchange/client.go:146
2024-07-15T06:59:34.532Z        INFO    chain   chain/sync.go:212       received block outside of consensus range at height 4077636

@nikugogoi
Copy link

Posting logs from 2 nodes that got stuck at block 4,099,270

  • CURL request

    curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["latest", false],"id":1}' http://localhost:1234/rpc/v1
{"jsonrpc":"2.0","result":{"hash":"0xe16bad75433abcbc33aac08ba3f4850455c1e59e44daed6562656cc44ad229d1","parentHash":"0xb8a602c4cec7aadda27e5505187b89ee881d499407ce302bfe61feef73bc8dcb","sha3Uncles":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347","miner":"0x0000000000000000000000000000000000000000","stateRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","transactionsRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","receiptsRoot":"0x0000000000000000000000000000000000000000000000000000000000000000","logsBloom":"0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff","difficulty":"0x0","totalDifficulty":"0x0","number":"0x3e8cc6","gasLimit":"0x2540be400","gasUsed":"0x1d3d4ab67","timestamp":"0x6698b794","extraData":"0x","mixHash":"0x0000000000000000000000000000000000000000000000000000000000000000","nonce":"0x0000000000000000","baseFeePerGas":"0x64","size":"0x0","transactions":[...],"uncles":[]},"id":1}

  • lotus version 1.28.0-rc5+mainnet+git.304e1681b

    2024-07-18T09:14:31.309Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-13752798", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-18T09:14:31.667Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-13752799", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-18T09:14:31.701Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-13752800", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-18T09:14:32.302Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-13752801", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-18T09:14:32.386Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-13752802", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-18T09:14:32.505Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-13752803", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-18T09:14:32.842Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-13752804", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-18T09:14:33.044Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-13752805", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-18T09:14:33.124Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-13752806", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-18T09:14:33.586Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-13752807", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-18T09:14:34.054Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-13752808", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-18T09:14:35.001Z        DEBUG   rcmgr   resource-manager/scope.go:480   blocked stream from constraining edge   {"scope": "stream-13752809", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}

  • lotus version 1.27.0+mainnet+git.95c42dc5c

    2024-07-18T09:14:17.686Z	DEBUG	rcmgr	resource-manager/scope.go:480	blocked stream from constraining edge	{"scope": "stream-27559465", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-18T09:14:18.785Z	DEBUG	rcmgr	resource-manager/scope.go:480	blocked stream from constraining edge	{"scope": "stream-27559466", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-18T09:14:19.755Z	DEBUG	rcmgr	resource-manager/scope.go:480	blocked stream from constraining edge	{"scope": "stream-27559467", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-18T09:14:22.704Z	DEBUG	rcmgr	resource-manager/scope.go:480	blocked stream from constraining edge	{"scope": "stream-27559468", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-18T09:14:24.759Z	DEBUG	rcmgr	resource-manager/scope.go:480	blocked stream from constraining edge	{"scope": "stream-27559469", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-18T09:14:26.904Z	WARN	sub	sub/incoming.go:555	cannot authenticate message	{"err": "miner id does not map to peer that sent message", "errVerbose": "miner id does not map to peer that sent message:\n    github.com/filecoin-project/lotus/chain/sub.(*IndexerMessageValidator).authenticateMessage\n        /home/lotus/lotus/chain/sub/incoming.go:628", "peer": "12D3KooWEmxCWa2a2mCt9sDEjvp1iCupxkGETATH4TiK5ckmma4Z", "minerID": "f03098989"}
2024-07-18T09:14:26.904Z	WARN	sub	sub/incoming.go:555	cannot authenticate message	{"err": "miner id does not map to peer that sent message", "errVerbose": "miner id does not map to peer that sent message:\n    github.com/filecoin-project/lotus/chain/sub.(*IndexerMessageValidator).authenticateMessage\n        /home/lotus/lotus/chain/sub/incoming.go:628", "peer": "12D3KooWEmxCWa2a2mCt9sDEjvp1iCupxkGETATH4TiK5ckmma4Z", "minerID": "f03098989"}
2024-07-18T09:14:26.904Z	WARN	sub	sub/incoming.go:555	cannot authenticate message	{"err": "miner id does not map to peer that sent message", "errVerbose": "miner id does not map to peer that sent message:\n    github.com/filecoin-project/lotus/chain/sub.(*IndexerMessageValidator).authenticateMessage\n        /home/lotus/lotus/chain/sub/incoming.go:628", "peer": "12D3KooWEmxCWa2a2mCt9sDEjvp1iCupxkGETATH4TiK5ckmma4Z", "minerID": "f03098989"}
2024-07-18T09:14:26.904Z	WARN	sub	sub/incoming.go:555	cannot authenticate message	{"err": "miner id does not map to peer that sent message", "errVerbose": "miner id does not map to peer that sent message:\n    github.com/filecoin-project/lotus/chain/sub.(*IndexerMessageValidator).authenticateMessage\n        /home/lotus/lotus/chain/sub/incoming.go:628", "peer": "12D3KooWEmxCWa2a2mCt9sDEjvp1iCupxkGETATH4TiK5ckmma4Z", "minerID": "f03098989"}
2024-07-18T09:14:26.904Z	WARN	sub	sub/incoming.go:555	cannot authenticate message	{"err": "miner id does not map to peer that sent message", "errVerbose": "miner id does not map to peer that sent message:\n    github.com/filecoin-project/lotus/chain/sub.(*IndexerMessageValidator).authenticateMessage\n        /home/lotus/lotus/chain/sub/incoming.go:628", "peer": "12D3KooWEmxCWa2a2mCt9sDEjvp1iCupxkGETATH4TiK5ckmma4Z", "minerID": "f03098989"}
2024-07-18T09:14:27.711Z	DEBUG	rcmgr	resource-manager/scope.go:480	blocked stream from constraining edge	{"scope": "stream-27559470", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}
2024-07-18T09:14:27.741Z	DEBUG	rcmgr	resource-manager/scope.go:480	blocked stream from constraining edge	{"scope": "stream-27559471", "edge": "transient", "direction": "Inbound", "current": 2268, "attempted": 1, "limit": 2268, "stat": {"NumStreamsInbound":0,"NumStreamsOutbound":2268,"NumConnsInbound":0,"NumConnsOutbound":0,"NumFD":0,"Memory":0}, "error": "transient: cannot reserve stream: resource limit exceeded"}

@rvagg
Copy link
Member

rvagg commented Jul 22, 2024

@nikugogoi we have 2 go-libp2p upgrades in master now that both address some connection leaks that may address this. I think we'll probably be getting those into an upcoming 1.28.0 release and we'll keep you informed.

@aarshkshah1992
Copy link
Contributor

@nikugogoi

#12261 should help with the performance problems in eth_getLogs. Will let you know once it's merged and relased.

@BigLep BigLep added this to the DX-Streamline milestone Jul 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aarshkshah1992 aarshkshah1992

Labels
area/eth-api kind/bug Kind: Bug
Projects
FilOz
Status: ⌨️In Progress
Milestone
DX-Streamline
Development

No branches or pull requests
5 participants
@ashwinphatak @BigLep @rvagg @aarshkshah1992 @nikugogoi