"datastore: key not found" / "failed to lookup index for mh" #499

RobQuistNL · 2022-05-02T11:40:45Z

Checklist

This is not a security-related bug/issue. If it is, please follow please follow the security policy.
This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
This is not a new feature request. If it is, please file a feature request instead.
This is not an enhancement request. If it is, please file a improvement suggestion instead.
I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
I did not make any code changes to lotus.

Lotus component

Lotus Version

1.15.1

Describe the Bug

I have a host that

has data sealed & proving
has fast retrieval enabled for this deal
has the data unsealed available
has had many restarts of the markets actor because it freezes / breaks

But when trying to retrieve the data from the host, the retrieve command returns the following;

$ lotus client retrieve --provider f01392893 --car Qmf1ykhUo63qB5dJ8KRyeths9MZfyxpVdT5xi1moLKefz7 ~/testingRetrieval.car
ERROR: offer error: retrieval query offer errored: failed to fetch piece to retrieve from: getting pieces for cid Qmf1ykhUo63qB5dJ8KRyeths9MZfyxpVdT5xi1moLKefz7: getting pieces containing block Qmf1ykhUo63qB5dJ8KRyeths9MZfyxpVdT5xi1moLKefz7: failed to lookup index for mh 1220f7ce2d20772b959c1071868e9495712f12785b1690ee88752af120dd49338190, err: datastore: key not found

Is the markets actor broken? Does it not know where to find the data? How can we repair it?

Logging Information

Repo Steps

The text was updated successfully, but these errors were encountered:

Reiers · 2022-05-02T16:03:39Z

Hi @RobQuistNL

Im seeing the same - currently re-initiating the dagstore again to see if that resolves it.

btw can you share the output of:
du -sch /.lotusmarkets/dagstore/* | sort -rh
or if Boost:
du -sch /.boost/dagstore/* | sort -rh

after that run:
lotus-miner dagstore gc

see if there are any change in size dagstore/*

Im seeing that after I pass around 100GB in size dagstore/datastore - my deals are starting to get stuck in Staged.
Just wondering if you are seeing the same - maybe its tied together.

RobQuistNL · 2022-05-02T20:06:10Z

$ du -sch ~/.lotusmarkets/dagstore/* | sort -rh
119G	total
106G	/home/master/.lotusmarkets/dagstore/index
13G	/home/master/.lotusmarkets/dagstore/datastore
4.0K	/home/master/.lotusmarkets/dagstore/transient

I have not seen any deals stuck in Staged yet - but the markets actor takes forever to start.

For sake of clarity I'm logging the how-to on re-initializing the dagstore here: https://lotus.filecoin.io/storage-providers/configure/dagstore/#first-time-migration

cryptowhizzard · 2022-05-03T10:53:58Z

I am trying to recover 3 shards here. Have removed and terminated 3 sectors and did the --recover afterwards to see if i can get it back on track.

lotus-miner dagstore list-shards | grep Recovering
baga6ea4seaqensd5t7wps4e7cx5omnd2lgegylfpqzrnspy2pdt5scikpb6q4aq Recovering failed to acquire reader of mount on initialization: failed to unseal deal 5328589: failed to unseal piece from sector 23702: unsealing piece: cannot unseal piece (sector: {{1240 23702} 8}, offset: 0 size: 34091302912) - unsealed cid is undefined
baga6ea4seaqhskb6cxxruponnhxz6g5tkkevgboigomj7iixdb6svt2ebt6xeoy Recovering failed to acquire reader of mount on initialization: failed to unseal deal 5328587: failed to unseal piece from sector 23701: unsealing piece: cannot unseal piece (sector: {{1240 23701} 8}, offset: 0 size: 34091302912) - unsealed cid is undefined
baga6ea4seaqkfxpjjdmyb5dioekpvbhjbvmwzk2v2hyyicjjuuzt7j2upsfmehi Recovering failed to acquire reader of mount on initialization: failed to unseal deal 2298967: failed to unseal piece from sector 15795: unsealing piece: worker UnsealPiece call: storage call error 0: unsealing sector: unseal range: No space left on device (os error 28)

brendalee · 2022-05-05T05:24:40Z

thanks for reporting! tagging @dirkmc @nonsense and @LexLuthr for awareness.

Tagging a dagstore related PR which could be helpful: #494

RobQuistNL · 2022-05-05T09:58:45Z

I have just ran the entire re-indexing (which took almost 2 days) but I still can not retrieve some deals that are 100% active on chain.

RobQuistNL · 2022-05-05T10:00:15Z

lotus client ls --provider=f01392893 QmVzK5WbFrVta19J3nbiMTE46XsBFCWSYFKzNHsEyo9NXJ
ERROR: retrieve: offer error: retrieval query offer errored: failed to fetch piece to retrieve from: getting pieces for cid QmVzK5WbFrVta19J3nbiMTE46XsBFCWSYFKzNHsEyo9NXJ: getting pieces containing block QmVzK5WbFrVta19J3nbiMTE46XsBFCWSYFKzNHsEyo9NXJ: failed to lookup index for mh 122071a7aeb3eb45f3a054cd64eb683291868550e870c1cfab1b9a306c55a2ce6431, err: datastore: key not found

$ lotus client ls --provider=f01392893 --pieceCid baga6ea4seaqnho54pbhiwiz5iltkb3lmexcw4bwejdy7gpqx35cdnzamtlbt4ba QmVzK5WbFrVta19J3nbiMTE46XsBFCWSYFKzNHsEyo9NXJ
ERROR: retrieve: offer error: retrieval query offer errored: failed to fetch piece to retrieve from: getting pieces for cid QmVzK5WbFrVta19J3nbiMTE46XsBFCWSYFKzNHsEyo9NXJ: getting pieces containing block QmVzK5WbFrVta19J3nbiMTE46XsBFCWSYFKzNHsEyo9NXJ: failed to lookup index for mh 122071a7aeb3eb45f3a054cd64eb683291868550e870c1cfab1b9a306c55a2ce6431, err: datastore: key not found

but the piecestore shows it:

master@daemon:~$ lotus-miner pieces cid-info QmVzK5WbFrVta19J3nbiMTE46XsBFCWSYFKzNHsEyo9NXJ
Info for:  QmVzK5WbFrVta19J3nbiMTE46XsBFCWSYFKzNHsEyo9NXJ
PieceCid                                                          Offset  Size
baga6ea4seaqnho54pbhiwiz5iltkb3lmexcw4bwejdy7gpqx35cdnzamtlbt4ba  0       0
master@daemon:~$ lotus-miner pieces piece-info baga6ea4seaqnho54pbhiwiz5iltkb3lmexcw4bwejdy7gpqx35cdnzamtlbt4ba
Piece:  baga6ea4seaqnho54pbhiwiz5iltkb3lmexcw4bwejdy7gpqx35cdnzamtlbt4ba
Deals:
DealID   SectorID  Length       Offset
4785546  152358    34359738368  0

master@daemon:/mnt/md0/dagstore$ lotus-miner --call-on-markets dagstore initialize-all --concurrency 5 --include-sealed
(...)
(4/9205) baga6ea4seaqkuhx7eqsfvkldaefzbc3y5xrrdjxcgr7tu7u4ak2cnyyej4zegny SUCCESS
(...)

$ lotus client ls --provider=f01392893 baga6ea4seaqkuhx7eqsfvkldaefzbc3y5xrrdjxcgr7tu7u4ak2cnyyej4zegny
ERROR: retrieve: offer error: retrieval query offer errored: failed to fetch piece to retrieve from: getting pieces for cid baga6ea4seaqkuhx7eqsfvkldaefzbc3y5xrrdjxcgr7tu7u4ak2cnyyej4zegny: getting pieces containing block baga6ea4seaqkuhx7eqsfvkldaefzbc3y5xrrdjxcgr7tu7u4ak2cnyyej4zegny: failed to lookup index for mh 922020aa1eff24245aa963010b908b78ede311a6e2347f3a7e9c02b426e3044f324337, err: datastore: key not found

dirkmc · 2022-05-05T13:27:58Z

@RobQuistNL I'm not sure what's going wrong here. As part of the DAG store initialization process the DAG store:

Gets the data for the piece from the sealer
Extracts the list of multihashes in the piece and adds them to two indexes:
1. An index from <piece cid> => <all multihashes in the piece>
2. An index from <multihash> => <cids of all pieces containing the multihash>

The output of initialize-all that you pasted shows that the indexing completed correctly for the piece.

When the client tries to retrieve a file using the payload CID:

The client sends the payload CID
The provider looks up the multihash of the payload CID in the index of <multihash> => <piece cids>
If it can't find the multihash in this index it outputs the error you're seeing: failed to lookup index for mh

Something I noticed that seems odd in the output that you pasted:

$ lotus-miner pieces cid-info QmVzK5WbFrVta19J3nbiMTE46XsBFCWSYFKzNHsEyo9NXJ
Info for:  QmVzK5WbFrVta19J3nbiMTE46XsBFCWSYFKzNHsEyo9NXJ
PieceCid                                                          Offset  Size
baga6ea4seaqnho54pbhiwiz5iltkb3lmexcw4bwejdy7gpqx35cdnzamtlbt4ba  0       0

The piece should not have a size of zero. I wonder if there's a problem with that particular CAR file.

Are you seeing this same error when attempting to retrieve other pieces?

RobQuistNL · 2022-05-09T09:42:47Z

Yes, this error happens with a lot of my deals. They have mostly been generated like so;

ipfs add -r --nocopy --pin=false folder > folder.out

then I check the last line of the .out (which should have the dag root hash)
I export that;

ipfs dag export -p=false dag_root > folder.car

and then that one is being used to make deals & seal.

I know it worked before - I was able to list the contents of that file using the ls commands and it all worked fine.

I have ~12k car files which I did that way, here's another;

$ lotus-miner pieces cid-info QmP2z7DdgLLeqBMxvcB3VJjr1wdCtLwzJZFuBw8jh5fZg5
Info for:  QmP2z7DdgLLeqBMxvcB3VJjr1wdCtLwzJZFuBw8jh5fZg5
PieceCid                                                          Offset  Size
baga6ea4seaqh77kzhlxsmetkntmnclnoxyz2arxs6bilnalydilecmfty7yx6mq  0       0

dirkmc · 2022-05-09T12:59:48Z

@RobQuistNL I'm going to do some testing today and try to reproduce this issue

RobQuistNL · 2022-05-09T13:12:06Z

I can send you some example .car files if you want to seal one of them.

Be aware that a TON of slingshot deals have been made this way (and the weird thing is that I'm positive I've seen the listing work on these deals before)

RobQuistNL · 2022-05-09T15:06:58Z

Here's a list of providerID's and retrievals that failed because of this bug;

Provider ID	Failure count by blocks missing
f010446	145
f0104671	5
f010617	184
f01105829	1
f01199430	659
f01199442	769
f01201327	819
f01207045	782
f01208862	846
f01240	1385
f01278	163
f0127896	2
f01337533	25
f01345523	32
f014768	165
f020378	585
f022352	567
f0230200	1
f02576	2
f02620	4
f030379	58
f033356	18
f03488	4
f0400920	24
f0413563	3
f0504054	18
f0694396	2
f0707721	878
f0757233	2
f07998	1
f08399	394

This is from running my evergreen bot for about 1 hour

edit: 3 hours

dirkmc · 2022-05-09T15:17:59Z

I was able to repro on our miner, I'm debugging now

dirkmc · 2022-05-09T16:02:26Z

So in my case the problem was actually that the deal failed, so it wasn't indexed correctly.

What happens if you do

lotus-miner dagstore initialize-shard <piece cid>

and then try to retrieve by payload cid?

$ lotus-miner dagstore initialize-shard baga6ea4seaqnho54pbhiwiz5iltkb3lmexcw4bwejdy7gpqx35cdnzamtlbt4ba
$ lotus client ls --provider=f01392893 --pieceCid baga6ea4seaqnho54pbhiwiz5iltkb3lmexcw4bwejdy7gpqx35cdnzamtlbt4ba QmVzK5WbFrVta19J3nbiMTE46XsBFCWSYFKzNHsEyo9NXJ

Note that in this example you provided it may not be working because you're trying to retrieve by piece cid (instead of payload cid):

$ lotus client ls --provider=f01392893 baga6ea4seaqkuhx7eqsfvkldaefzbc3y5xrrdjxcgr7tu7u4ak2cnyyej4zegny

(initialize-all outputs piece cids, not payload cids)

RobQuistNL · 2022-05-09T16:10:14Z

This is exactly what I've stated before (~~can't find my exact post, but think it was in slack~~ https://filecoinproject.slack.com/archives/C015G3B5RQE/p1651602740172029): There are a lot of deals that my markets node has marked as "Errored", but they are actually active, proving and on-chain.

#260
#185

RobQuistNL · 2022-05-09T16:14:29Z

master@daemon:~$ lotus-miner dagstore initialize-shard baga6ea4seaqh77kzhlxsmetkntmnclnoxyz2arxs6bilnalydilecmfty7yx6mq
ERROR: failed to get shard info: shard not found

master@daemon:~$ lotus-miner dagstore initialize-shard QmP2z7DdgLLeqBMxvcB3VJjr1wdCtLwzJZFuBw8jh5fZg5
ERROR: failed to get shard info: shard not found

master@daemon:~$ lotus-miner dagstore initialize-shard bafybeiakkuvje3ztt7rd2u6fb743clyyk44crjd2bpuxrmgmhz7a6uxori
ERROR: failed to get shard info: shard not found

This is carfile # 23:
root dag: QmP2z7DdgLLeqBMxvcB3VJjr1wdCtLwzJZFuBw8jh5fZg5
piece cid: baga6ea4seaqh77kzhlxsmetkntmnclnoxyz2arxs6bilnalydilecmfty7yx6mq
base32 datacid: bafybeiakkuvje3ztt7rd2u6fb743clyyk44crjd2bpuxrmgmhz7a6uxori

deal on f01392893:
dealCID: bafyreiggp6lifu4prhzld6ejduddidobeie7i5dv7ug2ueldmjziqastki
dealID: 3095799
Sector ID: 118718

Deal Status:

Dec 27 23:39:48  true      bafyreiggp6lifu4prhzld6ejduddidobeie7i5dv7ug2ueldmjziqastki  3095799  StorageDealError           f3w436qzcbnzpngwwkslethcbsnt7rjdslf25mk7ufoo27gmf4uxt3pa3yn5zilzaqwqduew7pqmu5v2nypuiq  32GiB   0 FIL              1443479                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1410528): failed to look up deal on chain: deal 3095799 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed

Pieces info:

$ lotus-miner pieces piece-info baga6ea4seaqh77kzhlxsmetkntmnclnoxyz2arxs6bilnalydilecmfty7yx6mq
Piece:  baga6ea4seaqh77kzhlxsmetkntmnclnoxyz2arxs6bilnalydilecmfty7yx6mq
Deals:
DealID   SectorID  Length       Offset
3095799  118718    34359738368  0

CID info:

$ lotus-miner pieces cid-info QmP2z7DdgLLeqBMxvcB3VJjr1wdCtLwzJZFuBw8jh5fZg5
Info for:  QmP2z7DdgLLeqBMxvcB3VJjr1wdCtLwzJZFuBw8jh5fZg5
PieceCid                                                          Offset  Size
baga6ea4seaqh77kzhlxsmetkntmnclnoxyz2arxs6bilnalydilecmfty7yx6mq  0       0

RobQuistNL · 2022-05-09T16:17:36Z

But yeah, if the index provider only receives "active" deals from the deal subsystem, then thats not going to work because that 2-year-old bug is snowballing through to the indexing subsystem.

RobQuistNL · 2022-05-09T16:21:58Z

More related bugs:
filecoin-project/lotus#5126
filecoin-project/lotus#7385

RobQuistNL · 2022-05-09T17:25:10Z

$ lotus state get-deal 3095799
{
  "Proposal": {
    "PieceCID": {
      "/": "baga6ea4seaqh77kzhlxsmetkntmnclnoxyz2arxs6bilnalydilecmfty7yx6mq"
    },
    "PieceSize": 34359738368,
    "VerifiedDeal": true,
    "Client": "f01615359",
    "Provider": "f01392893",
    "Label": "QmP2z7DdgLLeqBMxvcB3VJjr1wdCtLwzJZFuBw8jh5fZg5",
    "StartEpoch": 1434439,
    "EndEpoch": 2877918,
    "StoragePricePerEpoch": "0",
    "ProviderCollateral": "4975603048002379",
    "ClientCollateral": "0"
  },
  "State": {
    "SectorStartEpoch": 1412359,
    "LastUpdatedEpoch": 1791159,
    "SlashEpoch": -1
  }
}

dirkmc · 2022-05-09T17:34:32Z

I'm not so familiar with this part of the code, I'm looking through how it works as we go.

So it seems like there are a couple of stages that a deal goes through with the DAG store:

When the deal has been handed off to the sealer, it is Registered
This means the DAG store knows about the deal (knows its piece CID)
In the background, the deal is Initialized
This means the DAG store gets the deal data from the sealer, and builds an index of its cids

When you run initialize-shard with the deal's piece CID, it will only work if the DAG store already knows about the deal: ie the deal's piece was Registered. The reason you're getting shard not found is because the deal wasn't correctly registered with the DAG store.

So I think we need a command that will allow you to manually register a deal with the DAG store. I will add this as a ticket to Boost and close out this ticket.

RobQuistNL · 2022-05-09T17:37:06Z

I initially opened this issue in the lotus project - shouldn't the root cause be fixed there instead of having a patch / workaround here?

dirkmc · 2022-05-09T17:38:14Z

The DAG store is part of the markets subsystem, which is now being replaced by Boost. So the fix should be in Boost.

RobQuistNL · 2022-05-09T17:40:35Z

Okay.

I'm not really sure which one are really "Errored" but its safe to say there's a lot of data that actually is available, thats not unretrievable because of this (or underlying) issues;

master@daemon:~$ lotus-miner storage-deals list | grep "StorageDealActive" | wc -l
10303
master@daemon:~$ lotus-miner storage-deals list | grep "StorageDealError" | wc -l
307270

dirkmc · 2022-05-09T17:41:46Z

Closing in favour of #509

RobQuistNL · 2022-05-09T17:45:47Z

I mean if there's a way that we can patch it with a command like this, maybe we can at least restore our dagstore to a normal state using that.

The underlaying issue still seems to exist though. Here's a recent deal thats active and probably unretrievable;

master@daemon:~$ lotus state get-deal 5816931
{
  "Proposal": {
    "PieceCID": {
      "/": "baga6ea4seaqndlukp7vphibmpxn32rjhjqt6yhw7oo3fs2ktdzkys4zmk437iiq"
    },
    "PieceSize": 34359738368,
    "VerifiedDeal": true,
    "Client": "f01817237",
    "Provider": "f01392893",
    "Label": "uAXASILL83AGD7C185VjkO5CHUFTA0gKNLpeEgCGdY3fNRbP9",
    "StartEpoch": 1813627,
    "EndEpoch": 3237918,
    "StoragePricePerEpoch": "0",
    "ProviderCollateral": "5911184172583188",
    "ClientCollateral": "0"
  },
  "State": {
    "SectorStartEpoch": 1773590,
    "LastUpdatedEpoch": -1,
    "SlashEpoch": -1
  }
}

master@daemon:~$ cat ~/storagedeals.txt | grep "5816931"
May  2 15:34:18  true   bafyreigzonrfgfx5vessznhf6nzrtw3dirzkja2milexpk7ualuwcqnkyu  5816931  StorageDealError              f1efk7kg5hgmd2r6yr4kyro24pou5hd64ua7x2iii                                               32GiB   0 FIL             1424291                                                                                                                                 error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1768288): failed to look up deal on chain: deal 5816931 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed

master@daemon:~$ lotus-miner pieces piece-info baga6ea4seaqndlukp7vphibmpxn32rjhjqt6yhw7oo3fs2ktdzkys4zmk437iiq
Piece:  baga6ea4seaqndlukp7vphibmpxn32rjhjqt6yhw7oo3fs2ktdzkys4zmk437iiq
Deals:
DealID   SectorID  Length       Offset
5816931  161954    34359738368  0

master@daemon:~$ lotus-miner pieces cid-info uAXASILL83AGD7C185VjkO5CHUFTA0gKNLpeEgCGdY3fNRbP9
Info for:  bafybeifs7toada7mfv6okwhehoiioucuydjafdjos6ciaim5mn342rnt7u
PieceCid                                                          Offset  Size
baga6ea4seaqndlukp7vphibmpxn32rjhjqt6yhw7oo3fs2ktdzkys4zmk437iiq  0       0

RobQuistNL · 2022-05-09T18:40:36Z

I can provide a list of dealCID's / dealID's that are active on chain but are marked as failed by doing some simple CLI stuff with the tools available now - is there a way I can allow these to work in my "legacy" lotus markets setup? Or do I need to wait on #509 ?

RobQuistNL · 2022-05-09T18:43:54Z

I can safely say about 70% of my deals are currently unretrievable. Looking at other providers, I feel like they have similar values.

RobQuistNL · 2022-06-13T17:02:32Z

@LexLuthr

kernelogic · 2022-07-13T05:25:28Z

Tried several different miners and different CIDs, all same error.

LexLuthr · 2022-07-13T09:02:40Z

@kernelogic This is fixed in boost if you want to give it a try. Otherwise, it should be available in upcoming lotus stable release.
We need to to add the missing pieceCID in dagstore and retrievals should work. Currently, deals containing these files have not been indexed correctly by the dagstore.

RobQuistNL · 2022-08-10T21:24:28Z

Now that 1.17.0 is finally stable;

Before:

$ lotus client ls --provider=f01392893 --pieceCid baga6ea4seaqnho54pbhiwiz5iltkb3lmexcw4bwejdy7gpqx35cdnzamtlbt4ba QmVzK5WbFrVta19J3nbiMTE46XsBFCWSYFKzNHsEyo9NXJ
ERROR: retrieve: offer error: retrieval query offer errored: failed to fetch piece to retrieve from: getting pieces for cid QmVzK5WbFrVta19J3nbiMTE46XsBFCWSYFKzNHsEyo9NXJ: getting pieces containing block QmVzK5WbFrVta19J3nbiMTE46XsBFCWSYFKzNHsEyo9NXJ: failed to lookup index for mh 122071a7aeb3eb45f3a054cd64eb683291868550e870c1cfab1b9a306c55a2ce6431, err: datastore: key not found

Then ran:

$ lotus-miner dagstore register-shard baga6ea4seaqnho54pbhiwiz5iltkb3lmexcw4bwejdy7gpqx35cdnzamtlbt4ba
Registered shard baga6ea4seaqnho54pbhiwiz5iltkb3lmexcw4bwejdy7gpqx35cdnzamtlbt4ba

Then output was:

$ lotus client ls --provider=f01392893 --pieceCid baga6ea4seaqnho54pbhiwiz5iltkb3lmexcw4bwejdy7gpqx35cdnzamtlbt4ba QmVzK5WbFrVta19J3nbiMTE46XsBFCWSYFKzNHsEyo9NXJ
Recv 0 B, Paid 0 FIL, Open (New), 1ms [1659702364201620552|0]
Recv 0 B, Paid 0 FIL, DealProposed (WaitForAcceptance), 3ms [1659702364201620552|0]
Recv 0 B, Paid 0 FIL, DealAccepted (Accepted), 22ms [1659702364201620552|0]
Recv 0 B, Paid 0 FIL, PaymentChannelSkip (Ongoing), 22ms [1659702364201620552|0]
Recv 59 B, Paid 0 FIL, BlocksReceived (Ongoing), 143ms [1659702364201620552|59]
Recv 59 B, Paid 0 FIL, AllBlocksReceived (BlocksComplete), 146ms [1659702364201620552|59]
Recv 59 B, Paid 0 FIL, Complete (CheckComplete), 146ms [1659702364201620552|59]
Recv 59 B, Paid 0 FIL, CompleteVerified (FinalizingBlockstore), 147ms [1659702364201620552|59]
Recv 59 B, Paid 0 FIL, BlockstoreFinalized (Completed), 147ms [1659702364201620552|59]

QmUwQk2EhtpqRPop7cS372Vc4VBGSsex3yRMFAG3xXHHc8 PROJ14724	17609480856

So this fixes the issue. Now to automate it :)

beck-8 · 2023-07-30T08:29:56Z

filecoin-project/dagstore#160

RobQuistNL added kind/bug Kind: Bug need/triage labels May 2, 2022

Reiers added area/markets/retrieval and removed need/triage labels May 2, 2022

Reiers added the need/analysis label May 2, 2022

jennijuju transferred this issue from filecoin-project/lotus May 4, 2022

jacobheun added this to Boost May 4, 2022

dirkmc mentioned this issue May 9, 2022

CLI command: dagstore register-shard <piece cid> #509

Closed

dirkmc closed this as completed May 9, 2022

dirkmc moved this to Done in Boost May 9, 2022

kernelogic mentioned this issue Jul 24, 2022

[DataCap Application] Slingshot NEXRAD dataset for team Kernelogic filecoin-project/filecoin-plus-large-datasets#398

Closed

kernelogic mentioned this issue Aug 8, 2022

Slingshot Brazil Data Cube Dataset - CabrinaHuang filecoin-project/filecoin-plus-large-datasets#419

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"datastore: key not found" / "failed to lookup index for mh" #499

"datastore: key not found" / "failed to lookup index for mh" #499

RobQuistNL commented May 2, 2022

Reiers commented May 2, 2022

RobQuistNL commented May 2, 2022

cryptowhizzard commented May 3, 2022

brendalee commented May 5, 2022

RobQuistNL commented May 5, 2022

RobQuistNL commented May 5, 2022 •

edited

Loading

dirkmc commented May 5, 2022

RobQuistNL commented May 9, 2022

dirkmc commented May 9, 2022

RobQuistNL commented May 9, 2022

RobQuistNL commented May 9, 2022 •

edited

Loading

dirkmc commented May 9, 2022

dirkmc commented May 9, 2022 •

edited

Loading

RobQuistNL commented May 9, 2022 •

edited

Loading

RobQuistNL commented May 9, 2022 •

edited

Loading

RobQuistNL commented May 9, 2022

RobQuistNL commented May 9, 2022

RobQuistNL commented May 9, 2022

dirkmc commented May 9, 2022 •

edited

Loading

RobQuistNL commented May 9, 2022

dirkmc commented May 9, 2022

RobQuistNL commented May 9, 2022

dirkmc commented May 9, 2022

RobQuistNL commented May 9, 2022

RobQuistNL commented May 9, 2022

RobQuistNL commented May 9, 2022

RobQuistNL commented Jun 13, 2022

kernelogic commented Jul 13, 2022

LexLuthr commented Jul 13, 2022

RobQuistNL commented Aug 10, 2022

beck-8 commented Jul 30, 2023

"datastore: key not found" / "failed to lookup index for mh" #499

"datastore: key not found" / "failed to lookup index for mh" #499

Comments

RobQuistNL commented May 2, 2022

Checklist

Lotus component

Lotus Version

Describe the Bug

Logging Information

Repo Steps

Reiers commented May 2, 2022

RobQuistNL commented May 2, 2022

cryptowhizzard commented May 3, 2022

brendalee commented May 5, 2022

RobQuistNL commented May 5, 2022

RobQuistNL commented May 5, 2022 • edited Loading

dirkmc commented May 5, 2022

RobQuistNL commented May 9, 2022

dirkmc commented May 9, 2022

RobQuistNL commented May 9, 2022

RobQuistNL commented May 9, 2022 • edited Loading

dirkmc commented May 9, 2022

dirkmc commented May 9, 2022 • edited Loading

RobQuistNL commented May 9, 2022 • edited Loading

RobQuistNL commented May 9, 2022 • edited Loading

RobQuistNL commented May 9, 2022

RobQuistNL commented May 9, 2022

RobQuistNL commented May 9, 2022

dirkmc commented May 9, 2022 • edited Loading

RobQuistNL commented May 9, 2022

dirkmc commented May 9, 2022

RobQuistNL commented May 9, 2022

dirkmc commented May 9, 2022

RobQuistNL commented May 9, 2022

RobQuistNL commented May 9, 2022

RobQuistNL commented May 9, 2022

RobQuistNL commented Jun 13, 2022

kernelogic commented Jul 13, 2022

LexLuthr commented Jul 13, 2022

RobQuistNL commented Aug 10, 2022

beck-8 commented Jul 30, 2023

RobQuistNL commented May 5, 2022 •

edited

Loading

RobQuistNL commented May 9, 2022 •

edited

Loading

dirkmc commented May 9, 2022 •

edited

Loading

RobQuistNL commented May 9, 2022 •

edited

Loading

RobQuistNL commented May 9, 2022 •

edited

Loading

dirkmc commented May 9, 2022 •

edited

Loading