Snapshot metadata

[placer14]

In order to understand their contents before downloading the whole snapshot, as a consumer of these snapshots I would like separate metadata published for these snapshots such that I can programmatically parse and understand the snapshot.

Details to include:

• creation date,
• lotus version,
• epochs included
• (maybe) metadata version, so we can include additional fields later to be parsed conditioned on this version.

If you expect one snapshot to differ from another in ways other than the epochs covered (like the snapshot has a different finality of epochs, or manages the data in a different way) it would be useful to include some structure to capture these details.

[frrist]

Additionally, I'd like the following pieces of metadata:

1. Snapshot Size in Bytes
2. Root Tipset of Snapshot (largest height)
   • this is already in the CAR file header, but I'd I don't want to download the entire car file to get this info
3. Last TipSet in Snapshot (smallest height)

[frrist]

From our in-person conversation if I want 3. someone (I volunteer) will need to add a new a new method:

ChainExportRange(from, to types.TipSetKey) (<-chan []byte, error)

or similar to Lotus.

[travisperson]

This is what I think I can get without having to wait for a new lotus release

Property	Type	Description
CreatedAt	time.Time	Timestamp when snapshot export starts
Export.TipSet	types.TipSet	Tipset used on call to Filecoin.ChainExport
Export.Height	types.ChainEpoch	Height of TipSet
Export.Timestamp	uint64	Unix Epoch for TipSet
Export.HeightDelta	types.ChainEpoch	Value used in call to Filecoin.ChainExport for staterootsIncluded
LotusVersion	string	Version of lotus software used to create export
Size	int64	Size of car file in bytes

We can then later add a new property Export.LastTipSet, Export.LastHeight.

^ input welcome on naming / structure

[placer14]

• I like including a unit where it's unclear. size_bytes
• If you could use the HeightDelta to accurately show <epochFirst> - <epochLast> (inclusive) for included states, that would reduce off-by-one confusion.

Suggestions:

• createdAt
• tipsetID (I assume tipset is what is at epochLast)
• tipsetEpoch
• tipsetUnix
• epochStart
• epochEnd
• sizeBytes
• lotusVersion
• metadataVersion (always 0 until the meta properties change)

Either Snakecase or camelcase is okay w me. Camelcase is my personal pref.

[frrist]

If you could use the HeightDelta to accurately show - (inclusive) for included states, that would reduce off-by-one confusion.

This isn't possible with the current lotus API implementation due to null rounds, which is why I suggested: #16 (comment)

[travisperson]

Picking out the last TipSet that is included in the export isn't easy to do accurately because it's an implementation detail of ChainExport.

In this code b is the current TipSet being exported, ts is the provided TipSet to the ChainExport method and inclRecentRoots is the provided nroots to the ChainExport method.

b.Height > ts.Height()-inclRecentRoots

https://github.com/filecoin-project/lotus/blob/v1.15.2/chain/store/snapshot.go#L128

The last TipSet in the export depends on the operand used in the comparison as well as the null arounds around ts.Height()-inclRecentRoots.

To do this accurately and consistently we need a new export method that allows us to specify the last TipSet. Otherwise we can only really say it doesn't included states before some height, and even that is subject to off by one errors if lotus changes the operand.

For archival purposes of chain state we need the new method.

[placer14]

Okay, I understand that including the firstHeight would not be accurate.

When I consume these snapshots, is it reasonable for me to infer the latest possible firstHeight to be included in any snapshot would be lastHeight - heightDelta?

Additionally, since the snapshots will be performed (I assume) at regular intervals with coverage over every epoch, that worst case there would sometimes be duplicated state exported in two different exports (until snapshots start consuming this new Export method).

[f8-ptrk]

• adding when x+1 will be created/available in the meta data of x will be beneficial
• the exact params used to export the snapshot would be a nice addition