Ability for SP/client to mark deal as private (i.e. not publicly retrievable)

[honghaoq]

The Ask
SPs that are storing private data for clients have reported the need that these data should neither be publicly retrievable nor indexed. Specifically, they need a a place in deal to specify data as ‘private’ or ‘not publicly retrievable’, and nor would these data be published to indexer.

Use Case
Example scenarios that SP reported:

1. PiKNiK mentioned Shoah foundation does not expect any public retrieval of their data, nor should they be announced by indexer.
   PiKNiK’s current means of protecting the Shoah data is as follows:

• Encrypt each file (often more than 1 TiB) using a unique key
• Shard larger files (and internally index and map those shards) into smaller files and distribute to different deals across multiple SPs (no single sector or SP has a complete file)
• Require any SP participating to have a dedicated lotus-miner only for Shoah data and disable all retrievals on that miner
• If a retrieval is needed, then PiKNiK put in an ACL to allow retrieval from that particular client during the retrieval process, otherwise retrievals are disallowed

2. Factor8 reported business use cases that needs custom retrievability on a case by case basis (but not publicly retrievable), since client could ask for it back when needed. But they do not want the data be retrievable by others or be indexed. Today they maintain private index for these data.

[honghaoq]

It could be adding a config on boost market and/or passing ACL onchain so programs like slingshot can check and determine what data are retrievable. @dirkmc @willscott @brendalee @jacobheun thoughts?

[dirkmc]

It sounds like we should add a mechanism to allow SPs to prevent indexing of particular deals.

For access control of retrievals, the solution depends to some extent on which retrieval protocol is being used: graphsync vs bitswap vs HTTP.

For example for HTTP the SP could give the client an auth key, and put a web server in front of booster-http that checks the auth key against the piece id in the request.
For graphsync we have the retrieval deal filter and for bitswap we are going to add a filtering mechanism.

[honghaoq]

Agree on accessing by client IDs specified by SP, this is how Piknik handles it today, they put in an ACL to only allow retrieval from that particular client during the retrieval process. @dirkmc let us know what you think - happy to gather more inputs if needed

[LaurenSpiegel]

This seems like a good opportunity to think comprehensively about how ACL's should work. @honghaoq is going to put together some ideas and we can discuss.

[LexLuthr]

As per the discussion with PikNik yesterday, it makes most sense to allow client to chose whether they want to make deal retrievable and index it or not. This removes any overhead from SP side who would prefer to focus on running the miner rather than dealing with ACLs for individual clients.
We must remember that not all SPs are very technical and these are just infra folks running miners. We should avoid putting any overhead wherever possible.

[willscott]

• The SP is presumably going to need a way to mark a deal as 'not retrievable' anyway - for instance if they get abuse complaints about content in it.
• We could add additional ACLs / into the label field of the existing client deal proposal
• Would be good to have this fairly constrained initially: i would propose having 'public' and 'available only to me' as the initially supported modes, rather than a full ACL

[honghaoq]

The proposal of marking data as 'public' or 'available only to me' is generally approved by community SPs (https://filecoinproject.slack.com/archives/C02GQUMFQVA/p1663685653058249), I added this to bedrock Q4 plan. Next step is to agree on whether we need to post it onchain, would open a FIP discussion for it

[flyworker]

will the private data be eligible for Fil+?

[honghaoq]

It's definitely possible to be considered for Fil+, whether it is eligible would be determined by the Fil+ program on case by case basis I believe. Data program team (PM Deep Kapur) would be the right person to start with

[flyworker]

Shoah foundation is part of the Fil+ and not retrievable. It obviously becomes a special case, I assume lots of other projects will refer this case like, why they can and we cannot?

[BobbyChoii]

Private data that does not support retrieval clearly breaks the most basic rule of Filecoin. Why can Shoah Foundation be an exception?

[honghaoq]

my understanding is Fil+ program could allow cases for private enterprise data, versus slingshot program is the one that actually require all deals to be public data. There is an ongoing effort for E-Fil+ that is specifically built for enterprise Fil+ deals (Kevin Zakorchemny is driving that), so I think it's not an exception to have private data in Fil+. That said, I am not super familiar with the context of Shoah's case in Fil+, data program owner Deep Kapur would probably have the knowledge on it

[TorfinnOlsen]

⚠️ I have added comment to the open github issue which describes the course of action we've chosen to take in order to accomplish an MVP which has minimal impact while accomplishing additional configurability in boost over deal announcements.

• Capability to skip deals for indexing: need config to mark content as not retrievable in market #689 (comment)

We perceive this as the simplest change we can make in order to enable SP's to set the announce status, without impacting retrieval processes. Future efforts will address the broader goal of an access list service.

⚡ tl;dr Request to add a flag to boost to allow storage clients to elect to not announce deal data to IPNI. Default behavior will be to announce just as it is today.

🕐 When is a deal announced to the indexer?

• When a deal is made, if AnnounceToIPNI=True or not set at all, CID’s announced to indexer.

• If AnnounceToIPNI=False, no announcement.

🔧 How will this be implemented?

• Currently, we are on Storage Deal Protocol v1.2 (/fil/storage/mk/1.2.0) where every deal is announced.

• This new proposal is for Storage Deal Protocol v1.2.1 (/fil/storage/mk/1.2.1)

• Boost will be updated to recognize and store as metadata this new flag.

🥡 How does this impact whether the data may be retrieved?

• It doesn’t. No impact. ACL’s must be set separately. This just relates to IPNI.

🛃 How can a client change the announce status?

• No change currently permitted. The decision is made at the time the deal is made.

👴 What about legacy deals?

• They stay announced as they are today.

🍴 Related Github Discussion

• Ability for SP/client to mark deal as private (i.e. not publicly retrievable) #728

👋 Relegated github issues/discussions being closed as a result of this update

• Scope default indexing to only FIL+ deals with FastRetrieval enabled notary-governance#666

• Capability to skip deals for Indexing ipni/storetheindex#635

⌛ Timeline for implementation

• Current scope of effort is in the process of being analyzed and assessed by the boost team. Work not yet scheduled, but expected outcome is work completed early Q123.