Plan a transition of unixfs to go-ipld-prime

[willscott]

unixfs constitutes a major interpretation of IPFS data into concrete interpretation.
It currently makes use of go-merkeldag, which exposes a primary dagservice interface.
In addition to this interface, elements of unixfs have made extensive use of the concrete protonode implementation used by merkeldag (example)

It would be valuable to have this code transitioned to make use of a nascent DagStore providing similar functionality on go-ipld-prime. This will help to allow for continued performant and clean use of unixfs within the context of ipld-prime backed data fetching.

• should FSNode continue to use the protobuf-generated struct implementation, or should we generate an equivalent ipld-prime node representation of that data?
• enumerate the interfaces (e.g. a quick glance shows public methods like NewShard make use of a dag service) that would ideally transition from reliance on ipld-format/go-merkeldag to prime equivalents (maybe you haves started this, @gammazero ?)
• Find consumers within the IPFS org that would need to update for this, make sure that's reasonable, and come up with a work plan (ordering of package updates / releases) that is mutually agreeable.

cc @aschmahmann @Stebalien @gammazero

[Stebalien]

should FSNode continue to use the protobuf-generated struct implementation, or should we generate an equivalent ipld-prime node representation of that data?

I'd actually rather write custom code. We need to guarantee a somewhat funky encoding order anyways.

enumerate the interfaces (e.g. a quick glance shows public methods like NewShard make use of a dag service) that would ideally transition from reliance on ipld-format/go-merkeldag to prime equivalents (maybe you haves started this, @gammazero ?)

We'll likely have to throw away almost all of the current interfaces.

[willscott]

We'll likely have to throw away almost all of the current interfaces.

I guess the main question is what things do we care about stability on / are problematic to break / should continue to be supported. If we have the top level CLI functionality remain the same are we okay? Would you suggest making a separate implementation for the CLI to use and have a way for consumers to use this implementation with the go-ipld-legacy shim until they're ready to transition?

[Stebalien]

The top-level CLI should continue to work. In terms of this library, I'd actually just crate a new one (maybe go-ipfs-fs?) and merge in everything you need (likely pieces of go-ipfs-chunkers, go-unixfs, go-ipfs-files, and go-merkledag).

If we want to stick with a single repo, also feel free to go with a v2.

[gammazero]

@willscott I have stayed away from making changes to unixfs so far, but need to so was hoping to know what we want to do with unixfs very soon.

[willscott]

Things building on UnixFS we're aware of:

• CLI & Core API
• MFS
• Gateway

[Jorropo]

This repository is no longer maintained and has been copied over to Boxo. In an effort to avoid noise and crippling in the Boxo repo from the weight of issues of the past, we are closing most issues and PRs in this repo. Please feel free to open a new issue in Boxo (and reference this issue) if resolving this issue is still critical for unblocking or improving your usecase.

You can learn more in the FAQs for the Boxo repo copying/consolidation effort.

---

IMO redundent after ipfs/boxo#224 ping me if you want to reopen and move this anyway.