Playing with IPNS

[Erotemic]

Do not merge.

PR is mainly to provoke discussion surrounding IPNS R/W support and what would need to happen to get this repo minimally working with DVC.

My understanding is that if this had write support, it could be used in DVC (even if not efficient). IPNS is one solution (although I hear DNSLink is probably better).

I'm also just playing with this to get a better feel for the fsspec code. I've mainly added doctests that demo a few ways I've tried to use the repo.

I also added an IPNSFileSystem, because the current IPFSFileSystem does not work with ipns references. The reason for this was something I found odd. The get method in the Gateway hard-coded an "/ipfs/" prefix on the path. If that wasn't there, this would work with "/ipns/" prefixed addresses, although it would be cumbersome to type: ipfs:///ipfs/<cid> or ipfs:///ipns/<name> the triple / is not desirable.

I abstracted the entire thing to just use a '/{self.protocol}/ prefix, and I added that to both "ls" and "info" methods of the filesystem itself for consistency with the "get" call of the gateway. Not sure exactly what the right design decisions are here. Thoughts?

[d70-t]

Thank you very much for this PR 🎉 !

IPNS support (or any form of write-support) would be awesome. However I didn't come up yet with a good way of implementing any of those cleanly. Help is very welcome!

Indeed, the Gateway class should probably be modified to accept (and require?) proper IPFS addresses (starting with /ipfs/... and /ipns/...) where possible and require the use of CIDs only where necessary (if any). I believe I just accidentally programmed it this way as I mostly cared about read-only when writing it. That might also help when playing with the Mutable Filesystem which could be another way to enable write-support.

I still believe the self.protocol-based approach in the filesystem class, and two separate filesystems are the way to go.
The IPFS docs state that native URLs should be ipfs://CID/... and ipns://mutable-root/.... I'd argue that we want fsspec to accept exactly those native URLs.
Note that this is in contrast to an older discussion which included interesting variants up to ipfs://ipfs/ipns/domain.name.
There are ideas of having a unified dweb: namespace (including dweb:/ipfs/CID/... and dweb:/ipns/...) but that doesn't seem to be stable yet.

Based on how fsspec works, this seems to require two separate filesystem handlers. It also seems to be useful to have both handlers based on a shared codebase.

To summarize:

• Filesystems would then accept IPFS urls (ipfs://... and ipns://...) which seems to be more compatible with the fsspec way of thinking.
• Gateways would then accept only IPFS addresses (/ipfs/... and /ipns/...) which is the canonical way of expressing references in IPFS.

I've not yet used doctests since a long time, but I like the idea and am more than happy to see those added to the code.

I also am happy to hear that you are interested in a DVC integration. My main motivation for playing with IPFS is to share research data, and DVC fits very well into this place.