Replies: 2 comments 3 replies
-
|
I have a question a bit unrelated to your project: Do you know if it is possible somehow to add a huge file in my regular file system to IPFS in such a way that IPFS doesn't duplicate file's content? So if I have 82 GB local file then IPFS doesn't copy 82 GB over into its block-system. I can imagine that instead of copying whole file IPFS can slice this file into 256 KB blocks and just store light references to these blocks, like file_path + start/end_byte_number. Is there such feature in IPFS? If there is no such feature in IPFS itself maybe at least your utils are capable of doing such thing? The main idea for me is to share large data on my local file system without doing duplicating. Of cause I can use your filesystem util to map IPFS files into regular file system, but I prefer to keep my valuable and sensitive data inside regular file system as a main storage. Also there is another reason not to move everything into IPFS - because some of my large files I just want to share temporarily, meaning that I don't need them in IPFS at all, I want to share them with my friends for a while, after that it is not needed in IPFS. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @polkovnikov I don't think anyone has made efforts to improve it to the point where it can be considered stable / not experimental. Unfortunately. But it would be nice if someone improved that feature, and maybe the same functionality I had in the shell extension could be added to the IPFS desktop project when |
Beta Was this translation helpful? Give feedback.
-
|
Thanks a lot @djdv , your suggestion worked for me, did following on my Windows:
Both |
Beta Was this translation helpful? Give feedback.
-
|
@djdv As you're very familiar (unlike me) with IPFS can you please also suggest is it possible somehow to use IPFS as a proxy of torrents technology? Meaning if there is any proxy that connects two worlds - torrents and IPFS? They both seem to be very related technologies. For example if I have Then when somebody reads a small part of torrented file from IPFS (for example through your mounted disk magic) some underlying code of IPFS node tries to download this small portion from Torrent network. It looks like a very interesting feature because IPFS and Torrent technologies look very similar to each other. Hence I could imagine that there were some efforts of connecting them two. |
Beta Was this translation helpful? Give feedback.
-
|
@polkovnikov I know that this has been talked about before and I assume will be possible eventually, but I don't know the current status of it. |
Beta Was this translation helpful? Give feedback.
-
In response to: ipfs/kubo#5504 (comment)
Looks like I messed up the path translations somewhere.
I recently tested and ported over the code that handles the
cmdspkg stuff, and remember encountering this as well in my Linux environments. (I primarily develop on Windows so Unix multiaddrs were getting messed with a lot in the old versions. Some of the code became redundant after a patch was pulled ingo-multiaddr/netas well.)The standalone binary is currently pending reviews here: https://github.com/djdv/go-filesystem-utils/pulls
And there's some dependency issues that need to be patched as well before the CI will clear on Linux.
(I'll be filling issues in the repo later)
But after those are finished, the next section of code to be brought over and tested will be the
mountcommand. Specifically meaning; Read-only IPFS and IPNS via Fuse. With the rest of read-only things to follow. Then write support afterwards.This is how the standard
ipfs mountimplementation handles mounting IPNS. But that's the existing code.IIRC "local" is just a hardcoded name, and the system uses the MFS api to handle writes and publishes.
There may be some issue in the mount code, or the
mfspkg.In the new implementation, I do a local-only publish to the node every write, and intend to add a full publish after an idle period.
Even if writes are continuous forever and prevent the mount doing a full publish. The
daemon/node itself should eventually decide to republish its keys (irrelevant ofmount, but with the new local value).It sounds similar to how you described but the new implementation still needs to be refactored and stabilized.
I'll have to make sure to look into making tests that mock IPNS and use the file system, then validate publishes are happening when they should be. *At least to some basic degree for now.
Some other differences are that the new implementation distinguishes "ipns" and "keyfs" in its arguments.
So "keyfs" will contain a root that lists the keys as their names, the same as
ipfs key list -lwould;selfincluded.And handles keys that point to supported types (unixfs directories, files, links ...).
The existing implementation handles unixfs directories only IIRC.
The new implementation for mounting "ipns" specifically, has an empty root. But I might change that since it's inconsistent with how
ipfs mounthandles it.Version control sounds interesting in this way.
I'm sure since it's mountable, you could use conventional file system backup methods to handle it.
Like rsync, etc. but Venti also comes to mind here.
In regards to the config policies; I think it's viable to handle such policies, and associate them with keys/whatever.
The implementation that comes to mind is just automatically creating some kind of subdirectory within the key, like
/.ipns/versionssimilar to how ZFS exposes/.zfs/snapshot. And scheduling the tasks / adjusting sync times / using a name other than.ipns, etc. in the arguments.For reference, I've put some consideration into how setting values are being retrieved, and have notes on a
CmdsParameterSetset here with ideas on extending it later in a way that shouldn't cause too much hassle. It's a bit verbose, but I don't think the declarations can be avoided either way.Same. Doing what I can to push it forward a bit. I'd really like to be able to start using the system more seriously/frequently. With things running on top of it as well. Or sourced from it at least (Without exhausting machine resources).
No worries, the conversation about it alone is super helpful!
But also, I intend for code from any language to be able to interface with the file system service in some ways.
As-is,
fs service --api=...is implemented, and exposes an API in the same wayipfs daemon --api=...does. (over whatever multiaddr socket thecmdspkg supports, TCP, UDS, etc.)So C++ should be able to interface with it the same way, but with our own set of commands.
The rest of the program in itself is simply going to be a client of its own service. An implementation
fs listis here for reference. But is probably going to change asfs mountis refactored.As an aside, my own distant plan is to adapt a very old Windows shell extensions I wrote.
Making the user shell more IPFS aware. Allowing for things like "copy cid" in the context menu when viewed from explorer. (Will have to modernize this in C++, like it or not due to the OS API)
I found this extremely useful for linking files to people quickly, and would like to get that functionality back, but with the modern APIs.
The plan is to have the explorer's context call "/api/list" with encoding like JSON, and just use results.
You can imagine seeing "IPFS->publish" in the context menu while (only) inside an IPNS mountpoint. "IPFS>Unpin", etc.
By communicating with both the
fsandipfsdaemons, in some external client written in another language.I'm curious what other people might do with the API, and could always extend it if necessary.
Regards :^)
I appreciate it. Thanks for trying it out and thinking things through with me.
Beta Was this translation helpful? Give feedback.
All reactions