Large Transfer Support, Differential transfers.

[petersilva]

in sarracenia v2, there was support for transfers of partitioned files. That is large files could be split into chunks, and each chunk would be transferred independently, and the chunks re-assembled at a destination.

This allowed:

• parallelism in the transfer of individual large files (otherwise every file is transferred with a single thread.)
• this parallelism also allows better utilization of large tcp connections as often single stream transfers are limiting.
• There is no largest supportable file size, since files will be divided into blocks and transferred progressively.
• Avoids the "capybara in a python" problem of a large file filling up all disk space on intervening nodes, especially when using multi-node transfer arrays, one node fills up, and then empties. with blocks, the transfers will be on multiple nodes, and once a block is done, the file may be removed.

Did not support:

• supports incremental/progressive transfers for files that aren't complete. Example: a DNA sequencer writes out the sequence over a day or so, at a rate of a few MB/hour. at the end of the day the file could be a gigabyte or two. If you wait until the file is ready, you then need to consider the speed to transfer a gigabyte, whereas if you trigger incremental transfers throughout the period, one will only need to copy the last parts of the file after the file writes are done.

In v2:

• the partitioned file functionality was never used
• there were limitations known, such as the Gene sequencer case
• the code was pretty complicated.

So partitioned transfers were dropped in order to make it easier to do the heavy re-factoring needed to produce sr3. This brought rise to #396 which is kind of the basis for this discussion.
Then there is the question of whether a bespoke sort of differential copy mechanism is needed/helpful vs. integrating existing mature tech. #764 ... #765 ... but those only work point to point, not multi-hop.

If we need multi-hop, then something custom is needed. Perhaps validate whether the multi-hop is needed, and also look at using the other technologies might be simpler easier to apply to certain cases.

like just running rsync from a cron job on the gene sequencer... over the two day write period could get the data into the HPC DC. without any need for Sarracenia, and still getting "high speed" transfer in that it has file generation overlap with transfer. otoh, it's a lot more expensive because of the repeated checksum calculations on both ends... some amortization might be possible with custom tooling.

[petersilva]

Why bring this up? because when talking with science users outside meteorological real-time, one of the main concerns is large file transfers... like 100's of MB, or GB at a time. Enabling efficient large file transfer is a problem that our non-met clients have frequently. If we had a good answer to this, it would help a wide variety of clients.

[petersilva]

Additional discussion here #779