Introduce tosfs to enhance the volcengine tos storage

[yanghua]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template :) and fill in all the required fields.

Content

Hi community,

I'm very glad to see that Dify is becoming more and more popular in the field of LLM. It's an excellent project.

I noticed that dify had integrated the object storage of mainstream cloud vendors, including the TOS of Volcano Engine..

I have a suggestion to introduce TOSFS to replace the implementation based on the TOS Python SDK, so that our programming habits are more like using the file system access behavior with file semantics (just like the current implementation of local storage that we can see).

Please allow me to briefly introduce the differences between TOSFS and the TOS Python SDK. The TOS Python SDK is an object-semantic SDK open-sourced by Volcano Engine. TOSFS is a file-semantic SDK encapsulated by Volcano Engine based on the TOS Python SDK. It has implemented the widely popular fsspec interface in the Python field and has been accepted by the fsspec official organization.

Compared with the TOS Python SDK, TOSFS has the following advantages:

• File semantics VS object semantics: The file semantics are more in line with the usage and understanding habits of mainstream developers than the object semantics. It is simple, easy to understand, and easy to maintain.
• It has been implemented and refined through the practice of multiple LLM customers.
• Solid stability (fine-grained judgment on response codes for TOS services).
• Outstanding compatibility (cross-validation completed on the version matrix of four Python versions and two fsspec versions).
• TOS HNS (Hierarchical NameSpace) Bucket support (in adaptation and verification).

Hope to hear thoughts and feedback from the community.

If the community is willing to accept it, I'll be glad to take on this job.

[yanghua]

@takatost What do you think about this idea?

[bowenliang123]

cc @takatost @JohnJyong @laipz8200 @crazywoola

[laipz8200]

SGTM

[yanghua]

@laipz8200 Thanks, Will open a PR for this work.