Read/Write cache?

[mortenwp]

As I understand it gcsfuse can provide a read cache, but not a read-write cache. My use case is basically that quickly after a file has been written out, another process will read the file that was output. A workaround would be to store the file locally, before copying it to its gcfsuse mounted folder, but it would be useful if a read/write cache could move the files that has just been written to the bucket into the local cache before they are deleted, Has such a feature been discussed? A tool like rclone has such a feature already.

-Morten

[raj-prince]

@mortenwp thanks for reaching out!

I need a bit clarification of your use-case.
(a) How many mount points (gcsfuse mounting the bucket) involved in the use-case? Just wanted to understand if read and write will happen from the same mount point.
(b) Does your workload do repeated read once written or continuous write and read?
(c) If possible, could you please tell, what would the file-size in your read/write use-case?

CC: @marcoa6

[mortenwp]

My use case  is to and from the same mount point... I will have a mixture of write/read and write/repeated reads. The write/read was probably what I firstly had in mind... A process writes out data to bucket, next process reads it back in. 2. aug. 2024 10.44 skrev Prince Kumar ***@***.***>: @mortenwp thanks for reaching out! I need a bit clarification of your use-case. (a) How many mount points (gcsfuse mounting the bucket) involved in the use-case? Just wanted to understand if read and write will happen from the same mount point. (b) Does your workload do repeated read once written or continuous write and read? CC: @marcoa6 —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned Message ID: ***@***.***>

[mortenwp]

I realized that I did not reflect on the file sizes. Ideally both large and small file, but I think what hits me most is the short files. The "lag" I get on short files probably is most annoying... , (file of max a few Mb). I'm more prepared to take a hit on any large file (several Gb)... and if the larger file are not automatically injected into the ReadCahe (before you actually read them) it might be beneficial when it comes to the disk usage. Also in my use case the smaller files I tend to write/read a lot more times, than I would with the very big files. All this is very specific to what I do, so my view is not universal