What is the best practice for implementing microstream in a cluster?

[dcoracle]

Hello,

our team has been evaluating this product within a kubernetes cluster where we want to scale a helidon microservice to multiiple replica pods using a microstream file store in shared PV. I was a bit surprised when updates to one pod did not propagate to the other pods and eventually found this one section in the documentation that talks about the limitations of a shared Microstream database:

https://docs.microstream.one/manual/storage/faq/file-storage.html

My interpretation of the last paragraph is that if we wanted to use multiple pods, we also would have to create copies of the Microstream database (each pod with its own) OR we would have to create a data master microservice that acts as a gateway that the worker processes can call.

I feel that each solution kind of defeats the main advantages of microstream, which is simplicity.
1)If we replicate each database, how do we keep it in sync? Do we have to manually publish a message that all pods must consume and then update its local database? What will this mean in terms of space usage?
2)If we implement a data master, how will the other worker processes call the data master to keep it updated? Would it be through some API? or we have to create (REST/gRPC/etc) endpoints to update/refresh the data? Wouldn't this result in a big hit in performance? Also, this will require DTO mapping/conversions which microstream was supposed to help eliminate.

Can you share in more detail (via examples) what the best practice is on how to architect a microstream solution in a K8s cluster?

Thanks,
Derek

[hg-ms]

Hello Derek,

Your interpretation of the documentation is correct.
Currently Microstream does not provide the necessary features for replication and synchronization of several nodes.
We are currently working on all those features to provide Microstream – Cluster solution. But we can’t tell when this will be available and in which way it will be released.
Regarding the DTO mapping: It is possible to use Microstream as a serializer. A simplified API for that will be available with the next release.

Best regards
Your Microstream Team

[dcoracle]

Thank you for your candor. We will build the missing pieces for now and looking forward to the next release!

[ShivaniSK]

Does microstream provide the replication and synchronization features now?