Knowing the exact rows that changed in the middle tier.

[gedw99]

Here is my use case that marmot might be able to do …

marmot is CDC, so it knows what changed on which node and keeps the DB in sync.

the next logical thing we can do is leverage that information to update systems above the DB that want to react to those changes in a flexible way for manny use cases.

I have a feeling that exposing that might be highly valuable to certain patterns like mine.. here goes :

I have 5 servers in 5 locations around the world , with marmot and SQLite running in a single docker.

I want to update a htmx gui using a golang server and so when a data row changes in SQLite, I want to generate a new html fragment and send it to the browser over SSE. I only want to sent it Isf the user is connected . If not then I would instead create a notificstion( email or other user notification channel ).

So essentially I want marmot to send the data changes to all of my golang servers . They all will check if the client is connected to them ( for example if the user is in London then the Amsterdam server will be their nearest server ). We can assume that’s all magically done by geo routing over Anycast of fly.io.

Then the Amsterdam server ( and no other servers ) will process that data change payload coming from Marmot.

The golang server will see what rows are effected and then do a sql query to build the html . On delete it will send a removal to the browser. On upsert, it will send new html to the client to be replaced. On Create, it will send html to the browser to be new html in their view.

the golang server is responsible for knowing the context of the the user / browser. In other words, it must remember ( in memory or NATS ) that user X is currently looking at page Y and that page Y is using tables A ( rows 1000 to 1100 in terms of pagination ).

So the golang server is responsible for the subscriptions mapping of: users —> pages —> data rows. Marmot is not responsible.

marmot just needs to tell ONLY 1 server that these rows changed, and it’s your problem to do what you want with this data.

My understanding is that NATS Jetstream can do this using https://nats.io/blog/new-per-subject-discard-policy/

This is something I can built and open source if anyone is interested . I think it’s a very common pattern for almost all systems today that need to push updates based on when data changes in the DB.

For now, I just wanted to raise this for discussion and work out the design and technical aspects.

Marmot feels like the perfect way to do this because of NATS and the need for de-duplication which nats can provide.