State of the project and future plans

[InspiringCode]

Hi,

this project looks really great and I think something like this is greatly needed, especially for CQRS related projects.

Is this project still actively developed? What are the plans for the foreseeable future? Do you intend to continue development?

I would really love to use this DB in my projects but I am a little hesitant to do so since it seems to have quite a small user base at the moment unfortunately.

I would be really grateful for some infos on this!

[rofr]

this project looks really great and I think something like this is greatly needed, especially for CQRS related projects

I totally agree! But many are sceptical or just don't get it. Or it's a bad idea, poorly implemented or poorly communicated. There are not many nuget downloads, likes or forks. I've had contact with maybe 10-20 people that have evaluated, often an enthusiastic developer that in the end get a NO from management.

We're running both Memstate and the predecessor OrigoDB in production. OrigoDB (Net Framework 4.x only) is trustworthy but needs a package update, haven't touched the code in years. Memstate is a work in progress. I had to switch to some urgent tasks while in the middle of a major internal redesign, hopefully I can pick up from where I left when things calm down.

I did a POC using memstate in November using SqlStreamStore with SqlServer as a backing store. We are doing GeoLocation lookups and triangulation against an evolving model of well known locations. That code is now running in prod!

The next application will be an Organisation model with a hierarchical organisation tree, resources and a resource based permissions/policies. We want this in memory to save database roundtrips just for resource based authorization and memstate so we can have a synchronized replica on each application server.

[InspiringCode]

Sounds good, thank you very much for your elaborate answer! I really hope that you continue that project. I think it is just the perfect answer for how to do read models in an event sourced CQRS architecture. I couldn't find anything similar to memstate, is there something similar?

[srid]

I'm using this library in my own project, and I agree that it deserves more mindshare than it currently has. I posted this to r/fsharp, and hope @sergey-tihon will consider including it the next F# weekly. :-)

[michaelnero]

I've been pouring over the code recently. This is a really cool project!

Do you any thoughts around modifying how the reader/writer lock works, where the entire model is locked for writes? I completely understand why you went this direction. It works (and works well) and it's simple, but this does mean that read and write performance is coupled. And from what I can tell in the code, reads and writes are treated equally (no priority is given to one or the other).

Of course, you could just do heavy writes during off-hours, but I'm curious if you've had any thoughts around decoupling (to an extent, of course) read and write performance so that we can have both sustained reads and sustained writes. If so, I'd love to help if you're looking for it.

[goblinfactory]

Hi Michael
I'll let Robert reply in detail, but just wanted to drop a quick message to say that reads are run in parallel without locking, the only synchronisation is that the writes are held back until a series of reads are completed all in parallel, then all writes are paused while writing happens. Robert can let you know the actual details. It sounds like you have a concern about a problem that probably may not exist? I agree the performance is linked, but not necessarily coupled, i.e. the reads could be 10x or 100x of the writes? to be confirmed? (just my ideas) Robert to confirm...?

[michaelnero]

Hello! I like your name "goblinfactory", by the way. It's funny. :-)

I was referring to the Kernel class and the use of ReaderWriterLockSlim there. Yes, reads are in parallel, but blocked by writes, which are serialized. So, I was mainly thinking of this from the perspective of a use case where I have fairly frequent and sustained writes during peak hours, as well as reads. In this situation, one affects the other. Am I thinking about this correctly?

I haven't actually tested the performance in this situation... I imagine that it's probably just already better than read/writing to a DB, but was mostly curious to see if we had any thoughts around this.

[goblinfactory]

heheh! txs .. used to have some "goblins' working for me, a long time ago.
Fairly frequent writes with matching reads is the un-common situation; would you mind giving some more background to your use case? (oooh, just saw Robert reply) ...

[rofr]

Hi Michael, nice to get some attention around here, that could inspire things to start moving again :)

I don't think the ReaderWriterLockSlim will starve out readers when there is constant pressure from writes. But even if there is high pressure there is always a small window of time to squeeze in a few million queries while waiting for the commands to flush to disk or return from EventStore / RDBMS depending on the type of underlying storage. It is of course important that the commands execute quickly, it is only during execution that the lock is held.

But there is a pattern that can be used to never block reads. If the entire model is immutable then it's always safe to read. Commands need to run in series and return the next state of the model. An added bonus is if a command fails with an exception there is no risk of a semi failed transaction, so atomicity is now a guarantee of the system instead of a developer responsibility.

Memstate is a redesign of OrigoDB which targets .NET Framework which had this behavior as an option. I considered making it the default when starting out with memstate but it felt a bit too bold and opinionated. Now that we have c# records and the with keyword, it makes much more sense.

I haven't actually tested the performance in this situation... I imagine that it's probably just already better than read/writing to a DB, but was mostly curious to see if we had any thoughts around this.

The benchmarks using EventStore run at near 100K sustained commands per second (eventstore maximum) and millions of queries per second in parallel with the commands at full throttle! This was with a key/value model using simple reads and writes but still pretty impressive. At this rate of commands the problem becomes dealing with billions of journal records. Storage, storage cost, time to replay during startup, etc...

[rofr]

So with high volume writes I think snapshotting becomes a necessary feature. Snapshots are still TBD
But for my use cases so far journal size and replay time hasn't been an issue.

[goblinfactory]

Journal size was an issue for me when i was experimenting with using memstate with azure functions on consumption plans, using memstate as a rapid application development tool so that I didn't have to worry about databases. The size becomes an issue if you build a CRUD style "saas" service for web users to actually do stuff, rather than just a simple mostly readonly backing store for some mostly static webresource. I know this is not the stated goal for memstate, but if this could be achieved, then it really opens up memstate as being a simple database replacement in so many different scenarios, especially spike/test/experiment/early stage prototyping etc. In the azure function consumption plan scenario I was testing, having small (tiny) file sizes meant I could use local file storage / json or wire format with a really simple devops pipeline with a dynamically "mounted" file storage attached to the azure function.

[michaelnero]

Hi, this is great! An "immutable model" pattern was exactly the direction I was leaning as well. I also thought a bit about somehow having small partitions to localize the locks, but without an addressing system (like through a key) that memstate knows about, partitioning is a bit difficult here.

Glad you also mentioned snapshotting. I had some high-level thoughts about how to snapshot asynchronously so that read/writes continue while the snapshot is persisting. This means that the snapshot might be a bit behind the current model, but you can just apply committed events to the snapshot after restoring.

But, these both seem like "nice to haves" since it looks like even with heavy sustained writes, memstate performs well. Also, from what I can tell with my own tests, reading a large journal is still fast with an SSD.

Snapshotting, though, is probably the feature I'd prioritize though just so that my services start as quickly as possible. Happy to help with this feature if you're looking for help.

[goblinfactory]

mmm, maybe could have memstate open two journals when it starts, one is a command journal and another is a snapshot journal, and you read the snapshot journal backwards, and it has both the snapshot command, as well as a pointer to the offset (disk seek position, or similar) allowing you to skip ahead without needing journal rows to be fixed lengths. alternatively a special begining of record marker bytes?, that you could do a binary search in a very large journal file, say 100GB for example, that could find the seek position very quickly... then you restore snapshot and resume journal reads.

In any case, being able to snapshot in the background while updates are continuing to happen against the model (imo) would definitely require an immutable model approach, at least immutable partitions, not aggregate roots, since aggregates are still aggregated into mutable collections (typically ...table like collections) So you "lock" just a partition for reading (if you dont have a way to make an entire object graph / model totally immutable, which I suspect is and will be the case with most C# systems despites records and span improvements, ...this is still a hard (and unsolved ...generically at least) problem, as far as I know.

I started thinking about the idea of using memstate with a totally immutable model about 2 years ago and then "life happened" and I stopped, ... I looked for ways to do immutable appends at high speed, essentially how to thread safely clone a collection, add item and return new cloned immutable collection, and this is as far as I got using a linkedList and a mutable int pointer to "numrecords". I've not dissasembled the new span classes and various immutable collection methods, but the few I did briefly look at a while ago appeared to be built around a whole host of "concerns" and may be too generalised to easily extend without breaking the various concerns and generalised edge cases that would not apply to a memstate usage where we know 100% ... entirely everything about how the model will be accessed.

My "appendOnly List" is only 40 lines of C# : maybe this will spark some ideas?
https://github.com/goblinfactory/AppendOnly

There's no sychronisation in my code because I was wanting to use it with akka.net, and like memstate, we know only 1 thread will ever be updating the model at a time. (That reminds me, assuing you can run multiple memstate instances in the same app domain (which is a really big requirement) ... then would it make sense to have a factory that could return instance per partitions, then each partitioned instance can have its own "writer" threads?

Random thoughts for Friday ... Hope you all have a great Friday and a wickedly wonderful weekend! ciao...

cheers,
A

[rofr]

The MSFT immutable collections would be just fine to use.
https://docs.microsoft.com/en-us/dotnet/api/system.collections.immutable?view=net-6.0

And yes, you can run multiple engines within the same process and they will all be isolated from each other. Obviously they need their own separate journals.

[rofr]

@michaelnero I've also considered partitioning but came to the conclusion that a single global partition/lock is never a problem as long as read/write operations run quickly. That said, OrigoDB had a sharding option but that was mostly experimental: https://origodb.com/docs/core-0.19/client-api/partition-client/

[goblinfactory]

Some background for new readers; using memstate as a backing db for a CRUD system, e.g. content management system, is currently not a good fit with memstate because you end up creating a potentially expensive "large" journal record representing the "C" or "U" commands on every command, that would have to contain the text or body/content of the resource being updated. For example if you were building an ecommerce application, typically you'd use memstate for the mission critical domain commands split into micro services, ordering, billing, payments etc; these would be small and critical, and then have a normal table database like Postgres for the products, users, shopping basket services.

Effective snapshots could possibly ? solve this problem and open up a whole host of very powerful and effective use cases with immediate benefits for teams of all sizes.