Cluster

[btsimonh]

Hi All,
I've been using Mosca in production for > 6 years, and it's done very well. I had a few issues with 'retain', but have now removed all retained messages, prefering to handle persistance in other ways, and so only using MQTT for real time.
I've also replaced most publish from devices (behind firewalls) with http POST, so the MQTT is mainly so devices can hear commands from the central server (there is some minimal device to device communication, mainly to ensure that the device is not started twice).
I have also reduced the number of connections from each 'device' to one (from a wasteful 3).
So my requirements break down to:
1/ a persistent connection from each client; I'd like to imagine 10-20,000 capacity.
2/ a publish once every 15s from each client, which will most likely only go back to that client.
3/ occasional messaging to specific clients or groups of clients.

Now I'm looking to up my game, and see aedes is likely the upgrade path (I use custom JWT based authentication which would harder -for me- in a non JS broker). I also note aedes mentions clustering.
But in the same breath it mentions requiring a separate backend (e.g. MongoDB).
Because I'm not interested in persistance and retain, is there a way to have a set of interconnected aedes brokers acting as a 'cluster', e.g. routing messages to another broker if that topic has been subscribed to on that other broker?
Or is there a whole world of hurts if not using a persistance layer which supports clustering?
(note my wishing for a 'cluster' is more down to HA than performance benefits. I'm looking at a Galeria cluster for databasing, and so would likely use the same machines for duplicating my other services, MQTT being just one).

I'm happy to work on a solution in order to keep devops relatively simple, but maybe this has been discussed or considered, and someone can highlight where the pitfalls are...

Simon

[robertsLando]

MongoDb/Redis are used on clusters not only for persistence but also for mqemitter that's the module that allows each instance to communicate between each other (it essentially handles all the pub/sub logic). So to answer your question you must use redis or mongo in a cluster env, you could decide to don't use a persistence and maybe it could work (of course it will not be reliable when it comes to QoS > 0 packets, not-clean connections and many other cases I may have not think about).

[btsimonh]

how about the 'bridge protocol'? One assumes that aedes could be modified such that if a message came in over the bridge, we could suppress it being sent to other bridged brokers. Then as long as all brokers subscribe to all others using the bridge protocol, in a steady state, messages should flow. During a failure condition, maybe all bets are off... but I'm not too worried about during a failure. Mainly about before and after a failure.

[robertsLando]

Bridge protocol is not fully supported yet, I know there are users that tried it and made something but I never tried it myself so couldn't tell you if it would work or not

[btsimonh]

I think I need to fully document what I need from such an arrangement for myself, and work out a plan for how multiple aedes could work together, and a testing strategy to test my specific requirements. I think MQTT over WSS still suits me well.