Does NATS persist state?

[Ann-Geo]

Hi,
I had a question about persistent state and server restarts.
We have been running a 5 server NATS jetstream cluster for more than 20 days. We tried to stop one of the servers in the cluster.
At this time, we used the NATS CLI tool to retrieve the stream info/report, however the cluster gave an error showing: 'Jetstream service temporarily unavailable'. This happened even if there were other 4 four other servers (quorum) up and running in the cluster. All our streams are configured to use in-memory storage.

Following this we tried to restart the stopped server in the cluster. Even after performing the server restart, we could not retrieve the stream report for any of the streams.

So is it possible that NATS servers are persisting state somewhere in the filesystem? And this state is required for successful server restarts?
Our NATS server version is 2.10.14.

[wallyqs]

Hi, yes the nats-servers with JetStream require stable storage and if using Kubernetes need to make sure to use these charts: https://github.com/nats-io/k8s

[Ann-Geo]

Hi, thanks for your reply. We are not using Kubernetes. We are using systemd for managing the jetstream servers.

We see that our NATS servers are storing some data/metadata inside /tmp/nats/jestream directory even though we configured the stream storage to be in-memory . Could you give some info on what exactly it is storing? Is this where it stores cluster and stream state info?

So if I understand correctly, we can specify this directory using the store_dir field in the server conf file. So during server restarts, does the server always check this directory for any info that it needs for a successful restart and communication with cluster? Could you also give some info on how the restart sequence look like?

Just to understand NATS' design better, so NATS servers also rely on some state info stored persistently for restarts? Can't a server restart solely based on the RAFT communication with other servers in the cluster ?

[neilalexander]

There's only one store directory for a node, so if you're defaulting to /tmp/nats for any reason then that's because you haven't got store_dir set properly.

Each NATS node persists local state because in a failure scenario you have no way to know which nodes will survive or indeed whether all of them will go offline at once. Raft will help to reconcile state once the cluster comes back up.

[Ann-Geo]

Hi, any inputs on the above questions will be very helpful. Thanks in advance