Should clustering use RFC1996 or is it okay if it uses another solution?

[zbalkan]

Hi,

I wonder if the RFC 1996 should be the initial way of clustering Technitium. If used, it will not only implement another RFC but also improve interoperability with other DNS server implementations if one needs it.

On the other hand, if the long-term strategy is to use a pure Technitium cluster, then one can utilize any other replication method, like a database replication service such as rqlite or litestream That would decouple cluster synchronization and management from the DNS implementation and easier to implement. But it also increases complexity due to external dependencies.

There is also a possibility to use dqlite but one needs to write a C# wrapper around it. This would integrate into the software directly as it is not a service but a library.

The last option of replication without RFC 1996 is to write a Technitium-specific replication protocol, which would cover the replication.

The intent of this discussion is that implementing RFC 1996 would not be enough for a full replication implementation. Because, RFC 1996 mentions only sending notification messages to slaves in the master-slave architecture. But there are issues on it:

Left open by RFC 1996 is who is to be notified - which is harder to figure out than it sounds. All slaves for this domain must receive a notification but the nameserver only knows the names of the slaves - not the IP addresses, which is where the problem lies. The nameserver itself might be authoritative for the name of its secondary, but not have the data available.
To resolve this issue, PowerDNS tries multiple tactics to figure out the IP addresses of the slaves and notifies everybody. In contrived configurations, this may lead to duplicate notifications being sent out, which shouldn’t hurt.

Some backends may be able to detect zone changes, others may choose to let the operator indicate which zones have changed and which haven’t.

The second issue is getting out of sync when cluster members have not taken some notifications due to an issue. It is possible to track SOA serial number. But in case of a problem where synchronization is lost, there is a possibility that serial numbers match but the records do not. Because SOA serial numbers are not hashed of data, they are just version numbers. Though it is possible to use a date time based format like YYYYMMDDXX where XX is the version number of the changes on that date. Some implementations use YYYYMMDDSS or Unix EPOC. This approach decreases the possible problems, not eliminates. A diff based replication or a full replication may be needed in case of node failures. So, even if RFC 1996 DNS NOTIFY messages are implemented, there must be other means to replicate and ensure integrity of records amongst cluster nodes. Though the likelihood is so low, the impact may be high. It is up to the users' risk appetite.

I wanted to start a discussion and asking your feedback.

[zbalkan]

These problems occur on other DNS implementations, too. For instance, many BIND users utilize https://dotat.at/prog/nsdiff/ for DNS zone comparison, a tool which creates a diff and outputs an nsupdate script to help administrators. There are other issues that are mentioned in the code though: if DNS change is too large to fit in a DNS packet, it will raise an error. Or, DNSSEC implementations break the deterministic structure of SOA serial numbers, risking the sync operation, that is why the script raises error to protect integrity.

[ShreyasZare]

Thanks for the post. Technitium DNS server supports NOTIFY (RFC 1996) along with IXFR to sync zones. The DNS server stores zone data internally in an in-memory tree data structure and thus there is no database like sqlite being used so any database based solution wont work.

The clustering feature that is planned for Technitium DNS server will be using NOTIFY and IXFR standards along with Catalog Zones for automatic provision of secondary zones.

[zbalkan]

Another topic is the multi-master cluster -where is it the best way in larger environments- and the possible disaster scenario of connection loss between data centers. It may be a network loss or another error causing problems. The master election may result with a split-brain scenario because the DNS servers may elect themselves as the sole masters in two separate locations. Any changes conducted during the outage would increase complexity and must be solved after the resolution of the problem. Technitium's cluster synchronization should solve conflicts where both DNS master nodes have different values.

[ShreyasZare]

This is going to be a bit different here than the usual clustering implementations. This is since the primary zones that are DNSSEC signed will hold the private keys and thus any other node in the cluster cannot assume or be elected as the primary node. So a solution for this still needs to be worked out.

[zbalkan]

Yes, it is going to be hard. Because in large deployments, there is also the problem of stealth master for authoritative DNS servers. Which would increase complexity. I am not sure how to make it a manageable one.