Performance issues encountered with 16 million queries

[350887156]

One master server and 6 slave servers
master server
cpu :16 core,Memory:64g Network bandwidth: 20M
slave servers
cpu :16 core,Memory:32g Network bandwidth: 20M

There is no problem with remote operation of the main server, but the DNS software is stuck
Below is a screenshot of the server status.
Please help me analyze what caused this problem
Thank you very much

[ShreyasZare]

Thanks for the feedback. Does this issue occur only on one server or is it occurring on all instances that you have running? Do you have any zones hosted on the affected servers? What OS is being used for all the DNS instances? Is docker being used?

[350887156]

@ShreyasZare Hello
Currently, this situation mainly occurs on the main DNS instance. There are about 200,000 zones on the affected server. All DNS instances use Windows systems, not Docker.

Only the main server fluctuates, and the slave servers do not fluctuate.

[ShreyasZare]

Thanks for the details. Are any of the zones DNSSEC signed? If yes, are they using NSEC3?

From the main chart, it looks like the DNS service is recovering after approx 5 mins downtime. Is it recovering on its own or do you restart the DNS service?

Do you see any error logged in the DNS logs in the admin panel? Especially, any error log which has System.TimeoutException?

[350887156]

@ShreyasZare
No zone is DNSSEC signed. It will recover automatically. I did not do anything else. The log records are all related to timeouts.

[ShreyasZare]

No zone is DNSSEC signed. It will recover automatically. I did not do anything else. The log records are all related to timeouts.

Thanks for the details. Please share a few error logs here so that I can check the code against it.

[350887156]

The error log file occupies too much disk space and has been deleted by the operation and maintenance personnel. The current error information is as follows

Error! The operation on the socket could not be performed because the system buffer was insufficient in space or a queue was full.

[ShreyasZare]

Thanks for the details. It would be much useful if you can send the complete error listed in the DNS log file since a partial error message is not much useful since its unclear where its originating.

Does the DNS server have too many TCP requests? (check dashboard chart for protocol usage)

Is the OS Windows Server edition or Windows 10? Are there any other server applications like web server running on the same server as that of the DNS server?

Do you have and DNS apps installed?

[350887156]

This is the log collected so far. If it is not enough, I will add more later.
log.txt

[ShreyasZare]

Thanks for the logs. The logs just have operational errors logged, almost all of which are due to the upstream forwarders refusing to answer. I am not sure about the reason for this and you need to change the upstream forwarders to some other provider. If you are not using encrypted DNS protocols then enable either DoT/DoH with them. You can also consider running the DNS server without any forwarders.

Its possible that due to these errors with upstream, there is a large number of outstanding requests on the DNS server causing UDP port exhaustion which is causing the issue you have.