Use blocking lists with database instead of memory ?

[chmichael]

Hello,
Would be possible instead of loading all the blocking lists into the memory to use a disk database ? (sqlite or mysql etc)

Thank you

[ShreyasZare]

Thanks for the post. Loading block list in memory is done for two reasons, first is performance and second is to avoid dependency issues in core DNS server.

Like for example, if you have DNS server running on an old Ubuntu 18.04, the Sqlite support for it has stopped and so the Query Logs (Sqlite) app will fail to work on it. If it was used in core DNS server then the DNS server would have stopped working and you would be forced to upgrade OS with unplanned downtime.

Having mysql dependency means that you will need to administer a DB server too which adds another task for installing and maintaining the DNS server.

It still possible to have a DNS App that uses a database backend and provide blocking feature. But it depends on the use-case since in most cases, there is sufficient RAM available on the server to have a reasonably large block list loaded in memory.

[chmichael]

Well you can make it optional. Ofc it would be slight slower than memory.

I dunno if sqlite is fast enough for this job or it would better to use MySQL but currently my DNS Server is using 2GB of memory due the blocking lists.

Anyhow please consider it for a future release sometimes it's needed to do a restart of our servers.

Thank you

[ShreyasZare]

Sqlite is not fast enough. On an NVMe SSD, max throughput that can be achieved is approx 25,000 transactions/second which is too low for DNS server.

I have not tested MySQL but there is a plan to add Query Logs app support for MySQL. Will evaluate its performance during the implementation. It would still be slower and have effect on the DNS server's performance.

currently my DNS Server is using 2GB of memory due the blocking lists.

I guess you have millions of domain names loaded to use up this much memory. If the block lists that you use provide "wildcard" formatted lists then use them since it would reduce the memory usage and give you same level of blocking even if the number of domains that you see loaded on dashboard are lower.

[chmichael]

I understand. Another option is MariaDB (MySql compatible) with RocksDB engine if you want to experiment.

Thank you!