Parse domains using the TLD list maintained by publicsuffix.org.
Domains are parsed into their TLD, SLD, and subdomains. TLDs like
'co.uk' are handled properly. The list of possible TLDs is generated
from publicsuffix.org. The first time the library is run, an HTTP
request is made and a .tld.pickle file is created. Subsequent runs
don't make an HTTP request. Rather, they load the pickle file. If you
want to refresh the list, simply delete the pickle file.
There are tests, but not enough. I'm hoping that if people find this useful they'll contribute test cases to the projects.
- Use
pylruand TLD_CACHE to speed up parsing in long-running programs