Update subdomain sources

[edoardottt]

• Add SubdomainCenter as subdomain source #107
• Add ahrefs as subdomain source #108
• Add builtwith as subdomain source #109
• Add dnsrepo.noc.org as subdomain source #110
• Add grep.app/api/search as subdomain source #111
• Remove bufferoverrun as subdomain source #112
• Remove sonar as subdomain source #113

[ExitoLab]

@edoardottt how will this work? Can you give examples

[edoardottt]

Sure @ExitoLab .

Scilla with the subdomain subcommand tries to retrieve subdomains for a specific inputted domain:
e.g.

scilla subdomain -target google.com

The previous command performes a lot of requests against google.com subdomains.

If instead this command is used:

scilla subdomain -target google.com -db -no-check

The flag -db means that we're using API from some online sources in order to retrieve previously collected subdomains.
As example https://crt.sh/?q=%25.google.com&output=json.

At time of writing (12 oct 2023) scilla supports these online sources: anubis, bufferoverrun, crtsh, hackertarget, sonar, threatcrowd, threatminer and virustotal.

The problems are that some of these services stopped working and they are very few.

You can see here in the amass repo https://github.com/owasp-amass/amass/tree/5f1f7176bae60975c1e5be64273cb201f1bb37c3/resources/scripts/api or in the subfinder one https://github.com/projectdiscovery/subfinder/tree/main/v2/pkg/subscraping/sources how many of them are used.

The goal is to remove the not working ones and add new ones to the https://github.com/edoardottt/scilla/tree/main/pkg/opendb package in scilla. You can see in that directory how it works.

I'm gonna create a list of issues specific for each new online service to be added.

[iamshreeram]

@edoardottt , As of today (10/15/23) Below aren't working -
bufferoverrun, Sonar Omnisint, threat Crowd

@ExitoLab , We can probably split the work on implementing subdomain sources - subdomaincenter, ahref, dnsrepo, grep.app;

[ExitoLab]

@iamshreeram sure, we can split the work. Which one do you want to work on

[iamshreeram]

@ExitoLab , Im currently working on subdomaincenter. You can pick the next.

@edoardottt - Regarding grep.app as subdomain source -
From the Ada codebase, The API (https://grep.app/api/search?regexp=true&q=([_a-zA-Z0-9]{1}[_a-zA-Z0-9-]{0,61}[a-zA-Z0-9]{1}.google.com)) is failing with Forbidden Error.

But, this (https://grep.app/api/search?q=google.com&regexp=true) is returning a payload that is unrelated to the subdomain; Please have a look into this. thanks!

[edoardottt]

Got it...So this https://github.com/owasp-amass/amass/blob/5f1f7176bae60975c1e5be64273cb201f1bb37c3/resources/scripts/api/grepapp.ads#L24 is not working..

We can skip that for now :(