Skip to content
/ censys Public

Python code to query the Censys public scan database.

Notifications You must be signed in to change notification settings

gelim/censys

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

17 Commits
 
 
 
 
 
 
 
 

Repository files navigation

Python code to query the Censys public scan database. This script is made around library censys-python (https://github.com/Censys/censys-python) and is inteded to make censys queries quick & easy from command-line.

Requirements

You need to create an account on https://censys.io and get your API key and secret at https://censys.io/account

Important note: your queries will be throttled. What is allowed is 0.2 tokens/second (60.0 per 5 minute bucket).

$ sudo pip install -r requirements.txt

Usage

$ censys_io.py --help
usage: censys_io.py [-h] [-m MATCH] [-f FILTER] [--count] [-r REPORT]
                    [-B REPORT_BUCKET] [-a ASN] [-c COUNTRY] [-o CERT_ORG]
                    [-i CERT_ISSUER] [-s CERT_HOST] [-S HTTP_SERVER]
                    [-t HTML_TITLE] [-b HTML_BODY] [-T TAGS] [--api_id API_ID]
                    [--api_secret API_SECRET] [-d] [-v] [-l LIMIT] [-H]
                    [--tsv]
                    [arguments [arguments ...]]

Censys query via command line

-- gelim

positional arguments:
  arguments             Censys query

optional arguments:
  -h, --help            show this help message and exit
  -m MATCH, --match MATCH
                        Highlight a string within an existing query result
  -f FILTER, --filter FILTER
                        Filter the JSON keys to display for each result (use value 'help' for interesting fields)
  --count               Print the count result and exit
  -r REPORT, --report REPORT
                        Stats on given field (use value 'help' for listing interesting fields)
  -B REPORT_BUCKET, --report_bucket REPORT_BUCKET
                        Bucket len in report mode (default: 50)
  -a ASN, --asn ASN     Filter with ASN (ex: 36040 for Google Inc.)
  -c COUNTRY, --country COUNTRY
                        Filter with country
  -o CERT_ORG, --cert-org CERT_ORG
                        Cert issued to org
  -i CERT_ISSUER, --cert-issuer CERT_ISSUER
                        Cert issued by org
  -s CERT_HOST, --cert-host CERT_HOST
                        hostname cert is issued to
  -S HTTP_SERVER, --http-server HTTP_SERVER
                        Server header
  -t HTML_TITLE, --html-title HTML_TITLE
                        Filter on html page title
  -b HTML_BODY, --html-body HTML_BODY
                        Filter on html body content
  -T TAGS, --tags TAGS  Filter on specific tags. E.g: -T tag1,tag2,... (use keyword 'list' to list usual tags
  --api_id API_ID       Censys API ID (optional if no env defined
  --api_secret API_SECRET
                        Censys API SECRET (optional if no env defined)
  -d, --debug           Debug informations
  -v, --verbose         Print raw JSON records
  -l LIMIT, --limit LIMIT
                        Limit to N results
  -H, --html            Renders html elements in a browser
  --tsv                 Export result of search in TSV format

For full details about the formatting rules for arguments see search syntax in page https://censys.io/ipv4/help?q=x%3Ax

For a quick and dirty test, you can build queries like:

  • foo AND bar (will do a smart search by checking all keys with value foo and bar)
  • path.to.key:foo
  • key:foo (shortcut of previous, but will give strange results if there are collision with other keys)
  • key:/regex/ (regexp support via operator '/')
  • key:"long string with spaces" (need to quote those strings)
  • key:[200 TO 300] (int range queries)
  • key:192.168.0.0/24 (IP range query)

Note about looking for substrings

Censys is backed by Elasticsearch, plus they filter the requests for performance reasons.

If you want to look for all values beginning with the string Whatsapp, you can look for key:Whatsapp*. If you look only for key:Whatsapp you will get only fields that where analyzed (strings cut into pieces depending on language rules, specific tokens, etc.) and contains facets with the exact string "Whatsapp".

An example to highlight that is the values stored in the key 443.https.tls.certificate.parsed.subject.organization. By looking for 443.https.tls.certificate.parsed.subject.organization:Whatsapp you will find ~90 results. Those will be entries with values:

  • WhatsApp Inc.
  • WhatsApp Company Ltd
  • WhatsApp

But you will miss the values WhatsApp, Inc. that has ~350 entries. If we check censys.io "Data definitions" this field should be analyzed as a "String" and the comma should be removed by the tokenizer but something is not working as expecting.

So you need to be very careful when looking for substrings and try different methods by either doing wildcards search (beware that Censys disable beginning search term with a wildcard) or by using pure regexp like /.*Whatsapp.*/.

Examples

Generic query IP or host (look for anything matching the string in Censys indexed data)

Let's search for IP entries that contain the string "nmap" in one of their keys.

$ censy_io.py --limit 20 nmap
Number of results: 1002
5.196.225.134   Title: N/A                                        SSL: dawidstachowiak.pl                           AS: OVH, (16276)                        Loc: FR /                     OS: N/A        Tags: http, ssh, https
74.115.246.29   Title: BrainDump                                  SSL: philmcclure.duckdns.org                      AS: ENERGIZE (19215)                    Loc: US / Pulaski             OS: N/A        Tags: http, ssh, https
104.237.156.37  Title: Starlight Networking Security Lab          SSL:                                              AS: LINODE-AP (63949)                   Loc: US / Absecon             OS: N/A        Tags: http, ssh
69.160.84.231   Title: N/A                                        SSL:                                              AS: FIBER (5048)                        Loc: US / Orem                OS: CentOS     Tags: http, ssh
45.79.82.183    Title: nweb.io                                    SSL: nweb.io                                      AS: LINODE-AP (63949)                   Loc: US / Absecon             OS: N/A        Tags: http, ssh, https
60.32.137.218   Title: Kyodo2.0 Digital-Lab News Map Project      SSL: localhost.localdomain                        AS: OCN (4713)                          Loc: JP / Tokyo               OS: Fedora     Tags: dhe-export, rsa-export, http, https
104.237.129.231 Title: Ninja.Style                                SSL:                                              AS: LINODE-AP (63949)                   Loc: US / Absecon             OS: Ubuntu     Tags: http, ssh
192.109.14.42   Title: PASA Pallas                                SSL: pasa.pallas.com                              AS: PALLAS-AS, (24861)                  Loc: DE /                     OS: N/A        Tags: http, https
45.33.32.156    Title: Go ahead and ScanMe!                       SSL:                                              AS: LINODE-AP (63949)                   Loc: US / Absecon             OS: Ubuntu     Tags: http, ssh
104.224.137.222 Title:                                            SSL:                                              AS: IT7NET (25820)                      Loc: US / Phoenix             OS: CentOS     Tags: http
119.81.35.59    Title: SL Labs                                    SSL:                                              AS: SOFTLAYER (36351)                   Loc: SG / Singapore           OS: CentOS     Tags: http
81.27.98.98     Title: Check for Web Servers and more             SSL:                                              AS: UK-NETCETERA (24851)                Loc: GB /                     OS: Debian     Tags: http
212.237.16.237  Title: Infosec Notes                              SSL: 2d8.ru                                       AS: ARUBA-ASN, (31034)                  Loc: DK /                     OS: Ubuntu     Tags: http, smtp, https
198.23.94.99    Title: SL Labs                                    SSL:                                              AS: SOFTLAYER (36351)                   Loc: US / San Jose            OS: CentOS     Tags: http
77.109.162.35   Title: Citrin Toolbox                             SSL:                                              AS: INIT7, (13030)                      Loc: CH /                     OS: N/A        Tags: http
121.42.165.133  Title:                                            SSL:                                              AS: CNNIC-ALIBABA-CN-NET-AP (37963)     Loc: CN / Hangzhou            OS: CentOS     Tags: http, ssh
169.55.196.202  Title: SL Labs                                    SSL:                                              AS: SOFTLAYER (36351)                   Loc: US /                     OS: CentOS     Tags: http
119.81.209.6    Title: SL Labs                                    SSL:                                              AS: SOFTLAYER (36351)                   Loc: SG / Singapore           OS: CentOS     Tags: http
216.59.36.36    Title: Wait, wha?                                 SSL:                                              AS: IMMEDION (15085)                    Loc: US / Greenville          OS: N/A        Tags: http
204.152.250.58  Title: My Blog                                    SSL:                                              AS: BCC-65-182-96-0-PHX (33055)         Loc: US / Phoenix             OS: N/A        Tags: http

Count how much web servers have 'SAP' in their Server header

$ censys_io.py -S SAP --count
3299

Get geo reparition of server with 'ABAP' in their Server header

$ censys_io.py -S ABAP --report location.country.raw --report_bucket 10
Number of results: 602
count           raw
159             United States
90              Germany
30              Brazil
27              Italy
25              India
20              Australia
18              Canada
16              Denmark
16              Republic of Korea
16              Spain

Filter hosts by their HTML title

This is a very cool feature, but unfortunately works only for tcp/80 scans. They don't store http content on https scans.

$ censys_io.py --title "IVRE Web UI"
Number of results: 17
62.210.244.184  Title: IVRE Web UI                                SSL:                                              AS: AS12876, (12876)                    Loc: FR /                     OS: N/A        Tags: http, ssh
206.128.155.79  Title: IVRE Web UI                                SSL:                                              AS: CENTURYLINK-LEGACY-SAVVIS (3561)    Loc: US / Chesterfield        OS: Ubuntu     Tags: http
138.201.231.148 Title: IVRE Web UI                                SSL: pms.pditsolutions.eu                         AS: HETZNER-AS, (24940)                 Loc: DE /                     OS: N/A        Tags: http, ssh, https
140.86.51.236   Title: IVRE Web UI                                SSL:                                              AS: NETDYNAMICS (7160)                  Loc: FR / Paris               OS: Debian     Tags: http, ssh
5.196.197.153   Title: IVRE Web UI                                SSL:                                              AS: OVH, (16276)                        Loc: FR /                     OS: Ubuntu     Tags: http, ssh
206.128.155.47  Title: IVRE Web UI                                SSL:                                              AS: CENTURYLINK-LEGACY-SAVVIS (3561)    Loc: US / Chesterfield        OS: Ubuntu     Tags: http
45.77.23.20     Title: IVRE Web UI                                SSL:                                              AS: AS-CHOOPA (20473)                   Loc: US / Matawan             OS: Ubuntu     Tags: http, ssh
66.85.60.134    Title: IVRE Web UI                                SSL:                                              AS: 5THCOLUMN (394205)                  Loc: US /                     OS: Ubuntu     Tags: http
45.32.217.177   Title: IVRE Web UI                                SSL:                                              AS: AS-CHOOPA (20473)                   Loc: US / Atlanta             OS: Ubuntu     Tags: http, ssh
52.37.12.137    Title: IVRE Web UI                                SSL:                                              AS: AMAZON-02 (16509)                   Loc: US / Wilmington          OS: N/A        Tags: http, ssh
209.126.161.62  Title: IVRE Web UI                                SSL:                                              AS: CARINET (10439)                     Loc: US / San Diego           OS: Debian     Tags: http
45.32.35.133    Title: IVRE Web UI                                SSL:                                              AS: AS-CHOOPA (20473)                   Loc: US / Matawan             OS: Ubuntu     Tags: http, ssh
138.197.35.240  Title: IVRE Web UI                                SSL:                                              AS: DIGITALOCEAN-ASN (14061)            Loc: US / Wilmington          OS: N/A        Tags: http, ssh
52.32.21.139    Title: IVRE Web UI                                SSL:                                              AS: AMAZON-02 (16509)                   Loc: US / Wilmington          OS: Ubuntu     Tags: http, ssh
174.138.79.85   Title: IVRE Web UI                                SSL:                                              AS: DIGITALOCEAN-ASN (14061)            Loc: US / San Diego           OS: Ubuntu     Tags: http, ssh
103.78.158.19   Title: IVRE Web UI                                SSL:                                              AS: DIGITALCORPORATION-AS-AP (135671)   Loc:  /                       OS: N/A        Tags: http
13.58.144.246   Title: IVRE Web UI                                SSL:                                              AS: AMAZON-02 (16509)                   Loc: US / Norwalk             OS: N/A        Tags: http, ssh

Retrieve the hosts that have SSL certificate with organization 'Whatsapp'

$ censys_io.py --cert-org "Whatsapp*" --limit 10
Number of results: 456
104.236.63.164  Title: phpinfo()                                  SSL: web.whatsapp.com                             AS: DIGITALOCEAN-ASN-NY3 (393406)       Loc: US / New York            OS: Ubuntu     Tags: http, ssh, https
169.55.74.44    Title: N/A                                        SSL: *.whatsapp.net + *.whatsapp.net              AS: SOFTLAYER (36351)                   Loc: US /                     OS: N/A        Tags: https
169.55.69.140   Title: N/A                                        SSL: *.whatsapp.net + *.whatsapp.net              AS: SOFTLAYER (36351)                   Loc: US /                     OS: N/A        Tags: https
169.45.71.55    Title: N/A                                        SSL: *.whatsapp.net + *.whatsapp.net              AS: SOFTLAYER (36351)                   Loc: NL /                     OS: N/A        Tags: https
169.45.71.118   Title: N/A                                        SSL: *.whatsapp.net + *.whatsapp.net              AS: SOFTLAYER (36351)                   Loc: NL /                     OS: N/A        Tags: https
169.54.210.17   Title: N/A                                        SSL: *.whatsapp.net + *.whatsapp.net              AS: SOFTLAYER (36351)                   Loc: US /                     OS: N/A        Tags: https
169.55.235.181  Title: N/A                                        SSL: *.whatsapp.net + *.whatsapp.net              AS: SOFTLAYER (36351)                   Loc: US /                     OS: N/A        Tags: https
169.45.71.42    Title: N/A                                        SSL: *.whatsapp.net + *.whatsapp.net              AS: SOFTLAYER (36351)                   Loc: NL /                     OS: N/A        Tags: https
158.85.5.217    Title: N/A                                        SSL: *.whatsapp.net + *.whatsapp.net              AS: SOFTLAYER (36351)                   Loc: US / Chantilly           OS: N/A        Tags: https
177.75.8.102    Title: N/A                                        SSL: *.whatsapp.net + *.whatsapp.net              AS: Networld Provedor e Servicos de Internet Ltda, BR (28178)Loc: BR /                     OS: N/A        Tags: https

Highlight which keys are matching string for a specific search query

Let's have a look at SAP ABAP servers answering an error on their frontpage and let's highlight an interesting keyword:

censys_io.py -t "Logon Error Message" -S ABAP -m "ICF-nf-http" --limit 10

Printing raw JSON record from database for a specific request

$ censys_io.py ip:8.8.8.8 --verbose
Number of results: 1
{u'53': {u'dns': {u'lookup': {u'additionals': [],
                              u'answers': [{u'name': u'c.afekv.com',
                                            u'response': u'192.150.186.1',
                                            u'type': u'A'},
                                           {u'name': u'c.afekv.com',
                                            u'response': u'74.125.76.6',
                                            u'type': u'A'}],
                              u'authorities': [],
                              u'errors': False,
                              u'metadata': {},
                              u'open_resolver': True,
                              u'questions': [{u'name': u'c.afekv.com',
                                              u'type': u'A'}],
                              u'resolves_correctly': True,
                              u'support': True}}},
 u'autonomous_system': {u'asn': 15169,
                        u'country_code': u'',
                        u'description': u'GOOGLE - Google Inc., US',
                        u'name': u'GOOGLE',
                        u'organization': u'Google Inc., US',
                        u'path': [15169],
                        u'rir': u'unknown',
                        u'routed_prefix': u'8.8.8.0/24'},
 u'ip': u'8.8.8.8',
 u'location': {u'city': u'Mountain View',
               u'continent': u'North America',
               u'country': u'United States',
               u'country_code': u'US',
               u'latitude': 37.386,
               u'longitude': -122.0838,
               u'postal_code': u'94040',
               u'province': u'California',
               u'registered_country': u'United States',
               u'registered_country_code': u'US',
               u'timezone': u'America/Los_Angeles'},
 u'metadata': {},
 u'protocols': [u'53/dns'],
 u'tags': [],
 u'updated_at': u'2016-06-02T06:50:30+00:00'}

Listing example of fields we only want to dump in the records

$ censys_io.py -f list
['location.country',
 'location.country_code',
 'location.city',
 'ip',
 'protocols',
 'autonomous_system.name',
 'autonomous_system.asn',
 '443.https.tls.certificate.parsed.subject.organization',
 '443.https.tls.certificate.parsed.subject.common_name',
 '443.https.tls.certificate.parsed.extensions.subject_alt_name.dns_names',
 '993.imaps.tls.tls.certificate.parsed.subject.common_name',
 '993.imaps.tls.tls.certificate.parsed.subject.organization',
 '80.http.get.title',
 '80.http.get.headers.server',
 '80.http.get.body',
 'metadata.os',
 'tags']

Export to disk sites frontpage containing "Hacked by" in their title

$ censys_io.py --html-title "Hacked by" --limit 20 --html
Number of results: 1044
185.71.217.66   Title: Hacked By                                  SSL:                                              AS: SSDHOSTING, (201534)                Loc: CY /                     OS: Unix       Tags: pop3, ftp, http, pop3s, smtp, imap, imaps
45.55.207.28    Title: Hacked By M.e-dz – Hacked By M.e-dz  SSL:                                              AS: DIGITALOCEAN-ASN (14061)            Loc: US / New York            OS: Ubuntu     Tags: http, ssh
190.156.247.34  Title: hacked by suliman_hacker – hack[...] SSL:                                              AS: Telmex Colombia S.A., CO (10620)    Loc: CO /                     OS: Ubuntu     Tags: http
206.225.85.55   Title: Hacked By NorilaClasse | Hacked By No[...] SSL:                                              AS: CODERO-DFW (18501)                  Loc: US / Overland Park       OS: Ubuntu     Tags: ftp, http, ssh
69.89.28.35     Title: Hacked by bl4ck_cod3 | Hacked by bl4c[...] SSL: www.phliworldwide.com                        AS: UNIFIEDLAYER-AS-1 (46606)           Loc: US / Provo               OS: N/A        Tags: pop3, ftp, http, ssh, https, pop3s, imap, imaps
188.215.244.170 Title: Hacked by Ashura - Hacked by Ashura        SSL: rohc.ro + rohc.ro+                           AS: GTSCE (5588)                        Loc: RO /                     OS: N/A        Tags: pop3, ftp, http, https, pop3s, imap, imaps
95.213.235.31   Title: Hacked by ZeDaN-Mrx | Hacked by ZeDaN-Mrx  SSL:                                              AS: SELECTEL, (49505)                   Loc: RU /                     OS: Ubuntu     Tags: http, ssh
216.247.60.144  Title: Hacked By DeathCreppy Death-Stalkers [...] SSL: technology4vision.com + technology4vision.com+AS: PEER1 (13768)                       Loc: US / Atlanta             OS: N/A        Tags: pop3, ftp, http, ssh, https, pop3s, imap, imaps
149.56.147.247  Title: Hacked by zakiloup – Hacked by [...] SSL: savilerowtailors.ca + savilerowtailors.ca+   AS: OVH, (16276)                        Loc: US / Anaheim             OS: N/A        Tags: pop3, ftp, http, ssh, https, imap, imaps
216.235.2.38    Title: Hacked by CoLDHaCKeR , Hacked by Sensi     SSL:                                              AS: E-GATE-COMMUNICATIONS (13657)       Loc: CA / Toronto             OS: FreeBSD    Tags: ftp, http
31.14.23.187    Title: Hacked by Ashura | Hacked by Ashura        SSL: e-techdevelopment.ro + e-techdevelopment.ro+ AS: GTSCE (5588)                        Loc: RO /                     OS: N/A        Tags: pop3, ftp, http, https, pop3s, imap, imaps
188.215.244.141 Title: Hacked by Ashura – Hacked by Ashura  SSL: bisericafloresti.ro + bisericafloresti.ro+   AS: GTSCE (5588)                        Loc: RO /                     OS: N/A        Tags: pop3, ftp, http, https, pop3s, imap, imaps
118.96.152.47   Title: Hacked By $!R05                            SSL:                                              AS: TELKOMNET-AS2-AP (17974)            Loc: ID / Magelang            OS: N/A        Tags: http, ssh
207.7.94.210    Title: Hacked by Katib                            SSL: pureinternet.com.au + pureinternet.com.au+   AS: PRIVATESYSTEMS (63410)              Loc: US / Los Angeles         OS: N/A        Tags: pop3, ftp, http, https, pop3s, imap, imaps
209.29.150.141  Title: Hacked By GHoST61                          SSL: www.snakeoil.dom                             AS: ASN852 (852)                        Loc: CA /                     OS: Unix       Tags: ftp, dhe-export, rsa-export, http, https
46.229.175.179  Title: HaCKeD By PoLoNia WwW.NeTSaW.OrG           SSL:                                              AS: ADVANCEDHOSTERS-AS, (39572)         Loc: US /                     OS: Unix       Tags: ftp, http, ssh
71.18.197.222   Title: Hacked By #nof34rPT                        SSL: *.opentransfer.com + *.opentransfer.com+     AS: OPENTRANSFER-ECOMMERCE (32392)      Loc: US / Columbus            OS: N/A        Tags: ftp, http, https
88.198.101.66   Title: Hacked By Cyb3r Gl@diat0rs                 SSL:                                              AS: HETZNER-AS, (24940)                 Loc: DE /                     OS: Debian     Tags: http, ssh
176.226.210.113 Title: Hacked by Mr.Sh4hz3b-HaXoR                 SSL:                                              AS: INTERSVYAZ-AS (8369)                Loc: RU / Kurgan              OS: N/A        Tags: http
129.121.106.11  Title:  HaCked By virus3033                       SSL:                                              AS: NEXCESS-NET (36444)                 Loc: US / Redwood City        OS: N/A        Tags: ftp, http

We then can browse /tmp/tmpx5HjqB like in

Content is available in key '80.http.get.body' and saved to disk for offline analysis. Beware of malicious content that could be viewed from the browser. No filtering is done on the content stored on disk.

Use tags provided by censys scanner to look at servers that need to be secured

$ censys_io.py --tags heartbleed --report location.country.raw
Number of results: 213034
46565     United States
26009     China
12383     Germany
8138      India
8004      Russia
6471      United Kingdom
6111      France
5817      Italy
5387      Republic of Korea
5257      Japan

But what the heck, you want to hack the planet?!

Well, I use this great OSINT source for my assignments sometime when I want to show the client his internet exposure, or for bug bounties to do landscape discovery. This is as well a good example to show that port scanning is not evil and help to increase security posture awareness.

About

Python code to query the Censys public scan database.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages