CTFdScraper is a simple scraper for automating challenges gathering from a CTFd platform. Generally, it's utilizing the CTFd endpoint on /api/v1/challengesor /chalsfor older version.
In order to run this properly, you need a python2 or python3 environment installed. For a certain circumstance, termux user need to install several packages before installing lxml.
# Termux-only (lxml)
$ pkg install clang libxml2 libxslt
$ pip install cythonAfter fulfilling the prerequisites, you can directly clone this repository or download the source from our releases.
$ git clone https://github.com/hanasuru/CTFdScraperAfter that, go install its dependencies by using pip.
$ cd CTFdScraper/
$ pip install -r requirements.txtFor instances, you can check helper section by passing -h or --help.
$ python ctfd.py --help
usage: ctfd.py [-h] [--data data] [--proxy proxy] [--path path]
[--worker worker] [--scheme scheme] [--enable-cloud]
[--override] [--no-download] [--export]
[user] [passwd] [url]
Simple CTFd-based scraper for challenges gathering
positional arguments:
user Username/email
passwd Password
url CTFd platform url
optional arguments:
-h, --help show this help message and exit
--data data Populate from challs.json
--proxy proxy Request behind proxy server
--path path Target directory, default: CTF
--worker worker Number of threads, default: 10
--scheme scheme URL scheme, default: https
--enable-cloud Permit file download from a cloud drive, default=False
--override Override existed chall file
--no-download Don't download chall file
--export Export challenges directory as zip, default=False
By default, you need to define username, password, and CTFd url respectively (use HTTPS by default).
$ python ctfd.py user passwd https://some-domainBy default, this option are disabled due to file size issues where user accidentally spend most of internet quotas from downloading a large file. Otherwise, you can enable this opting by passing --enable-cloud argument
$ python ctfd.py user passwd https://some-domain --enable-cloudCurrently, only Google-drive and Dropbox that are supported.
As a note, the review that shows size output may be wrong (false positive) due to the usage of request.get(url, stream=True) in order to get the Content-Length header
During CTF competition, the organizer may update the current binary/file. Unfortunately, by default CTFdScraper couldn't override the existed challenges file. If you insist to modify the current binary/file, you need to pass --override argument
$ python ctfd.py user passwd https://some-domain --overrideAlternatively, you can also populate only the challenges from an existed challs.json. Additionally, you can also combine with --enable-cloud & --override to download files from a cloud drive. Furthermore, check the examples.
# Populate challenges only from existed data
$ python ctfd.py --data challs.json
# Populate challenges alongwith files
$ python ctfd.py user passwd https://some-domain --data chall.json
# Populate challenges alongwith interal file (if public) and external file
$ python ctfd.py --data chall.json --enable-cloud --overrideTo be able to conduct a requests behind proxy you need to pass --proxy proxy-server argument or manually add export http_proxy="http://host:port".
$ python ctfd.py user passwd https://some-domain --proxy 127.0.0.1:8080
# or set up an environtment variable
$ export http_proxy = "http://127.0.0.1:8080"
$ export https_proxy = "http://127.0.0.1:8080"For archiving purpose, you can save the CTFd folder as a Zip file by passing the --export argument
$ python ctfd.py user passwd https://some-domain --exportLastly, the collected challenge will be saved to CTF/${ctf_name}
directory. You can customize the path by passing --path pathnameargument.
$ tree
.
└── Online Playground CTF for Beginner
├── challs.json
├── Cryptography
│ ├── Base64
│ │ └── README.md
│ └── Single-Byte XOR Cipher
│ └── README.md
├── Forensic
│ ├── Data Exfil
│ │ └── README.md
│ └── Volatility 4
│ └── README.md
├── Pwn
│ ├── cariuang
│ │ └── README.md
│ └── vault
│ └── README.md
├── Reverse
│ ├── IFEST-password
│ │ └── README.md
│ └── Password
│ └── README.md
└── Web
├── babyPHP
│ └── README.md
└── Optimus Prime
└── README.md# Optimus Prime [50 pts]
**Category:** Web
**Solves:** 23
## Description
>Optimus Prime is coming
http://18.139.8.91:3015/
author: Arkavidia5
**Hint**
* -
## Solution
### Flag
For instances, here is the demonstration of CTFdScraper. For simplicity, the svg file was removed
▶ python ctfd.py test 12345 http://playgroundctf.xyz
✔ Login Success
✔ Found 43 new challenges
✔ Found 24 files (0.5 MB downloaded)
[Summary]
✔ Web (3)
✔ Reverse (3)
✔ Cryptography (9)
✔ Forensic (10)
✔ Intro (7)
✔ Survey (1)
✔ Pwn (5)
✔ Hacktoday 2019 - Penyisihan (5)
[Finished in 3.44 second]▶ python ctfd.py test 12345 http://playgroundctf.xyz --enable-cloud
✔ Login Success
✔ Loaded 43 challs from challs.json
⚠ There are no new challenges
✔ Found 26 files (143.0 MB downloaded)
[Summary]
✔ Web (3)
✔ Reverse (3)
✔ Cryptography (9)
✔ Forensic (10)
✔ Intro (7)
✔ Survey (1)
✔ Pwn (5)
✔ Hacktoday 2019 - Penyisihan (5)
[Finished in 9.56 second] ▶ python ctfd.py --data challs.json
✔ Loaded 43 challs from challs.json
✔ Found 24 files (0.0 MB downloaded)
[Summary]
✔ Web (3)
✔ Reverse (3)
✔ Cryptography (9)
✔ Forensic (10)
✔ Intro (7)
✔ Survey (1)
✔ Pwn (5)
✔ Hacktoday 2019 - Penyisihan (5)
[Finished in 0.09 second]
- hanasuru - Initial work
See also the list of contributors who participated in this project.
This project is licensed under the MIT License - see the LICENSE file for details
- CTFd, a Capture The Flag framework focusing on ease of use and customizability.
- markdown2png, a module to render John Gruber's markdown format to
PNG(or other image formats) by usingPillowandMarkdowndependencies.