Proxy Scraper is a tool that allows you to check the connectivity to HTTP, HTTPS, SOCKS4, and SOCKS5 proxy servers, as well as collect IP addresses and ports from websites specified in a text file.
To install and use Proxy Scraper, follow these steps:
- Clone the repository using the command:
git clone https://github.com/timursevimli/proxy-scraper/To use Proxy Scraper, follow these steps:
- Create a text file that lists the websites to collect IP addresses and ports from. Each URL should be specified on a separate line. For example (sources/proxy_sources.txt):
https://example.com
http://example.net- Edit the config.json file to configure the script execution and proxy checking options.
{
"mode": "single",
"timeout": 10000,
"source": "proxy_sources.txt",
"test": false,
"channels": 100
}- Run the script by executing the command:
npm start- The logs will be recorded in the logs/proxy-scraper.log file. You can view the log file to track the execution and check for results or errors.
When running the proxy-scaper, you can specify the following options:
-
mode< string >(default: 'single'): The type of proxy checking execution. Available values are 'single' for sequential checking and 'multi' for parallel checking.
-
timeout< number >(default: 10000): The timeout for each proxy check in milliseconds.
-
source< string >(default: 'proxy_sources.txt'): The path to the file containing the websites to collect IP addresses and ports from.
-
test< boolean >(default: false): A flag indicating whether to run in test mode (checking only a few websites).
-
channels< number >(default: 100) This property limits the number of operations and provides load balance in the application.
You can contribute to the development of Proxy Scraper by creating new features, fixing bugs, or improving the documentation. If you have any suggestions, please feel free to open an issue or submit a pull request.
This project is licensed under the MIT LICENSE