-
Notifications
You must be signed in to change notification settings - Fork 31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Byte limit #75
Labels
Comments
At the moment, it's possible to generate large (fake or valid) robots.txt files, with the aim to trap the robots.txt crawler, slow down the server, and even cause it to hang or crash. It's also (depending on the setup) possible to trap the crawler in an infinite retry-loop, if the external code utilizing this library, isn't handling repeating fatal errors correctly... Related to #62 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Feature request: Limit the maximum number of bytes to parse.
Source: Google
Source: Yandex
The text was updated successfully, but these errors were encountered: