-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Follow redirects feature #35
Comments
It looks like this project hasn't been touched in a couple of years, so I'm not sure if it is still maintained or not. If so, I suspect this could be an easy fix by simply pulling another setting from app.settings and using that in
Setting FollowRedirects to true seems to have the desired effect in testing, though the original link still shows in the output with a 200 status rather than say
I could also be way off base here! Happy to do a PR if I'm not. |
Can I work on this feature if it is still open? |
If the server responds with redirect (301 or 302), the crawler should follow these redirects (like curl is able for example), otherwise it's missing a bunch of crawlable content
The text was updated successfully, but these errors were encountered: