Feature/updating indeed scraper (#166) #170
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* - Updated to mobile endpoints and user agents to prevent CAPTCHA - Updated parsing of indeed scraper - Fixed tags not being parsed correctly - Fixed remoteness not being parsed correctly - Changed to only scrape the first page of each search by default for speed * - Updated method of loading user agent files - Updated user agent file of indeed scraper * - Updated versions in requirements.txt - Added in black configuration file for formatting - Added a pre-commit hook so all contributors will have consistent formatting on upload - Updated all python files to conform to black formatter * Updated Python version * More black formatting updates * - Added prettierrc and prettierignore - Formatted all files other than python * Updated prettierignore so prettier can search through subdirectories * Reset formatting to longer line width * Reverted to previous commit * Updating again to longer line width after accounting for missing files * Updated prettierrc and prettierignore files and reran formatting * Updated version
PaulMcInnis
requested changes
Sep 13, 2024
- Reverted settings_USA changes - Updated readme - Removed extra user-agent from phone user agents list - Removed extra comments
PaulMcInnis
requested changes
Sep 14, 2024
There was a problem hiding this comment.
wow makes me so happy to see this working again :)
a few things
- we should update readme to say that 3.11 is required at least
- we need to add
user_agent_mobile.txtto theMANIFEST.infile so that it is properly packaged when you install/run usingpip install .and thefunnel load -s...per the instructions for new users in the readme.
…mobile user agent list to the MANIFEST.in
|
Thanks @PaulMcInnis, we should be good to go! As I was updating, I was wondering why this project still uses I will open a new issue (#172) on this as I believe we should probably switch while the project is still small, but would love to hear if there are any reasons not to. |
Hey there, no reason other than age really - this is an old project and it needs some love |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Merging fixed indeed scraper into master branch. For more information, see #166
Additional changes include updates to formatting, adding black and prettier configuration files, and performing minor updates to the repo (versioning, updating modules, etc.).
Biggest point of discussion is whether we want to keep the current method of scraping each individual job page, or scraping the job list with description summaries. The latter (default implementation in this PR), is much faster, though will result in less information in the description. It is possible to manually switch back, but do we want to keep it as the default implementation?
Context of change
Type of change
How Has This Been Tested?
Same tests as in #166
Checklist: