Automated workflow for data scraping

Process

Get data from URL
Clean data in Pandas dataframe
Convert dataframe to .csv
Store .csv in S3 bucket as text file
Deploy scraper to AWS lambda and run it there every 15 minutes

Creating a new scraper

Create a copy of the scraper_template.py file or one of the existing scrapers
Implement your scraper
Add your scraper to the list of scrapers in handler.py

How to test locally

Run your scraper .py skript: python get_data_scrapername.py

Deploy

Push your code to the repo, it will get deployed automatically

Contributing

Make pull request (or ask for repo membership)
After review your new scraper will be deployed automatically

Error reporting

Any errors in a scraper are reported to Sentry
Use asserts to confirm the data is valid
If non-critical but interesting changes to the data are noticed, report them to Slack via Sentry message