Instacart Scraper

Simple spider and workflow system for scraping Instacart's (US) default store for a given user.

How to run

First, you have to setup your environment variables. Samples (.env.sample and .db.env.sample) were provided.

Then, execute (Docker and Docker Compose are required):

$ make run

Done 😄

Note: Recaptcha solving may fail. Retries are already in place, but in rare cases they are insufficient. In these cases, you try and run again.

If you want to run a query in the database, execute:

$ POSTGRES_USER=<YOUR-POSTGRES-USER> QUERY=<YOUR-QUERY> make sql-query

However, to make things easier, a shortcut to make a SELECT * on all tables is available through:

$ POSTGRES_USER=<YOUR-POSTGRES-USER> make sql-select-all

This project uses Docker. To destroy created images, volumes, etc., execute:

$ make destroy

To setup you developer environment, create a virtualenv and execute:

$ make dev-setup

This project uses pre-commit for managing code formatting and pip-tools to manage dependencies.