This pulls user and node data from Social Media Today.
This is setup to run in a docker container. If you're in a place with a docker runtime, simply run:
make
and it will build the image locally and run the crawl. Afterwards the results will be in the feeds directory.
If you're developing this code, hop the container with the following:
docker run -it --rm -v ${PWD}:/app industrydive/smt-scrapy:latest bash
and run scrapy things to your heart's content.
This runs on Scrainghub Cloud which is much much better and more useful than running locally when you're not developing for the code.