A web scraping program that gathers online listings to generate used car purchase recommendations by Dallin Stewart
Table of Contents
Casper is a python program that uses selenium to gather data used car sales online from retailers including AutoTrader, CarsDirect, CarGuru, Carvana, Lowbook, and KSL. As of the most recent update, Casper only searches for cars in the Provo, Utah area. After gathering data, Casper scores each car on how good of a deal the sale is and generates a CSV report of the current market in addition to the market history since November 2022.
The name Casper is inspired by the Spanish word 'raspar', which means 'to scrape'. It is also a reference to 'Casper the Friendly Ghost' because when Casper is running, it looks like a ghost is controlling your computer!
- AutoTrader.py scrapes basic listing data according to parameters from autotrader.com
- CarGuru.py scrapes basic listing data according to parameters from carguru.com
- CarsDirect.py scrapes basic listing data according to parameters from carsdirect.com
- Carvana.py scrapes basic listing data according to parameters from carvana.com
- KSL.py scrapes basic listing data according to parameters from ksl.com
- Lowbook.py scrapes basic listing data according to parameters from lowbook.com
- CG_Detail.py scrapes detailed information according to parameters
- KSL_Detail.py scrapes detailed information according to parameters
- CD_Detail.py scrapes detailed information according to parameters
- Compressor.py compresses images scraped from websites using a custom image compression algorithm
- Search.py performs operations for scraping that are input and website agnostic
- main.py contains configuration details and calls each of the website scrapers to build final CSV reports
You'll need to start by downloading a few python packages with these commands:
- pip install selenium
- pip install pandas
- pip install scipy
- pip install matplotlib
- pip install numpy
- pip install plyer
- pip install imageio
- pip install pytorch
You can then clone this project to your own machine from GitHub and then run the webscraper in any Python development environment.
(top)
- Create a CSV file for each website and put them in a folder called 'Data':
a) autotrader.csv
b) CarsDirect.csv
c) CarGuru.csv
e) Carvana.csv
e) ksl.csv
f) current_market.csv - Run 'main.py'
(top)
(top)
Dallin Stewart - dallinpstewart@gmail.com
(top)
- Benjamin McMullin for writing Lowbook.py
- Brigham Young University, Applied and Computational Mathematics - About
- Brigham Young University Volume 1, Volume 2 Labs - Lab Descriptions
(top)