A data scraper from texas govt site and a helping web app for managing, reviewing and editing the data
A friend of mine taken data entry contract for for their farm. It was a lot of data. You need to go the govt site of Texas, serch city one by one, go to the company/customer lists and copy those data to excel steet. It could take one mongth (two weeks if two person did this) for a single data entry operator or analyst to add those data and keep those data organized in excel steet.
In the middle of their work I told them to tell me all the problems they are facing and all the fetures/issues this site have. I analyzed the source code of the site and also saw that there are some issue with sessions so it is not an easy task to make a scraper fot the site. But still after testing and tricking the site I made a very simple but perfect scraper. Mined all the data and then made an admin panel for the data entry operators and/or analysts.
It's no facncy work but it's a two pieces software that made their work very easy and some great amount of money with perfect data output and client satisfaction. I am uploading this project for those who may need such apps and may use my work as reference or helper to build theri project or may directly use this (with some variable value change) if their work is similar.
Note: I made this two pieces software for having the work done perfectly and deveoped it very quickly. So, I did not follow all the best practices of software development and good practices of python. Even regular expressions used here are perfect but not clean because it was made to have the work done perfectly not to make it a viral project.
You can contact me through http://sabuj.me or go to my linkedin account where my user name is SabujXi or also same user name on twitter or facebook.
Below are some screenshots - I haven't taken any screenshot during scraping, so the following screenshots are from the panel of data entry/analyst operator.
