Use airflow dag scripts to extract and load data to database By using airflow, we can automate the process of extracting and loading our sensor data to our database.
Data Source
- https://anson.ucdavis.edu/~clarkf/I80_davis.txt.gz
- https://anson.ucdavis.edu/~clarkf/I80_stations.csv
Tasks
- Write Scripts for extracting and loading data
- Design the schema
- Connect and run the scripts with airflow
Resources
- https://airflow.apache.org/
- https://towardsdatascience.com/getting-started-with-apache-airflow-df1aa77d7b1b
Contributors Dibora Haile Gebreyohannes