Skip to content

trallard/airflow-tutorial

Folders and files

NameName
Last commit message
Last commit date

Latest commit

ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 

Repository files navigation

Airflow Tutorial

License: CC BY 4.0

This repo contains the materials for the pipelines tutorial on Pycon -> from scripts soups to Airflow.

The tutorial covers:

  • Setting up local databases
  • Creating basic ETL pipelines in Python: query APIs, load data to databases, perform data cleaning and filtering and persist the consumption ready data
  • How to set a local instance of Airflow and get it running
  • Creating basic DAGS in Airflow
  • Transform script soups ETLS into Airflow dags
  • Set up an Airflow instance in Azure

To add:

  • Setting a Kubernetes powered instance on Azure AKS
  • Adding CI/CD to using Azure pipelines

If you are interested in following along visit: https://airflow-tutorial.readthedocs.io/en/latest/

The setup instructions can be found at: https://airflow-tutorial.readthedocs.io/en/latest/setup.html

If you would like to experiment with Azure follow this link to get a free trial subscription with 150 dollars.

๐Ÿš€ PRs and Issues are welcome

License

License: CC BY 4.0

This repo is licensed using a CC-BY so you are free to use, remix, and share so long attribution is provided to the original author.

About

๐Ÿ๐Ÿ’จ Airflow tutorial for PyCon 2019

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published