Skip to content

Latest commit

 

History

History
95 lines (78 loc) · 2.43 KB

README.md

File metadata and controls

95 lines (78 loc) · 2.43 KB
Raw

Apache Airflow

Apache Airflow comes with 2 components

  1. [Airflow Operator-Controller] (#Airflow Operator-Controller)
  2. [Airflow Base + Cluster] (#Airflow Base + Cluster)

Airflow Operator-Controller

Airflow Operator-Controller is deployed via the Airflow Cluster and Airflow Operator components in KfDef

Airflow Base + Cluster

Airflow Base and Airflow Cluster are deployed via the base.yaml and cluster.yaml here Examples for all available configurations

Airflow Base

Airflow Base deploys base components for Airflow and can be customized.

example

apiVersion: airflow.apache.org/v1alpha1
kind: AirflowBase
metadata:
  name: pc-base
spec:
  postgres:
    operator: False
    image: "registry.access.redhat.com/rhscl/postgresql-10-rhel7"
    version: "latest"
  storage:
    version: ""

Parameters

  1. Database

Can be postgres or mysql

  1. Database Image

Can be changed to custom image and version

Airflow Cluster

Airflow Cluster deploys cluster components and can be customized

example

apiVersion: airflow.apache.org/v1alpha1
kind: AirflowCluster
metadata:
  name: pc-cluster
spec:
  executor: Celery
  redis:
    operator: False
  scheduler:
    image: "quay.io/opendatahub/docker-airflow"
    version: "openshift"
  ui:
    image: "quay.io/opendatahub/docker-airflow"
    replicas: 1
    version: "openshift"
    enableroutes: true
  worker:
    image: "quay.io/opendatahub/docker-airflow"
    replicas: 2
    version: "openshift"
    ForceRoot: "true"
  flower:
    image: "quay.io/opendatahub/docker-airflow"
    replicas: 1
    version: "openshift"
    enableroutes: true
  dags:
    subdir: "airflow/dags/"
    git:
      repo: "https://github.com/VedantMahabaleshwarkar/airflow-dags"
      once: true
  airflowbase:
    name: pc-base

Parameters

  1. Executor
  • Celery
  • Kubernetes
  1. Routes enableroutes should be true when deployed on OpenShift Clusters.
  2. DAG Source Dag source can be customized to point to any github repo where DAG files are located in {your_repo}/airflow/dags

Additional Info

This component will create a CRD for kubernetes-sig/application which is a dependency for the Airflow cluster deployment.