diff --git a/docs/content/integrations/airlift/airflow-dagster-equivalents.mdx b/docs/content/integrations/airlift/airflow-dagster-equivalents.mdx index 4040089266f71..14d8b92e374cd 100644 --- a/docs/content/integrations/airlift/airflow-dagster-equivalents.mdx +++ b/docs/content/integrations/airlift/airflow-dagster-equivalents.mdx @@ -370,3 +370,262 @@ Airflow Pools allow users to limit the number of concurrent tasks that can be ru ### Airflow Task Groups Airflow task groups allow you to organize tasks into hierarchical groups within the Airflow UI for a particular DAG. Dagster has _global_ asset groups which can be applied to any asset. Learn more about [asset groups](/concepts/assets/software-defined-assets#assigning-assets-to-groups). + +## Cheatsheet + +Here's a cheatsheet for Airflow users migrating to Dagster: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ Airflow concept + + Dagster concept + Notes
Directed Acyclic Graphs (DAG) + Jobs +
Task + Ops +
Datasets + Assets + + Dagster assets are more powerful and mature than datasets and include + support for things like{" "} + + partitioning + + . +
Connections/Variables + +
DagBags + Code locations + + Multiple isolated code locations with different system and Python + dependencies can exist within the same Dagster instance. +
DAG runsJob runs
+ depends_on_past + + + + An asset can{" "} + + depend on earlier partitions of itself + + . When this is the case, + backfills + and + Declarative Automation + will only materialize later partitions after earlier partitions have + completed. +
Executors + Executors +
Hooks + Resources + + Dagster resource contain a superset of + the functionality of hooks and have much stronger composition + guarantees. +
Instances + Instances +
OperatorsNone + Dagster uses normal Python functions instead of framework-specific + operator classes. For off-the-shelf functionality with third-party + tools, Dagster provides{" "} + integration libraries. +
Pools + Run coordinators +
Plugins/Providers + Integrations +
Schedulers + Schedules +
Sensors + Sensors +
SubDAGs/TaskGroups + + + Dagster provides rich, searchable{" "} + + metadata and tagging + {" "} + support beyond what’s offered by Airflow. +
+ task_concurrency + + + Asset/op-level concurrency limits + +
Trigger + Dagster UI Launchpad + + Triggering and configuring ad-hoc runs is easier in Dagster which allows + them to be initiated through the{" "} + Dagster UI, the{" "} + GraphQL API, or the CLI. +
XComs + I/O managers + + I/O managers are more powerful than XComs and allow the passing large + datasets between jobs. +