Orchestration
Schedulers and asset graphs that run pipelines in order.
- Apache Airflow
Open-source platform for authoring, scheduling, and monitoring data pipelines.
- Dagster
Asset-oriented orchestrator for data and ML pipelines, with a local developer loop.
- Prefect
Python-native workflow orchestration for pipelines in and out of the cloud.
- Kestra
Orchestrator that defines workflows in YAML and runs tasks across languages.
- Luigi
Python library from Spotify for building pipelines of batch jobs with dependencies.
- Flyte
Kubernetes-native workflow engine for data and machine learning pipelines.
- Argo Workflows
Kubernetes-native workflow engine that runs each step in a container.
- Mage
Pipeline tool for building and running data workflows in notebooks and code.
- Bruin
Open-source data platform (CLI and Bruin Cloud) that covers ingestion, SQL and Python, quality checks, and lineage.
- Astronomer
Commercial platform for running Apache Airflow.