Tools
Factual directory entries: name, category, website, whether the project is open source, the license and GitHub repo when those are known, a short description, and a high-level pricing model. Empty fields are unknown. This is not a ranking and it does not list prices or star counts.
Orchestration
Schedulers and asset graphs that run pipelines in order.
- Apache Airflow
Open-source platform for authoring, scheduling, and monitoring data pipelines.
Open source · Open source
- Dagster
Asset-oriented orchestrator for data and ML pipelines, with a local developer loop.
Open source · Open core
- Prefect
Python-native workflow orchestration for pipelines in and out of the cloud.
Open source · Open core
- Kestra
Orchestrator that defines workflows in YAML and runs tasks across languages.
Open source · Open core
- Luigi
Python library from Spotify for building pipelines of batch jobs with dependencies.
Open source · Open source
- Flyte
Kubernetes-native workflow engine for data and machine learning pipelines.
Open source · Open source
- Argo Workflows
Kubernetes-native workflow engine that runs each step in a container.
Open source · Open source
- Mage
Pipeline tool for building and running data workflows in notebooks and code.
Open source · Open source
- Bruin
Open-source data platform (CLI and Bruin Cloud) that covers ingestion, SQL and Python, quality checks, and lineage.
Open source · Open core
- Astronomer
Commercial platform for running Apache Airflow.
Not open source · Commercial
Ingestion and ELT
Connectors and libraries that land source data in a warehouse or lake.
- Fivetran
Managed connectors that land SaaS and database data in a warehouse.
Not open source · Usage-based
- Airbyte
Connector platform for moving data into warehouses and lakes, offered self-hosted and as a cloud service.
Open source not labeled · Commercial
- dlt
Python library for building ingestion pipelines. This is the data load tool from dltHub, not other uses of the letters dlt.
Open source · Open source
- Estuary Flow
Estuary's platform for CDC, batch, and streaming delivery from one capture into warehouses and other destinations.
Not open source · Commercial
- Meltano
Open-source ELT runner built around the Singer connector spec.
Open source · Open source
- Debezium
Open-source platform for change data capture from databases into Kafka and other sinks.
Open source · Open source
- Apache NiFi
Flow-based system for routing and transforming data between systems.
Open source · Open source
- Stitch
Commercial ETL service that loads SaaS and database sources into a warehouse.
Not open source · Commercial
- Hevo
Commercial data pipeline product for loading sources into a warehouse.
Not open source · Commercial
- Matillion
Commercial data integration and transformation product aimed at cloud warehouses.
Not open source · Commercial
Transformation
SQL and dataframe frameworks that turn landed data into models.
- dbt
SQL-first framework for tested, documented warehouse models. This entry is dbt Core.
Open source · Open source
- dbt Cloud
Commercial hosted product for developing, scheduling, and documenting dbt projects.
Not open source · Commercial
- SQLMesh
SQL transformation framework with virtual environments and plan-based deploys.
Open source · Open source
- Apache Spark
Engine for large-scale batch processing, with a streaming API on the same model.
Open source · Open source
- Polars
Dataframe library for local analytical work in Python and Rust.
Open source · Open source
- Dataform
SQL workflow framework for warehouse models. Google Cloud also offers a managed Dataform product.
Open source · Open source
Warehouses and lakehouses
Engines that store and query analytical data.
- Snowflake
Cloud data warehouse used for analytics and data sharing.
Not open source · Commercial
- Databricks
Lakehouse platform for engineering, analytics, and AI workloads on Spark and Delta.
Not open source · Commercial
- Google BigQuery
Serverless data warehouse for SQL analytics.
Not open source · Usage-based
- Amazon Redshift
Cloud data warehouse on AWS.
Not open source · Commercial
- ClickHouse
Columnar database for analytical queries, with a commercial cloud.
Open source · Open core
- DuckDB
In-process analytical database for SQL over local files.
Open source · Open source
- MotherDuck
Commercial cloud service built around DuckDB.
Not open source · Commercial
- Trino
Distributed SQL engine for querying data where it sits.
Open source · Open source
- StarRocks
Open-source analytical database for SQL queries over lakehouse tables.
Open source · Open source
Table formats
Open table formats that sit on object storage.
- Apache Iceberg
Open table format for large analytic datasets on object storage.
Open source · Open source
- Delta Lake
Open table format that adds transactions and schema handling on data lakes.
Open source · Open source
- Apache Hudi
Open table format for incremental upserts and streaming ingestion on data lakes.
Open source · Open source
Streaming
Logs and processors for continuous event data.
- Apache Kafka
Distributed event log used for real-time data pipelines.
Open source · Open source
- Apache Flink
Stream processing engine for stateful computations.
Open source · Open source
- Apache Pulsar
Distributed messaging and streaming system.
Open source · Open source
- Redpanda
Kafka-compatible streaming platform with a commercial cloud.
Not open source · Commercial
Data quality and observability
Tests, monitors, and lineage for data at rest and in motion.
- Great Expectations
Framework for data tests, documentation, and validation.
Open source · Open source
- Soda
Data quality checks. Soda Core is the open-source library. Soda Cloud is the commercial product.
Open source · Open core
- Monte Carlo
Commercial data observability product for pipeline and data downtime.
Not open source · Commercial
- Elementary
Data observability tooling that runs beside dbt, with a commercial cloud.
Open source · Open core
- OpenLineage
Open standard and reference implementation for pipeline lineage events.
Open source · Open source
Catalogs
Metadata, discovery, and governance.
- DataHub
Metadata platform for discovery, lineage, and governance.
Open source · Open source
- OpenMetadata
Metadata catalog with lineage and quality views.
Open source · Open source
- Apache Atlas
Metadata and governance framework from the Hadoop ecosystem.
Open source · Open source
- Amundsen
Data discovery and metadata service.
Open source · Open source
- Unity Catalog
Open catalog for tables, files, and models, including Iceberg.
Open source · Open source
- Apache Polaris
Open catalog service for Apache Iceberg tables.
Open source · Open source
BI and semantic layer
Dashboards and metric layers in front of a warehouse.
- Cube
Semantic layer that serves metrics from a warehouse to BI tools and apps.
Open source · Open core
- Looker
Commercial BI product from Google, built around LookML models.
Not open source · Commercial
- Metabase
BI tool for asking questions of a warehouse, with an open-source edition and a commercial cloud.
Open source · Open core
- Apache Superset
Open-source BI application for charts and dashboards on SQL databases.
Open source · Open source
- Lightdash
BI tool that reads metrics from a dbt project.
Open source · Open core
- Tableau
Commercial BI product for dashboards and visual analysis.
Not open source · Commercial
- Power BI
Commercial BI product from Microsoft.
Not open source · Commercial
Reverse ETL
Tools that sync warehouse tables out to operational apps.
Vector and AI data
Vector stores used beside pipelines. Not a ranking of AI products.
- Pinecone
Managed vector database for storing embeddings used in search and retrieval.
Not open source · Commercial
- Weaviate
Vector database for embeddings, with filters on structured fields. You can run it yourself or use the hosted service.
Open source · Open core
- Qdrant
Vector database for similarity search, with a self-hosted build and a cloud.
Open source · Open core
- pgvector
PostgreSQL extension that stores and queries vector embeddings in Postgres.
Open source · Open source
- Chroma
Embedding database used for retrieval beside application data.
Open source · Open source
- Milvus
Open-source vector database for similarity search, with a commercial hosted offer from Zilliz.
Open source · Open core
- LanceDB
Vector database built on the Lance columnar format.
Open source · Open source