Tools

Factual directory entries: name, category, website, whether the project is open source, the license and GitHub repo when those are known, a short description, and a high-level pricing model. Empty fields are unknown. This is not a ranking and it does not list prices or star counts.

Submit a tool

Orchestration

Schedulers and asset graphs that run pipelines in order.

  • Apache Airflow

    Open-source platform for authoring, scheduling, and monitoring data pipelines.

    Open source · Open source

  • Dagster

    Asset-oriented orchestrator for data and ML pipelines, with a local developer loop.

    Open source · Open core

  • Prefect

    Python-native workflow orchestration for pipelines in and out of the cloud.

    Open source · Open core

  • Kestra

    Orchestrator that defines workflows in YAML and runs tasks across languages.

    Open source · Open core

  • Luigi

    Python library from Spotify for building pipelines of batch jobs with dependencies.

    Open source · Open source

  • Flyte

    Kubernetes-native workflow engine for data and machine learning pipelines.

    Open source · Open source

  • Argo Workflows

    Kubernetes-native workflow engine that runs each step in a container.

    Open source · Open source

  • Mage

    Pipeline tool for building and running data workflows in notebooks and code.

    Open source · Open source

  • Bruin

    Open-source data platform (CLI and Bruin Cloud) that covers ingestion, SQL and Python, quality checks, and lineage.

    Open source · Open core

  • Astronomer

    Commercial platform for running Apache Airflow.

    Not open source · Commercial

Ingestion and ELT

Connectors and libraries that land source data in a warehouse or lake.

  • Fivetran

    Managed connectors that land SaaS and database data in a warehouse.

    Not open source · Usage-based

  • Airbyte

    Connector platform for moving data into warehouses and lakes, offered self-hosted and as a cloud service.

    Open source not labeled · Commercial

  • dlt

    Python library for building ingestion pipelines. This is the data load tool from dltHub, not other uses of the letters dlt.

    Open source · Open source

  • Estuary Flow

    Estuary's platform for CDC, batch, and streaming delivery from one capture into warehouses and other destinations.

    Not open source · Commercial

  • Meltano

    Open-source ELT runner built around the Singer connector spec.

    Open source · Open source

  • Debezium

    Open-source platform for change data capture from databases into Kafka and other sinks.

    Open source · Open source

  • Apache NiFi

    Flow-based system for routing and transforming data between systems.

    Open source · Open source

  • Stitch

    Commercial ETL service that loads SaaS and database sources into a warehouse.

    Not open source · Commercial

  • Hevo

    Commercial data pipeline product for loading sources into a warehouse.

    Not open source · Commercial

  • Matillion

    Commercial data integration and transformation product aimed at cloud warehouses.

    Not open source · Commercial

Transformation

SQL and dataframe frameworks that turn landed data into models.

  • dbt

    SQL-first framework for tested, documented warehouse models. This entry is dbt Core.

    Open source · Open source

  • dbt Cloud

    Commercial hosted product for developing, scheduling, and documenting dbt projects.

    Not open source · Commercial

  • SQLMesh

    SQL transformation framework with virtual environments and plan-based deploys.

    Open source · Open source

  • Apache Spark

    Engine for large-scale batch processing, with a streaming API on the same model.

    Open source · Open source

  • Polars

    Dataframe library for local analytical work in Python and Rust.

    Open source · Open source

  • Dataform

    SQL workflow framework for warehouse models. Google Cloud also offers a managed Dataform product.

    Open source · Open source

Warehouses and lakehouses

Engines that store and query analytical data.

  • Snowflake

    Cloud data warehouse used for analytics and data sharing.

    Not open source · Commercial

  • Databricks

    Lakehouse platform for engineering, analytics, and AI workloads on Spark and Delta.

    Not open source · Commercial

  • Google BigQuery

    Serverless data warehouse for SQL analytics.

    Not open source · Usage-based

  • Amazon Redshift

    Cloud data warehouse on AWS.

    Not open source · Commercial

  • ClickHouse

    Columnar database for analytical queries, with a commercial cloud.

    Open source · Open core

  • DuckDB

    In-process analytical database for SQL over local files.

    Open source · Open source

  • MotherDuck

    Commercial cloud service built around DuckDB.

    Not open source · Commercial

  • Trino

    Distributed SQL engine for querying data where it sits.

    Open source · Open source

  • StarRocks

    Open-source analytical database for SQL queries over lakehouse tables.

    Open source · Open source

Table formats

Open table formats that sit on object storage.

  • Apache Iceberg

    Open table format for large analytic datasets on object storage.

    Open source · Open source

  • Delta Lake

    Open table format that adds transactions and schema handling on data lakes.

    Open source · Open source

  • Apache Hudi

    Open table format for incremental upserts and streaming ingestion on data lakes.

    Open source · Open source

Streaming

Logs and processors for continuous event data.

  • Apache Kafka

    Distributed event log used for real-time data pipelines.

    Open source · Open source

  • Apache Flink

    Stream processing engine for stateful computations.

    Open source · Open source

  • Apache Pulsar

    Distributed messaging and streaming system.

    Open source · Open source

  • Redpanda

    Kafka-compatible streaming platform with a commercial cloud.

    Not open source · Commercial

Data quality and observability

Tests, monitors, and lineage for data at rest and in motion.

  • Great Expectations

    Framework for data tests, documentation, and validation.

    Open source · Open source

  • Soda

    Data quality checks. Soda Core is the open-source library. Soda Cloud is the commercial product.

    Open source · Open core

  • Monte Carlo

    Commercial data observability product for pipeline and data downtime.

    Not open source · Commercial

  • Elementary

    Data observability tooling that runs beside dbt, with a commercial cloud.

    Open source · Open core

  • OpenLineage

    Open standard and reference implementation for pipeline lineage events.

    Open source · Open source

Catalogs

Metadata, discovery, and governance.

  • DataHub

    Metadata platform for discovery, lineage, and governance.

    Open source · Open source

  • OpenMetadata

    Metadata catalog with lineage and quality views.

    Open source · Open source

  • Apache Atlas

    Metadata and governance framework from the Hadoop ecosystem.

    Open source · Open source

  • Amundsen

    Data discovery and metadata service.

    Open source · Open source

  • Unity Catalog

    Open catalog for tables, files, and models, including Iceberg.

    Open source · Open source

  • Apache Polaris

    Open catalog service for Apache Iceberg tables.

    Open source · Open source

BI and semantic layer

Dashboards and metric layers in front of a warehouse.

  • Cube

    Semantic layer that serves metrics from a warehouse to BI tools and apps.

    Open source · Open core

  • Looker

    Commercial BI product from Google, built around LookML models.

    Not open source · Commercial

  • Metabase

    BI tool for asking questions of a warehouse, with an open-source edition and a commercial cloud.

    Open source · Open core

  • Apache Superset

    Open-source BI application for charts and dashboards on SQL databases.

    Open source · Open source

  • Lightdash

    BI tool that reads metrics from a dbt project.

    Open source · Open core

  • Tableau

    Commercial BI product for dashboards and visual analysis.

    Not open source · Commercial

  • Power BI

    Commercial BI product from Microsoft.

    Not open source · Commercial

Reverse ETL

Tools that sync warehouse tables out to operational apps.

  • Hightouch

    Commercial reverse ETL product that syncs warehouse query results to other apps.

    Not open source · Commercial

  • Census

    Commercial reverse ETL product that syncs warehouse data to operational tools.

    Not open source · Commercial

Vector and AI data

Vector stores used beside pipelines. Not a ranking of AI products.

  • Pinecone

    Managed vector database for storing embeddings used in search and retrieval.

    Not open source · Commercial

  • Weaviate

    Vector database for embeddings, with filters on structured fields. You can run it yourself or use the hosted service.

    Open source · Open core

  • Qdrant

    Vector database for similarity search, with a self-hosted build and a cloud.

    Open source · Open core

  • pgvector

    PostgreSQL extension that stores and queries vector embeddings in Postgres.

    Open source · Open source

  • Chroma

    Embedding database used for retrieval beside application data.

    Open source · Open source

  • Milvus

    Open-source vector database for similarity search, with a commercial hosted offer from Zilliz.

    Open source · Open core

  • LanceDB

    Vector database built on the Lance columnar format.

    Open source · Open source