← Latest products

I built an open-source data pipeline tool in Go

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows. Every data pipeline job I had to tackle required quite a few components to set up: - One tool to ingest data - Another one to transform it - If you wanted to run Python, set up an orchestrator - If you need to check the data, a data quality tool Let alone this being hard to set up and taking time, i.

Open source

1,798 stars · GitHubexternal site

Visit github.comexternal site

Reactions (1)

Similar products

by description similarity

  1. I built a local data lake for AI powered data engineering and analytics

    stream-sock-3f5.notion.site

    I got tired of the overhead required to run even a simple data analysis - cloud setup, ETL pipelines, orchestration, cost monitoring - so I built a fully local data-stack/IDE where I can write SQL/Py, run it, see results, and iterate quickly and interactively.

    Hacker News14 pts

  2. I built an open-source data copy tool called ingestr

    github.com

    ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

    Hacker News156 pts

  3. PipeRider

    github.com

    open-source Data Impact Analysis for dbt changes

    Hacker News25 pts

  4. A Database Written in Golang

    github.com

    An persistent relational database built in Go.

    Hacker News277 pts

  5. DBOS transact

    github.com

    Ultra-lightweight durable execution in Python

    Hacker News89 pts