5/8/24, 2:30 PM (7) Open source tools for Data Engineering | LinkedIn

7 Unlo
Home My Network Jobs Messaging Notifications Me For Business

Open source tools for

Data Engineering
Midhun Pottammal
Data Engineer and Full Stack Expert | Hadoop, 14 articles Follow
Spark, Kafka, Python, and NoSQL (Hive, Hbas…

February 14, 2024

Open Immersive Reader

Data Integration

1. Apache NiFi: A powerful and easy-to-use tool for

moving data between systems.

2. Airbyte: An open-source data integration platform that

helps you replicate your data in your warehouses,
lakes, and databases.

3. Meltano: An open-source data integration tool that

simplifies the process of extracting, loading, and
transforming data.

4. Apache Inlong: A platform for real-time data ingestion

and complex event processing.

5. Apache SeaTunnel: A data transfer tool for efficiently

moving large volumes of data.

Storage

1. HDFS: The Hadoop Distributed File System, designed

for storing large files across multiple machines.

2. Apache Ozone: A scalable, redundant, and distributed

object store for Hadoop.

3. Ceph: A distributed object, block, and file storage

platform.

https://www.linkedin.com/pulse/open-source-tools-data-engineering-midhun-pottammal-cxvtf/ 1/5
5/8/24, 2:30 PM (7) Open source tools for Data Engineering | LinkedIn

4. MinIO: A high-performance, distributed object storage

server.

Data Lake Platform

1. Apache Hudi: A data lake solution for managing large

analytical datasets.

2. Apache Iceberg: A table format for storing huge, slow-

moving tabular data.

3. Delta: An open-source storage layer that brings ACID

transactions to Apache Spark.

4. Paimon: A data lake platform for managing and

analyzing data at scale.

Event Processing

1. Kafka: A distributed event streaming platform capable

of handling trillions of events a day.

2. Redpanda: A Kafka-compatible event streaming

platform with a focus on performance and scalability.

3. Pulsar: A cloud-native, distributed messaging and

streaming platform.

Data Processing & Computation

1. Apache Spark: An open-source, distributed computing

system that provides an interface for programming
entire clusters with implicit data parallelism and fault
tolerance.

2. Apache Flink: A framework and distributed processing

engine for stateful computations over unbounded and
bounded data streams.

3. Vaex: A Python library for lazy, out-of-core

DataFrames.

4. Ray: A fast and simple framework for building and

running distributed applications.

5. Dask: A flexible parallel computing library for analytic

computing.

6. Polars: A blazingly fast DataFrame library implemented

in Rust and using Apache Arrow.

Database

https://www.linkedin.com/pulse/open-source-tools-data-engineering-midhun-pottammal-cxvtf/ 2/5
5/8/24, 2:30 PM (7) Open source tools for Data Engineering | LinkedIn

OLTP:SQL — RDBMS(MySQL, Postgres), In

Memory(Apache Ignite)NoSQL — KV(Aerospike),
Document (MongoDB), Graph(Neo4J),
Multimodel(ArangoDb)

HTAP:NewSQL — stonedb, TiDB

OLAP:Oflline — Columnar(Databend), Time Series

(TimeScale)Realtime — Realtime OLAP (Druid, Pinot,
Clickhouse, StarRocks), Search Engine, Streaming
Database (Materialize, RisingWave)

Visualization

1. Superset: A modern, enterprise-ready business

intelligence web application.

2. Rath

3. Redash: A visualization and dashboarding tool.

4. Metabase: An easy way to generate charts and

dashboards, ask simple ad hoc queries without using
SQL, and see detailed information about rows in your
Database.

Data Infrastructure

Kubernetes: An open-source container orchestration

platform.

Ambari: A software project designed to enable system

administrators to provision, manage, and monitor a
Hadoop cluster.

Workflow Management & DataOps

1. Airflow: A platform to programmatically author,

schedule, and monitor workflows.

2. Dagster: A data orchestrator for machine learning,

analytics, and ETL.

3. Kestra: A workflow orchestrator for data pipeline

management.

4. Temporal: An open-source, stateful microservices

orchestration platform.

5. Mage: A workflow engine for orchestrating data

pipelines.

6. Windmill: A platform for building and running data

pipelines.

https://www.linkedin.com/pulse/open-source-tools-data-engineering-midhun-pottammal-cxvtf/ 3/5
5/8/24, 2:30 PM (7) Open source tools for Data Engineering | LinkedIn

7. DolphinScheduler: A distributed and easy-to-expand

visual DAG workflow scheduling system, dedicated to
solving the complex dependencies in data processing,
making the scheduling system out of the box for data
processing.

Monitoring

Prometheus + Mimir & Grafan + Loki

EFK

Metadata Management

1. Datahub: An open-source metadata platform for the

modern data stack.

2. Amundsen: A data discovery and metadata platform.

3. Marquez: An open-source metadata service for the

collection, aggregation, and visualization of a data
ecosystem's metadata.

Report this

Published by

Midhun Pottammal 14
Data Engineer and Full Stack Expert | Hadoop, Spark, Kafka, Python, and NoSQ… Follow
Published • 2mo articles

🌟 Exciting Tools in the World of Data Engineering! 🌟

#DataEngineering #OpenSource #TechTrends #DataEngineering #OpenSource

#TechTrends #DataIntegration #Storage #DataLake #DataProcessing #Database
#EventProcessing #Visualization #DataInfrastructure #WorkflowManagement
#DataOps #Monitoring #MetadataManagement

Like Comment Share 17

Reactions

+5

0 Comments

Add a comment…

Midhun Pottammal
Data Engineer and Full Stack Expert | Hadoop, Spark, Kafka, Python, and NoSQL (Hive,
Hbase, Iceberg) | Specialised in Informatica, Nifi, Cloudera CDP, and Databricks

https://www.linkedin.com/pulse/open-source-tools-data-engineering-midhun-pottammal-cxvtf/ 4/5
5/8/24, 2:30 PM (7) Open source tools for Data Engineering | LinkedIn

Follow

More from Midhun Pottammal

Apache Iceberg Schema Benefit of Data Observability:

Evolution Unlocking the Insights 🚗
Midhun Pottammal on Linke… Midhun Pottammal on Linke…

Star Schema vs Snowflake

Schema: Key Differences
Between The Two

Midhun Pottammal on Linke…

See all 14 articles

https://www.linkedin.com/pulse/open-source-tools-data-engineering-midhun-pottammal-cxvtf/ 5/5