Delta Lake
More info
- Creator
- Databricks
- Released
- Owner
- Linux Foundation / Databricks
- Type
- Lakehouse storage layer
- License
- Apache 2.0
- Website
- delta.io
- Source
- github.com/delta-io/delta
- Wikipedia
- en.wikipedia.org
Open-source storage layer from Databricks — adds ACID transactions, time travel and schema enforcement on top of Parquet files in a data lake.
The trick: a JSON transaction log (_delta_log/) next to the Parquet files; every commit appends a new log file describing which files come in and out. Optimistic concurrency — works without locking as long as conflicts are rare. Supports MERGE, UPSERT, VACUUM and OPTIMIZE.
Competes with Apache Iceberg and Apache Hudi. Iceberg often wins in open ecosystems (Trino, Snowflake, BigQuery); Delta wins when you're already on Spark/Databricks.