Apache Arrow
More info
- Creator
- Wes McKinney et al.
- Released
- Owner
- Apache Software Foundation
- Type
- In-memory columnar format
- License
- Apache 2.0
- Website
- arrow.apache.org
- Source
- github.com/apache/arrow
- Wikipedia
- en.wikipedia.org
Language-independent columnar in-memory format for analytics — lets different systems share data without serialisation/deserialisation.
Started in 2016 by Wes McKinney (pandas) and others. Defines both an in-memory layout and a wire format (Arrow Flight). A pandas DataFrame, a DuckDB table, a Polars frame and a Spark partition can all point at the same Arrow buffer — zero copy. Supports dictionary encoding, run-length encoding and nested types.
Underpins DuckDB, Polars, DataFusion, ClickHouse Connect and the Snowflake Python connector. Arrow IPC on disk = Feather format; the related Parquet is Arrow-compatible but optimised for long-term storage.