PySpark
The Python API for Apache Spark — all Spark functionality (DataFrame, Spark SQL, MLlib, Structured Streaming) available from Python.
Works through Py4J which bridges Python ↔ JVM. Initially had overhead due to JVM context switching; Spark Connect (3.4, 2023) uses gRPC instead for better performance. Pandas API on Spark (formerly Koalas) gives pandas syntax scaled across Spark. Default in Databricks, AWS Glue, many data engineering jobs.