Topic

PySpark

2 posts.

Oct 04, 2026 10 min read

Following up on the Spark 4.2.0 UDF hang

In my last post I left one problem unexplained: ordinary Python UDFs that never finished on cached or Parquet data. This weekend I took a closer look. Here is how I narrowed it down, what was going on, and where the fix stands.
Oct 01, 2026 6 min read

Spark 4.2 turns on Arrow for ordinary Python UDFs

Spark 4.2 changes how ordinary Python functions trade data with the JVM. The option has been there since Spark 3.4. Here is what it does, how it got here, and what I measured.