Spark 4.2 turns on Arrow for ordinary Python UDFs

One line in the Spark 4.2.0 release notes is easy to read past:

[SPARK-54555] Enable Arrow-optimized Python UDFs and Arrow-based PySpark IPC by default

It changes something that has worked the same way for as long as PySpark has had user-defined functions. It also asks nothing of you: no new API, no rewrite. The same function you wrote years ago runs, and for many functions it simply runs faster. This note explains what changed, where the feature came from, and what I measured when I tried it.

What actually changed

Spark runs on the JVM. Your Python function does not. So every time Spark needs to call a Python UDF, it has to hand the values to a separate Python process, wait, and take the results back. That handoff is the expensive part of a small UDF, and it is the only thing Spark 4.2 changes.

Before, Spark sent the values over in small groups of 100 rows using pickle, Python’s general-purpose format for saving objects. Every value is packed up as its own little Python object on one side and unpacked on the other.
Now, Spark sends them as Apache Arrow batches of up to 10,000 rows. Arrow stores a column as one continuous block of memory with the values side by side, so there is far less packing and unpacking to do.

The boundary One function, two ways across

Pickle the default through Spark 4.1

0.00 s

100 rows per batch, one object per value 0 batches

Arrow the default in Spark 4.2

0.00 s

10,000 rows per batch, one buffer per column 0 batches

A replay of real runs. Each lane moves at the speed I measured for that function on Spark 4.2.0 (the median of 9 runs, slowed down so you can watch). The packets are not to scale: an Arrow batch carries 100 times more rows than a pickle batch. Notice that the number of calls to f(x) is the same in both lanes.

One thing does not change, and it is worth saying plainly: your function is still called once for every row, with ordinary Python values. Arrow makes the trip faster. It does not make the function faster, and it does not turn your UDF into a vectorized one.

It was there all along

“Enabled by default” does imply the feature already existed, and it did. The switch is a setting named spark.sql.execution.pythonUDF.arrow.enabled, and it has been around for a while:

The history Three years as an option, then the default

Pickle Arrow (setting on) seconds for the whole query, shorter is better

3.3 Jun 2022

Not there

The setting does not exist yet. Pickle is the only way across.
3.4 Apr 2023

Hidden · off

The setting first appears, marked internal and left out of the documentation.

1.69 s → 0.92 s

1.83×

tested on 3.4.4
3.5 Sep 2023

Public · off

Announced as “Arrow Python UDFs”. You can opt in for a session, or per function with useArrow=True.

1.70 s → 0.93 s

1.82×

tested on 3.5.9
4.0 May 2025

Public · off

Still opt-in.

1.67 s → 1.05 s

1.59×

tested on 4.0.4
4.1 Dec 2025

Public · off

Reworked: values no longer take a detour through pandas on the way in and out.

1.78 s → 0.63 s

2.82×

tested on 4.1.3
4.2 Jul 2026

On by default

The default flips. Every ordinary Python UDF now uses Arrow unless you turn it off.

1.57 s → 0.57 s

2.74×

tested on 4.2.0

The same three functions, on the same machine, on the newest patch release of each version. The setting is switched off and on by hand so that every version runs both paths. Everything else follows from the release: each Spark version brings its own supported versions of PyArrow and pandas.

So this is not a new engine. It is a three-year-old option that the Spark project now trusts enough to switch on for everyone. On any release from 3.5 onward you could already have asked for it:

# for the whole session
spark.conf.set("spark.sql.execution.pythonUDF.arrow.enabled", "true")

# or for one function
@udf(returnType="long", useArrow=True)
def add_one(x):
    return x + 1

In Spark 4.2 you no longer have to. The same setting, set to false, is how you get the old behavior back.

The measurements in the figure show one more thing. The speedup for simple numbers jumped in 4.1, when Spark stopped routing values through pandas on their way in and out. Timestamps went the other way in the same release: Arrow used to be more than three times faster for them, and since 4.1 it is slightly slower than pickle. I have not looked into why.

What I measured

I ran eight small functions on Spark 4.2.0, each one twice: once with the setting off and once with it on. Nothing else changed between the two runs, and both always returned the same answer.

The measurements Eight functions, the same code both ways

Pickle (setting off) Arrow (Spark 4.2 default)

seconds per million rows, shorter is better

Add one to an integer x + 1

2.90×
Square root of a decimal number sqrt(x) * 1.5

2.81×
Arithmetic on four columns a * b + c - d

2.12×
Clean up a short string s.upper().replace("-", "_")

1.36×
Regex rewrite of a string regex rewrite of every number

1.26×
Length of a long string len(s), 256-char strings

0.98×
Sum an array of 16 numbers sum(xs), 16 doubles per row

0.97×
Day of the week from a timestamp ts.weekday()

0.93×

Show the numbers

Function	Types	Rows	Pickle	Arrow	Faster by	Built-in	Python part only	Same answer
Add one to an integer	bigint -> bigint	8 million	1.59 s	0.55 s	2.90×	0.04 s	3.04×	yes
Square root of a decimal number	double -> double	8 million	1.63 s	0.58 s	2.81×	0.03 s	2.92×	yes
Arithmetic on four columns	4 x bigint -> bigint	8 million	2.07 s	0.98 s	2.12×	0.04 s	2.16×	yes
Clean up a short string	string -> string	4 million	1.97 s	1.45 s	1.36×	1.05 s	2.31×	yes
Regex rewrite of a string	string -> string	2 million	1.52 s	1.21 s	1.26×	n/a	n/a	yes
Length of a long string	string -> int	4 million	1.27 s	1.29 s	0.98×	0.72 s	0.96×	yes
Sum an array of 16 numbers	array<double> -> double	2 million	0.97 s	1.00 s	0.97×	0.50 s	0.94×	yes
Day of the week from a timestamp	timestamp -> int	8 million	5.88 s	6.30 s	0.93×	0.11 s	0.93×	yes

Spark 4.2.0 on one machine, 8 cores, 9 timed runs per bar after 3 warm-ups; the bars are medians. Whole query is what a user would feel. Python part only subtracts the time the same query takes with a built-in Spark expression in place of the function, which leaves just the cost of going to Python and back.

A few things stand out:

Simple numeric functions gain the most. Adding one to an integer went from 1.59 seconds to 0.55 seconds for 8 million rows, 2.9 times faster. When the function itself does almost nothing, the trip is the cost.
The more work the function does, the smaller the gain. The two string functions that build a new string improved by about 1.3 times. Switch the figure to “Python part only” and the string cleanup shows 2.3 times: the trip got much cheaper, but generating the strings and running the Python code takes the same time as before.
Some functions do not gain at all. Long strings, arrays and timestamps came out within a few percent of pickle, and slightly behind it. Turning one big value into a Python object costs about the same either way.

Why the gain is not bigger

Arrow has a reputation for being dramatically more efficient, and nearly three times faster at best can look modest next to it. Both things are true. Arrow fixes the trip. It does not touch the work that an ordinary UDF still does on every single row:

Making Python objects. Your function expects an ordinary Python int, str, list or datetime, so each value still has to be turned into one, and each result turned back. For long strings, arrays and timestamps that step costs about as much as pickle did, which is why those three tests came out flat.
Calling the function. One Python call per row, exactly as before.

Arrow’s reputation comes mostly from cases where neither of those happens: pandas UDFs, Arrow UDFs and toPandas(), where a whole column is handed over and processed at once and no per-row Python objects are created. That is where the large multiples come from.

So “Arrow is much more efficient” describes the transport, and “ordinary UDFs get a modest boost” describes the whole query. The new default gives you the first for free. The next section shows what it takes to get the rest.

Where it sits

Here is the new default next to the other ways of running the same logic. The first two bars are the same unchanged function. The others need you to do some work, and that work is exactly what removes the per-row costs above.

In context Free speed, with more left on the table

Ordinary UDF, pickle the old default

1.59 s
Ordinary UDF, Arrow the Spark 4.2 default

0.55 s
pandas UDF rewritten to take a whole pandas Series

0.36 s
Arrow UDF rewritten to take a whole Arrow array

0.33 s
Built-in expression no Python at all

0.04 s

Median seconds for the whole query on Spark 4.2.0, 8 million rows. The first two bars are the same unchanged function. The next two need the function rewritten to work on a whole column at a time, and the last one needs no Python.

For adding one to an integer, the unchanged function went from 1.59 to 0.55 seconds. Rewritten as an Arrow UDF it takes 0.33 seconds, and as a built-in expression 0.04. If a function really matters to your job’s run time, a pandas UDF, an Arrow UDF or a built-in expression is still the better tool. The new default is for everything else: the many small UDFs nobody was going to rewrite.

Before you upgrade

PyArrow has to be there. Spark 4.2 needs PyArrow 18 or newer for this. When I ran 4.2 in an environment without PyArrow, the UDF still worked: Spark quietly used the old pickle path instead.
Edge cases in return types can behave differently. Arrow is stricter than pickle when a function returns something that does not match its declared type. The PySpark documentation describes the differences.
There is an off switch. Set spark.sql.execution.pythonUDF.arrow.enabled to false and you are back on pickle.

How I tested

Everything ran on one machine: an 8-core Intel Xeon E-2286M with 123 GB of memory, Spark in local mode on 8 cores, Python 3.11 and Java 17. For the main results that was Spark 4.2.0 with PyArrow 25.0.1. For the history figure I installed the newest patch release of each version (3.4.4, 3.5.9, 4.0.4, 4.1.3 and 4.2.0) side by side.

Each test builds between 2 and 8 million rows inside Spark, applies the function, and adds up the results so that Spark cannot skip the work. I ran every test 12 times, alternating pickle and Arrow, dropped the first 3 as warm-up and took the median of the other 9. Before trusting a timing I checked the query plan to confirm which path Spark really used: BatchEvalPython for pickle, ArrowEvalPython for Arrow.

These are numbers from one machine, in local mode, on generated data. Treat them as a guide to the shape of the change, not as a promise for your cluster.

Sources

Spark 4.2.0 release notes and SPARK-54555, which made it the default
SPARK-40307, “Introduce Arrow Python UDFs”, from Spark 3.5
Upgrading PySpark, which lists what changed in 4.1 and 4.2
Apache Arrow in PySpark, the user guide