Topic
3 posts.
I wanted to see how Kong handles traffic that has nothing to do with AI, so I followed Microsoft's HDInsight article on connecting a local Jupyter to Spark through Livy, and put Kong where the HDInsight gateway sits.
In my last post I left one problem unexplained: ordinary Python UDFs that never finished on cached or Parquet data. This weekend I took a closer look. Here is how I narrowed it down, what was going on, and where the fix stands.
Spark 4.2 changes how ordinary Python functions trade data with the JVM. The option has been there since Spark 3.4. Here is what it does, how it got here, and what I measured.