WARNING:splink.internals.spark.database_api:Unable to load custom Spark SQL functions such as jaro_winkler

[MariaWerner-Wien]

Hello,

I am trying to run Splink on PySpark but every time I get one and the same error. I tried to implement the example in the tutorial provided in the Splink documentation both in version 4 and version 3. In particular, I tried:

• v4.0.2
• v4.0.0
• v3.9.13
• v3.9.6

I tested it in:

• Jupyter Notebook on Dataproc cluster in GCP - Vertex AI, and
• Jupyter Notebook on Google Colab

I get the following error:

WARNING:splink.internals.spark.database_api:Unable to load custom Spark SQL functions such as jaro_winkler from the jar that's provided with Splink.
You need to ensure the Splink jar is registered.
See https://moj-analytical-services.github.io/splink/demos/example_simple_pyspark.html for an example.
You will not be able to use these functions in your linkage.
You can find the location of the jar by calling the following function:
from splink.spark.jar_location import similarity_jar_location

Full error:
Can not load class uk.gov.moj.dash.linkage.JaroSimilarity, please make sure it is on the classpath.

The GitHub link provided in the error message doesn't work.

It seems the similarity_jar_location is not located in spark.jar_location. Instead, I found it in internals:
from splink.internals.spark.jar_location import similarity_jar_location.

Nonetheless, the error persists.

Any suggestions how to overcome this issue would be highly appreciated. I need to run Splink on a Spark cluster.

Error was: [UNRESOLVED_ROUTINE] Cannot resolve function `JARO_WINKLER` on search path [`system`.`builtin`, `system`.`session`, `spark_catalog`.`default`].; line 38 pos 11

Many thanks in advance!

[Priebe1]

Hi,
did you try to explicitly add the jar to your Spark session? E.g. spark.sql(f"ADD JAR {similarity_jar_location()}")

[MariaWerner-Wien]

Hi Priebe1,

Thanks a lot for your reply. I think I add jar to the spark session but use the approach suggested in the Splink spark example

When I try the script you suggest, I got an error saying that there is no 'sql' attribute in the spark backends.

[Priebe1]

Hi,
maybe something is wrong with your Spark setup. The spark session should be an instance of https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.SparkSession.html, conf should be an instance of https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.SparkConf.html. You could also check the SparkUI to ensure that the correct config values and jars are set