Error on PySpark deduping example: no jars directory in splink

[mmagoffin-sd]

I just moved from using splink with DuckDB to using it with Spark. I tried to follow the first notebook cell, but I get the following error:

I tried removing the problem line, but then I get a different error:

So, I looked into what path the similarity jar location gave me:
'd:\Users\mmagoffin\AppData\Local\hatch\env\virtual\ds-misc-mmagoffin-S1xDGiIH\ds-misc-mmagoffin\lib\site-packages\splink\jars/scala-udf-similarity-0.0.9.jar'

However, looking inside of the splink directory there, the jars directory does not exist. I also tried just copying the Jaro-Winkler jar to the working directory, but I got the same error.

Any clue how I can fix this?

More context if it helps: I'm on Windows 10, Spark 3.3.0, and splink 3.4.1

And update: even though I've been able to fix the secondary jars directory problem, because I'm unable to set a Checkpoint directory here, I'm unable to run linker.estimate_u_using_random_sampling(target_rows=5e5) as I get a checkpointing error within the spark_linker script. Its even more confusing as setCheckpointDir is successful in creating a directory and writing to it, so I'm sure where the error is.

[mamonu]

Can you either

• put the jar in the path :
  d:\Users\mmagoffin\AppData\Local\hatch\env\virtual\ds-misc-mmagoffin-S1xDGiIH\ds-misc-mmagoffin\lib\site-packages\splink\jars/

• or change the path (where currently you store the output of the function similarity_jar_location)
  to your working directory.

Also a warning: when you define settings for the spark session you need to restart your kernel in
order for these to be updated. If you just change the path and run the cell you dont get the jars loaded into the session

[RobinL]

I think the setCheckpointDir is to do with the difference between the demo setup (in Spark local mode) the Spark setup you're using. I think you probably need to set the path to a temporary directory accessible to your cluster - probably somewhere in your hadoop file system.

[mmagoffin-sd]

I'm currently starting a SparkSession with configuring the master to "local[4]"- should that be needing a file path to something in my Hadoop file system?

[RobinL]

It just needs to be any path to which Spark can write temporary files, wherever that would be in your setup

[mmagoffin-sd]

hmm must be an issue with my spark set up then, thank you!

[mmagoffin-sd]

update: to be able to use setCheckpointDir I had to download the x64 C++ file: https://www.microsoft.com/en-us/download/details.aspx?id=26999 and run the command [your hadoop dir]\bin\winutils.exe chmod -R 777 [your path to your tmp dir] in whatever your working directory is