Is there any way that we can save linker object in spark after training and predicting

[rapidAmbakar]

I have just predicted on a sparkdata frame which take roughtly around 1.5 hours. Is the return object coming from linker.predict() is just limited to save it to dataframe? or is their any any which i dont have to predict every time.

[RobinL]

To save the outputs you will need to save linker.predict() to a dataframe using:

df_predict.as_pandas_dataframe()

or

df_predict.as_spark_dataframe().write.parquet("output_folder") 

To make it run faster, you could consider using tighter blocking rules, see
https://moj-analytical-services.github.io/splink/demos/02_Blocking.html