Registering prediction parquet(s) for clustering with Splink 4

[AdamChapnik1]

Hi!

I was able to do this with Splink 3 but it seems like the functionality has been removed from the docs for Splink 4 so that's why I'm asking here. Basically I have run the prediction step for a few models and saved those predictions as parquet files. I want to create a table with DuckDB which is the UNION ALL (BY NAME) of those parquet files, register that DuckDB table as a Splink data table, and then run linker.clustering.cluster_pairwise_predictions_at_threshold. How would you do this with Splink 4? Thanks!

[RobinL]

Is this what you want?

splink/splink/internals/linker_components/table_management.py

Line 147 in 116e2ff

def register_table_predict(self, input_data, overwrite=False):

That function should be available at linker.table_management
It looks like it's not showing up in the docs because we've forgotten to write a do string, thanks for bringing to our attention!

Edit: I've just added a docstring here
#2328

[AdamChapnik1]

Yep, that's it, thank you!

[AdamChapnik1]

Hi @RobinL, sorry I was thinking this was the answer but I'm not getting the code to work. Maybe I'm just misunderstanding.

This code isn't reproducible but this is basically what I'm doing:

linker = Linker(
    ['dfA', 'dfB'],
    settings = 'model.json',
    db_api = DuckDBAPI(con)
               )

con.execute(
    """
    CREATE OR REPLACE TABLE predictions AS 
    SELECT * FROM
        (SELECT * FROM 'predictions_1.parquet'
        UNION ALL BY NAME
        SELECT * FROM 'predictions_2.parquet');
    """
       )

# in Splink 3 I was able to do this instead of the following two lines
# predictions = DuckDBDataFrame("predictions", "predictions", linker)

predictions = pd.read_parquet('predictions.parquet') 
linker.table_management.register_table_predict(predictions, overwrite = True)

clusters = linker.clustering.cluster_pairwise_predictions_at_threshold(predictions, threshold_match_probability = 0.75)

which gives the error AttributeError: 'DataFrame' object has no attribute 'metadata'. If I move things around I can get it to give the very similar error AttributeError: 'str' object has no attribute 'metadata'. I assume I'm making a very small syntax mistake or not understanding the correct order of operations. The help is much appreciated!

[RobinL]

On my phone so difficult to write code, but the linker.table_management function returns a SpminkDataFrame, and it's this SplinkDataFrame that should be passed to linker.inference.predict() rather than the original predictions

[ADBond]

So in terms of code, that is the last few lines should be:

predictions_pd = pd.read_parquet('predictions.parquet') 
predictions_sdf = linker.table_management.register_table_predict(predictions_pd, overwrite = True)

clusters = linker.clustering.cluster_pairwise_predictions_at_threshold(predictions_sdf, threshold_match_probability = 0.75)

[AdamChapnik1]

Oh that's very simple, I should have realized that. Thanks for the help!