Unable to run find_matches_to_new_records asynchronously

[silvaio]

As part of a Customer Platform project, we're currently assessing Splink. We've managed Splink to train on a dataset with roughly 10M records. Overall have an event-based setup which means we would ideally require a near-real-time lookup of records, based on events as they come in.

Ideally this would require performing linker.inference.find_matches_to_new_records() in an async context for processing the results efficiently. However, it seems this function cannot be executed in a parallel context. Any advice or feedback is greatly appreciated!

[silvaio]

Hi @RobinL .. any thoughts? 🙂

[RobinL]

There's two answers. One is that we're currently working on supporting this better/faster

The second is that, as it stands, you probably need to just set up a new linker each time, load in term frequency tables (if you have tf afjustments), and then call predict()

        linker = Linker(df, "saved_settings.json", db_api=db_api)

            # You should load or pre-compute tf tables for any tables with
            # term frequency adjustments
            linker.table_management.compute_tf_table("first_name")
            # OR
            linker.table_management.register_term_frequency_lookup(df, "first_name")

This can be reasonably faster (latency of like 0.3 seconds). Genearlly it's possible to get that down to about 0.02 seconds with some tricks. Specifically, if you save the SQL generated by Splink and hard code it into your async routine, it's a lot faster