Splink 4.0.0: How to debug value error in training.py?

[aalexandersson]

On my test data, this code worked in Splink 3.9.15:

deterministic_rules = [
    "l.ssn = r.ssn"
]
linker.estimate_probability_two_random_records_match(deterministic_rules, recall=0.85)

The output was:

Probability two random records match is estimated to be 1.42e-06.
This means that amongst all possible pairwise record comparisons, one in 705,054.21 are
expected.

But this equivalent "training" code fails in Splink 4.0.0:

deterministic_rules = [
     block_on("ssn"),
]
linker.training.estimate_probability_two_random_records_match(deterministic_rules, recall=0.85)

The error message is:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "V:\Testing\Splink_2025\.venv\Lib\site-packages\splink\internals\linker_components\training.py", line 120, in estimate_probability_two_random_records_match
    raise ValueError(
ValueError: Deterministic matching rules led to more observed matches than is consistent with supplied recall. With these rules, recall must be at least -0.00.

This is the affected code in training.py:

        if num_observed_matches > num_total_comparisons * recall:
            raise ValueError(
                f"Deterministic matching rules led to more "
                f"observed matches than is consistent with supplied recall. "
                f"With these rules, recall must be at least "
                f"{num_observed_matches/num_total_comparisons:,.2f}."
            )

I arbitrarily tried several other "deterministic_rules", but the error message is the same in Splink 4.0.0. How can I easily calculate num_observed_matches and num_total_comparisons to be able to debug the error? I tried to re-use code from lines 103-119 but failed.

[RobinL]

Thanks for the report.

I'm not sure. The error message suggests there are no observed matches since that appears to be the only way you can get a value of -0.0 from f"{num_observed_matches/num_total_comparisons:,.2f}." but it's a bit unclear to me why the
if num_observed_matches > num_total_comparisons * recall: if statement has been triggered.

I would start by adding these lines

        print(f"num_observed_matches: {num_observed_matches}")
        print(f"num_total_comparisons: {num_total_comparisons}")
        print(f"recall: {recall}")
        print(f"estimated matches: {num_total_comparisons * recall}")

to here:

splink/splink/internals/linker_components/training.py

Line 117 in 89eae40

num_total_comparisons = summary_record["cartesian"]

Here's a testing script I wrote to try and produce a reprex (minimal reproducible example), but I can't reproduce the error and it seems to give the right answer:

testing script

import pandas as pd

import splink.comparison_library as cl
from splink import DuckDBAPI, Linker, SettingsCreator, block_on, splink_datasets

db_api = DuckDBAPI()

# fmt: off
data_1 = [
    {"unique_id":1, "first_name": "Alice", "surname": "Smith", "dob": "1970-01-01", "city": "London"},
    {"unique_id":2, "first_name": "Bob", "surname": "Jones", "dob": "1984-02-02", "city": "London"},
    {"unique_id":3, "first_name": "Dave", "surname": "Smith", "dob": "1970-01-02", "city": "London"},
]

data_2 = [
    {"unique_id":4, "first_name": "Alice", "surname": "Smith", "dob": "1971-05-05", "city": "London"},
    {"unique_id":5, "first_name": "Bob", "surname": "Jones", "dob": "1990-10-10", "city": "London"},
]
# fmt: on

df_1 = pd.DataFrame(data_1)
df_2 = pd.DataFrame(data_2)
df_all = pd.concat([df_1, df_2])

settings = SettingsCreator(
    link_type="dedupe_only",
    comparisons=[
        cl.ExactMatch("first_name"),
        cl.ExactMatch("surname"),
        cl.ExactMatch("dob"),
        cl.ExactMatch("city"),
    ],
)

linker = Linker(df_all, settings, db_api)

deterministic_rules = [
    block_on("first_name"),
]
linker.training.estimate_probability_two_random_records_match(
    deterministic_rules,
    recall=1.0,
)


settings = SettingsCreator(
    link_type="link_only",
    comparisons=[
        cl.ExactMatch("first_name"),
        cl.ExactMatch("surname"),
        cl.ExactMatch("dob"),
        cl.ExactMatch("city"),
    ],
)

linker = Linker([df_1, df_2], settings, db_api)

linker.training.estimate_probability_two_random_records_match(
    deterministic_rules,
    recall=1.0,
)

settings = SettingsCreator(
    link_type="link_and_dedupe",
    comparisons=[
        cl.ExactMatch("first_name"),
        cl.ExactMatch("surname"),
        cl.ExactMatch("dob"),
        cl.ExactMatch("city"),
    ],
)

linker = Linker([df_1, df_2], settings, db_api)

linker.training.estimate_probability_two_random_records_match(
    deterministic_rules,
    recall=1.0,
)



settings = SettingsCreator(
    link_type="link_and_dedupe",
    comparisons=[
        cl.ExactMatch("first_name"),
        cl.ExactMatch("surname"),
        cl.ExactMatch("dob"),
        cl.ExactMatch("city"),
    ],
)

linker = Linker([df_1, df_2], settings, db_api)

deterministic_rules = [
    block_on("dob"),
]
linker.training.estimate_probability_two_random_records_match(
    deterministic_rules,
    recall=0.85,
)


settings = SettingsCreator(
    link_type="link_and_dedupe",
    comparisons=[
        cl.ExactMatch("first_name"),
        cl.ExactMatch("surname"),
        cl.ExactMatch("dob"),
        cl.ExactMatch("city"),
    ],
)

linker = Linker([df_1, df_2], settings, db_api)

deterministic_rules = [
    block_on("city"),
]
linker.training.estimate_probability_two_random_records_match(
    deterministic_rules,
    recall=0.85,
)

[aalexandersson]

Your test data gives me the same error message on my end:

----- stdout -----
num_observed_matches: 376302
num_total_comparisons: -2147483648
recall: 0.85
estimated matches: -1825361100.8
------------------

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [1], in <cell line: 0>()
     34 linker = Linker([df_1, df_2], settings, db_api)
     36 deterministic_rules = [
     37     block_on("ssn"),
     38 ]
---> 39 linker.training.estimate_probability_two_random_records_match(
     40     deterministic_rules, recall=0.85
     41 )

File V:\Testing\Splink_2025\.venv\Lib\site-packages\splink\internals\linker_components\training.py:124, in LinkerTraining.estimate_probability_two_random_records_match(self, deterministic_matching_rules, recall, max_rows_limit)
    121 print(f"estimated matches: {num_total_comparisons * recall}")
    123 if num_observed_matches > num_total_comparisons * recall:
--> 124     raise ValueError(
    125         f"Deterministic matching rules led to more "
    126         f"observed matches than is consistent with supplied recall. "
    127         f"With these rules, recall must be at least "
    128         f"{num_observed_matches/num_total_comparisons:,.2f}."
    129     )
    131 num_expected_matches = num_observed_matches / recall
    132 prob = num_expected_matches / num_total_comparisons

ValueError: Deterministic matching rules led to more observed matches than is consistent with supplied recall. With these rules, recall must be at least -0.00.

In case it helps, I attach my requirements.txt file.

Your SQL code on your test data works fine on my end. The is the row_count output:

┌───────────┬───────────┐
│ row_count │ match_key │
│   int64   │  varchar  │
├───────────┼───────────┤
│    377511 │ 0         │
└───────────┴───────────┘

Your SQL code on my test data (with data frames df1 and df2 rather than df_1 and df_2) results in multiple syntax errors. Maybe some simple typo, wrong linebreak or similar beginner copy-paste error somewhere in my code?

>>> CREATE TABLE __splink__df_count_cumulative_blocks_07f6a1693 AS 
  File "<string>", line 1
    CREATE TABLE __splink__df_count_cumulative_blocks_07f6a1693 AS
           ^^^^^
SyntaxError: invalid syntax
>>> WITH
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'WITH' is not defined
>>> 
>>> __splink__df_concat as (
  File "<string>", line 1
    __splink__df_concat as (
                        ^^
SyntaxError: invalid syntax

[RobinL]

Thanks, I'll take a look again tomorrow. At least we have something close to a a reprex now. Thank for the requirements file that'll be very helpful in testing this. What version of python are you running? Tomorrow I'll try your requirements on Mac and if I still can't reproduce I'll try on Windows. Hopefully getting close to pinning this down now

[aalexandersson]

I use Python 3.11.0. The original data are the pseudopeople data from Abraham Flaxman, which I accessed here. If needed, I could ask him for permission to share the derived data frames df1 and df2 with you privately.

[RobinL]

I have tried my test script above (the one that creates ssn_1 = np.random.randint(1, 670251, size=670251) etc.) on

• Mac OS Sonoma 14.3, Python 3.11.8, with your requirements.txt.
• Windows 11, Python 3.10, with a vanilla duckdb install. (I didn't use the whole of requirements.txt because it's pretty heavyweight and there's not much space on my old windows laptop)

I can't replicate the error on either

What happens if you create a fresh venv, install only splink, and try the test script. Do you still get the same error? How about on a different version of Python?

A bit stumped on this one!

If you run the test script in VS code debugger (Debug -> Start Debugging), then it'd be useful narrow down there the negative number first appears

[aalexandersson]

The VS Code debugger (on the existing venv) seems not to add any information, see the screenshot. I used VS Code version 1.92.1 (current) for Windows:

A fresh install on another Windows 10 computer of Python 3.11.9 with splink, numpy and pandas in venv ran your test script fine. Finally some progress on my end:

Update: The issue has been resolved on my main work computer using my test data. I updated Python from 3.11.0 to 3.11.9, which did not solve the issue. However, a fresh venv solved the issue. I attach the updated requirements.txt.

I can no longer save offline charts to PNG for PDF but I will open a separate issue 2335 (now resolved) for that. Thanks for the help!

[fscholes]

Not really to add a solution but I was also experiencing this on an environment that previously upgraded from Splink 3 to Splink 4, and the issue was also fixed with a fresh venv. I am running Python 3.12. It would be interesting to see if this affects other IDEs, not just VS Code on Windows (but I use VS Code myself)

[RobinL]

Thanks both for the updates and the solution!

[aalexandersson]

The issue affected both my IDEs: RStudio and VS Code. I'm guessing instead that the dependency issue is related to numpy. Previously, I used numpy==1.26.4 because pseudopeople==1.1.1 requires numpy<2.0.0. In the fresh venv I have numpy==2.0.1 but no longer pseudopeople installed. I will try to find time to test this.