Using Splink for adress Matching

[machtelinck]

Hello , I'm actually doing a final project of my School . Somehow at my company I was confronted with matching things without true keys so to speak , matching data from outside . One of my core task is to match adress my company has with BAN adress "Base adresse nationale" which plan to be something big for adress in the next future . The issue being , it is mayor that writes adresses . So I came confronted with the case of a street name like "rue du 18 mai 1948" which has more than 8 way to be written across city . Even more annoying , some city have "impasse pasteur" and "rue pasteur" which are different . But in a lot of case the road type can be switched . So I implemanted something in python which use different metrics LCS ratio , ratio , Jaro-Winkler (wasn't of much use), Levenshtein, Damereau Levenshtein .
all on the same column , because i wanted the other column to be strictly equal .
Moreover one of the terms of my project is to use and fine tune a ml model . Somehow i can tell than edit distance are some kind of hyperparameter .
Anyway i would like to ask , what would be a way to do that , is splink really adapted to such case or was I mistaken ?
is there a way to somehow use distinct metrics for the same columns (maybe duplicate them in both dataset to join) , How to say to the splink model than there is only one good adress to be associated with ?
(BAN has some missing data but still in my usecase over 3M rows , while the company data , thus that aren't merged by a simple joint are about 300K ) .
I would be personally thrilled if it is easily feasible to create our own distance metrics like a SNN similarity metrics (my first idea was to use Splink to create labels , reviews them to avoid mistake, and create the distance )
I would like any idea on the matter. And i would like to know if this is a good idea to use API geocoding on both dataset to be linked to have a radius of search .

Sorry for the long message , but i felt likethis wasn't trivial and i needed to further explain my case

[RobinL]

If you only have an address and no other column, Splink is unlikely to work terribly well for you because it violates the conditional indepenedence assumption of the Fellegi Sunter model. it's essentially like having a single column of data - see here

[machtelinck]

Thanks for the answer ,this is definitely something I have to keep in mind .
From memory I have | Street name | number of the adress | city | zip code | rep
Number of the adresss should be equal of course, but maybe Street name and city can be two different column not very much correlated each other , cities from ban and from my company can somehow be written differently , zip code too , but lat long will almost never be the same.
So three column or four (street name , lat , long , maybe city)
can make Splink usable in that case theorically ?
(I assumed Splink could work there first because to me adress is an entity , I might be mistaken) Weirdly at the point of having lat and long , there is way less interest in using Splink even more if this is done by the same geocoder

[RobinL]

Address is an unusual type of entity in that almost all columns are strongly correlated. The lat/long comparison would typically be used in the case where you have more information e.g. a business name and an address or a person's name and an address.

[RobinL]

For any future readers, there's an example here of high performance, accurate address matching using Spink:
https://github.com/RobinL/address_matching_example/

It's designed with UK addresses in mind but could be used for any addresses with a few tweaks