Help with missingness_chart()

[pkky22]

Hi,

Please bear with my beginner level programming skill and help on the following.

I am trying to replicate splink demo available on github. I am successfully doing many things from the demo except charting. To start with, missingness_chart() just spits json looking output rather than a chart. My working code is:

import pandas as pd 
import altair as alt
alt.renderers.enable('default')

df = pd.read_csv(r~\fake_1000_from_splink_demos.csv")
from splink.duckdb.duckdb_linker import DuckDBLinker
linker = DuckDBLinker(df)
linker.missingness_chart()

output:

{'config': {'view': {'continuousWidth': 400,
   'continuousHeight': 300,
   'width': 400},
  'axis': {'labelFontSize': 11}},
 'title': '',
 'layer': [{'mark': 'bar',
   'encoding': {'color': {'type': 'quantitative',
     'field': 'null_proportion',
     'legend': {'format': '.0%'},
     'scale': {'range': 'heatmap'},
     'title': 'Missingness'},
    'tooltip': [{'type': 'nominal', 'field': 'column_name', 'title': 'Column'},
     {'type': 'quantitative',
      'field': 'null_count',
      'format': ',.0f',
      'title': 'Count of nulls'},
     {'type': 'quantitative',
      'field': 'null_proportion',
      'format': '.2%',
      'title': 'Percentage of nulls'},
     {'type': 'quantitative',
      'field': 'total_record_count',
      'format': ',.0f',
      'title': 'Total record count'}],
    'x': {'type': 'quantitative',
     'axis': {'format': '%', 'title': 'Percentage of nulls'},
     'field': 'null_proportion'},
    'y': {'type': 'nominal',
     'axis': {'title': ''},
     'field': 'column_name',
     'sort': '-x'}},
   'title': 'Missingness per column out of 1,000 records'}],
 'data': {'values': [{'null_proportion': 0.0,
    'null_count': 0,
    'total_record_count': 1000,
    'column_name': 'None'},
   {'null_proportion': 0.0,
    'null_count': 0,
    'total_record_count': 1000,
    'column_name': 'unique_id'},
   {'null_proportion': 0.10000002384185791,
    'null_count': 100,
    'total_record_count': 1000,
    'column_name': 'first_name'},
   {'null_proportion': 0.07700002193450928,
    'null_count': 77,
    'total_record_count': 1000,
    'column_name': 'surname'},
   {'null_proportion': 0.0,
    'null_count': 0,
    'total_record_count': 1000,
    'column_name': 'dob'},
   {'null_proportion': 0.10900002717971802,
    'null_count': 109,
    'total_record_count': 1000,
    'column_name': 'city'},
   {'null_proportion': 0.1119999885559082,
    'null_count': 112,
    'total_record_count': 1000,
    'column_name': 'email'},
   {'null_proportion': 0.0,
    'null_count': 0,
    'total_record_count': 1000,
    'column_name': 'group'}],
  'name': 'data-0e7bce5a1d2f132e282789d6ef7780fe'},
 '$schema': 'https://vega.github.io/schema/vega-lite/v4.8.1.json'}

Any ideas to fix the issue here?

Thanks!

[RobinL]

When using Splink to generate a chart, it will by default render the chart only in interactive environments. These include Jupyter notebooks or the interactive IPython kernel within Visual Studio Code. However, if you're running them as a Python script, the charting functions will only produce a JSON representation of the chart.

If you wish to save the chart as an HTML file that can be viewed offline, assign the chart to a variable. Following this, utilize the save_offline_chart function. More information about this function can be found at the following URL:

splink/splink/linker.py

Line 2436 in aba0f5f

>>> from splink.charts import save_offline_chart

Also note that if Altair is not installed, all charting functions will return the json representation rather than the Altair chart

[mashby1966]

I have version 5.0.1 installed.

BTW - (not sure if it is relevant) if I use the line "save_offline_chart(c.spec,"test_chart.html",overwrite=True)"` as suggested, I get a "'dict' object has no attribute 'spec'" error however if I use "save_offline_chart(c,"test_chart.html",overwrite=True)" it works and I can view the test_chart.html in a browser.

Thanks

[RobinL]

Thanks I wondered whether that maybe the case. There have been quite significant changes in the Altair 5 release. It may be that downgrading Altair to a 4.x version helps?
pip install altair==4.2.2
If that works we will want to update our dependencies to specify Altair < 5 for the moment, although in the longer run there's an open issue for us to upgrade everything to Altair 5

[mashby1966]

Perfect, I have just done exactly that before I saw your reply.
The missingness chart is now rendering in VS Code and also the c.spec error is gone.

Thanks

[RobinL]

Thanks for confirming!

[RobinL]

This is now fixed by #1306 so will be included in the next release. Specifically once released apip install splink will install a version 4 of Altair. Thanks again for your help 🙏

[pkky22]

Thanks for the information. I tried to get Splink generate and render chart in both notebooks and the interactive IPython kernel within Visual Studio Code, but it only produced JSON outputs. In Jupyter notebook, save_offline_chart works for me as I can refer to the HTML outside of the notebook now. Thank you! From: Robin Linacre ***@***.***> Sent: Tuesday, May 23, 2023 2:36 AM To: moj-analytical-services/splink ***@***.***> Cc: Yadav, Pramod K ***@***.***>; Author ***@***.***> Subject: Re: [moj-analytical-services/splink] Help with missingness_chart() (Discussion #1247) You don't often get email from ***@***.******@***.***>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> When using Splink to generate a chart, it will by default render the chart only in interactive environments. These include Jupyter notebooks or the interactive IPython kernel within Visual Studio Code. However, if you're running them as a Python script, the charting functions will only produce a JSON representation of the chart. If you wish to save the chart as an HTML file that can be viewed offline, assign the chart to a variable. Following this, utilize the save_offline_chart function. More information about this function can be found at the following URL: https://github.com/moj-analytical-services/splink/blob/aba0f5f2ba40958ee642f69df836c1e53cacf9c1/splink/linker.py#L2436 - Reply to this email directly, view it on GitHub<#1247 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BAAEHKI4A7NJ7ZPGOQAHHHDXHRSHFANCNFSM6AAAAAAYJUHC4Y>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

[staropram]

I actually had a similar issue which was related to the vegalite extension. After some fiddling around I actually couldn't get it to work without wrapping the chart output in a VegaLite(chart) call and importing from vega import VegaLite.

Also, I wrote a wrapper to work from ipython:

import uuid

def viewSplinkChart(splinkChart):
    splinkChart['width'] = 1000
    splinkChart['height'] = 500
    out = "tmp/" + str(uuid.uuid4()) + ".html"
    save_offline_chart(splinkChart, out,overwrite=True)
    webbrowser.open(out,new=1)

[pkky22]

My bad, I should have mentioned before - to save the offline chart, I did use "save_offline_chart(c,"test_chart.html",overwrite=True)" From: mashby1966 ***@***.***> Sent: Wednesday, June 7, 2023 3:56 PM To: moj-analytical-services/splink ***@***.***> Cc: Yadav, Pramod K ***@***.***>; Author ***@***.***> Subject: Re: [moj-analytical-services/splink] Help with missingness_chart() (Discussion #1247) I have version 5.0.1 installed. BTW - (not sure if it is relevant) if I use the line "save_offline_chart(c.spec,"test_chart.html",overwrite=True)"` as suggested, I get a "'dict' object has no attribute 'spec'" error however if I use "save_offline_chart(c,"test_chart.html",overwrite=True)" it works and I can view the test_chart.html in a browser. Thanks - Reply to this email directly, view it on GitHub<#1247 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BAAEHKLSDBR7V6BKONVKBP3XKDTFVANCNFSM6AAAAAAYJUHC4Y>. You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

[RobinL]

Yeah, it's the recently released version 5 of Altair that's causing the problem. This is now fixed by #1306 so will be included in the next release. Specifically once released apip install splink will install a version 4 of Altair.

[mahakti]

Hi Can anyone help me resolve below error please?

Whenever I am running below code in jupyter-
model = linker.model
model.all_charts_write_html_file(filename=f"../OUT/charts/splink_charts.html", overwrite=True)

I am getting error like -
UnicodeEncodeError Traceback (most recent call last)
Cell In[25], line 2
1 model = linker.model
----> 2 model.all_charts_write_html_file(filename=f"../OUT/charts/splink_charts_{day}.html", overwrite=True)

File ~\PycharmProjects\pythonProject1\venv1\lib\site-packages\splink\model.py:257, in Model.all_charts_write_html_file(self, filename, overwrite)
253 fmt_dict["proportion_of_matches"] = self.current_settings_obj[
254 "proportion_of_matches"
255 ]
256 with open(filename, "w") as f:
--> 257 f.write(template.format(**fmt_dict))

File C:\Users\2202055\AppData\Local\Programs\Python\Python39\lib\encodings\cp1252.py:19, in IncrementalEncoder.encode(self, input, final)
18 def encode(self, input, final=False):
---> 19 return codecs.charmap_encode(input,self.errors,encoding_table)[0]

UnicodeEncodeError: 'charmap' codec can't encode character '\u2212' in position 23084: character maps to

But when I am running code for getting other charts like below, charts are getting displayed without any error in jupyter-
model.probability_distribution_chart()
model.bayes_factor_chart()

Why are individual chart's code running properly but code for making all charts giving error? How can I resolve it?

I am uisng altair version 4, alt.renderers.enable("default")

[RobinL]

I'm afraid Splink 2.x is now no longer supported, since 3.x is now almost a year old. I think this issue is now fixed, but only in 3.x, see here #562

[mahakti]

ok so does this means that there is no way I can save charts using splink2? Any workaround? Please suggest

[RobinL]

If you take a look at the changes in the pr and make similar changes to your copy of Splink 2, then it might work. But generally we recommend upgrading to version 3. Also worth noting these charts do usually work in Splink 2 (they're not always broken, just occasionally) - it could be something to do with the encoding of your data.

[mahakti]

Thanks I will try as you suggested. Also it would be great help if you can tell whether method for linking two datasets is present in version 2 or not? Can I give "link_type": "link_only" in settings, will it work? How to pass both datasets as argument to splink 2? Is there any documentation of version 2 present?

[RobinL]

There's no documentation site for Splink 2 I'm afraid. The best place to look would be here, specifically this notebook for linking.

But I would recommend updating your code to Splink 3 if at all possible, it's faster, better supported, and has much more functionality