Add --threshold option

This adds:

• The ability to use a simple core threshold (#49)
• Use external clusters in generate viz mode (#55)

Other:

• Add scripts into package (#56)

Better reference selection

The main reason for this release is the fix in #50. Although not necessarily a problem, in some populations the reference picking step can lead the pruned network to appear inconsistent with the clusters, giving warning messages when assigning new queries. This is fixed here.

Bug fixes:

• Fixes to --generate-viz mode (#44).
• Problems using relative paths (#48).
• Ensuring consistent network and cluster definitions when choosing references (#50).

Other:

• Update travis-CI to use correct DISPLAY for matplotlib
• Add Azure pipelines CI

Generate viz mode

The main feature added is --generate-viz mode.

New features:

• Allow visualisation to be generated independently #44
• Add extra output to STDERR for GMM fits #45

Bug fixes:

• Fix in plot of DBSCAN introduced in previous version #45
• Fix README rendering in PyPI
• Fix readthedocs build (details)
• Fix to genome length check when inputs not present

Pathogenwatch API

This release is mostly to create a function to assign samples, usable by an API for pathogenwatch.

Other changes

New features:

• Added script to generate the microreact output (yet to be fully exported)

Bug fixes

• Fixes to --use-model mode
• Guard against huge memory use in making plots of refined fits with very large datasets
• Correct error message when fit fails #41

Post-paper updates

Adding some changes and fixes that have arisen from more people testing out on their data!

New features:

• Add basic quality control for assemblies input (#37, #42).
• Generate a network and cluster assignment by applying a fitted model to a distance database (--use-model).
• Added script to calculate Rand distances between cluster assignments (calculate_rand_indices.py)
• Added script to extract .graphml files for all clusters (extract_components.py).

Bug fixes:

• More general parsing of epiData for --microreact (#35).
• Correct paths in output prefix with mash sketch.
• Removed the zero level contour in distance plots (#38).

Other:

• Updated docs (troubleshooting, installation, README.md)
• Removed n subscript on pi on x-axis of plots.
• Updated networkx version for pip (#39).
• Catch errors when curve fit fails (#41).
• Travis-CI uses conda rather than pip to install dependencies.

Second revision

This release contains an important bug fix and some changes to the output files

New features:

• Available as a conda package (#31)
• Plots are now plotted in terms of pi_n and a, rather than in the arbitrary scaled space (#32)
• Easy run mode now uses HDBSCAN and fit refinement, a more intuitive default (#34)
• Min and max k-mer sizes set as noted in paper

Bug fixes:

• Clusters are output in reverse frequency order, in every mode (#33)
• Use of databases with reference only (i.e. without --full-db) work with future --assign-query commands (#35)
• Parsing of CSV column headers for microreact output fixed for some cases

Python version requirement relaxing

Small change to setup.py for compatibility with bioconda

Revised paper

This release contains features and fixes in response to review comments in the manuscript, as well as other raised issue fixes.

New features:

• Arbitrary cluster definitions can be merged with the clusters found using --external-clustering (#30)
• A new, simpler, example has been added to the documentation
• All plots are clearer and show the unscaled distances a and pi_n (#32)

Bug fixes:

• Check distance file provided when using --assign-query (#29)
• Ensure python3 is being used (v2.7 unsupported)
• General tidying of code from pylint errors and warnings

Updating databases fixes

New features:

• Cluster IDs now start from 1 rather than 0 (#26)
• Added various error checking (#27)
• Report merged clusters when using --assign-query
• Sort _clusters.csv by cluster ID, rather than sample name

Bug fixes:

• --update-db works with reference only databases (#28)

python 2/3 writing fixes

Bug fixes:

• An error in stderr writing models.py which causes problems with some versions of python