Skip to content

Commit

Permalink
Add a proposed mechanism for tracking known "quality" implementations.
Browse files Browse the repository at this point in the history
There is obvious slight overlap here with our existing list of
implementations on the web page -- note though that we have loose (or
no) criteria for listing implementations on the page.

Here instead we wish to specifically raise languages for which we know
no good implementation to exist, despite the language being popular or
widespread, if any.

What's here does *not* yet render this data anywhere, nor does it
implement the dynamic API calling mentioned in the body (for retrieving
statistics from any listed repository).

What it does do is identify a simple initial list of languages (from
TIOBE and GitHub's own lists), and propose a format for tracking data
points about implementations meeting the proposed criteria.

The initial goal is to get feedback on this template, and to solicit
help in filling it in for these languages.

Refs: #418
  • Loading branch information
Julian committed Nov 28, 2023
1 parent 48e6d9c commit 809489a
Showing 1 changed file with 74 additions and 0 deletions.
74 changes: 74 additions & 0 deletions programs/cross-language-coverage/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# Cross-language Implementation Quality

This directory collects metrics to assist in confirming the presence of (at least one) "high quality" implementations across programming languages.

Absence from this directory is not meant to be a slight against anyone or anyone's work certainly.

The goals are to identify gaps in the ecosystem that might need assistance, and ultimately to help the JSON Schema ecosystem and its users in finding solid, compliant implementations.

The intial collection of TODO programming languages which appears below is simply a starting point, based largely on:

* the [GitHub popularity list](https://octoverse.github.com/2022/top-programming-languages)
* the [TIOBE index](https://www.tiobe.com/tiobe-index/)

where it combines languages at the top of the popularity list with those on the rising / fastest growing list.

You can help this exercise!
For each language, we wish to track whether one or more known implementations exist which:

* Fail less than 2% (around 30) tests from the official JSON Schema test suite
(This 2% number was chosen simply because it represents the dividing point between one cluster of implementations supported by Bowtie.
Above this number is a collection of implementations failing larger orders of magnitude of tests).
* Supports one of the most recent 2 releases of JSON Schema
* Has had a release of itself in the last 3 years (which is around the timeline on which JSON Schema itself has seen its last release)
* Has had some human-authored commit in the last year
* Does not have explicit indication, by its maintainer or documentation, that it is not production ready

and then highlight any languages for which this is not the case.

For languages that *do* have one or more such implementations, we then wish to include:

* the name and link to the implementation
* the date of its last commit
* the date of its last release
* the number of open security vulnerabilities it has, if known
* the number of open issues it has on its issue tracker
* the number of open pull requests it has
* whether it implements some or all of the interfaces specified on our Common Interfaces page (https://json-schema.org/implementers/interfaces),
and if so, which

A number of the above data points are likely to change often.
We therefore assume only some of the above should be tracked in this directory, and the remainder will be retrieved programmatically if or when this information is used.

An example entry for this directory is:

```yaml
language: Blub
implementations:
- name: jsonschema-ftw
url: https://example.com/someone/jsonschema-ftw
known_to_have:
many_failing_tests: false
recent_draft_support: true
recent_release: true
recent_commit: true
non_production_status: false
metrics:
# last_commit_date: fetched from GitHub
# last_release_date: fetched from GitHub
# open_security_vulnerabilities: fetched from GitHub
# open_issues: fetched from GitHub
# open_prs: fetched from GitHub
known_interfaces:
instance-validation: https://docs.jsonschema-ftw.com/validation/validate_function
version-detection: https://docs.jsonschema-ftw.com/validation/version_of_schema
```
Note that *not* all of the above must precisely match in order to be considered for inclusion!
Specifically, an implementation with *no* recent commits is not disqualified from being considered "high quality".
One with a high number of failing tests however, is indeed likely not to meet the definition.
The above is simply meant to track these data points without necessarily imposing all of them as gospel -- users of these libraries should make their own judgements there.
Please pick a language you're familiar with and feel free to fill in such an entry!
Feedback on this format itself is also welcome in an issue!

0 comments on commit 809489a

Please sign in to comment.