Support for inferring project version from any PEP 517 build backend

[mikenerone]

Currently, other than explicitly passing a project's version to towncrier (config file or command line), the only implicit way it can determine that version is via the common practice of having a __version__ attribute on the top-level package itself. While common historically, this approach is not a standard, and, in fact, creates a second source-of-truth that can potentially conflict with the build backend's true definition of the version. While many projects that still use the legacy setup.py build system delegate to the same source file to prevent such conflicts (either directly or via a distutils plugin) there is no enforcement of this. Further, the advent of PEP 517/PEP 518 has opened the door to many new build systems, selectable via a project's pyproject.toml file. Many of these build systems have data-based configuration files (most, in fact, use sections within pyproject.toml) that are never actually executed, so there is no opportunity to populate the version or other metadata dynamically. As a result of these recent changes to Python's build standards, as well as the availability of importlib.metadata in the standard library (and a backport) for proper determination of package metadata at runtime, the __version__ approach is increasingly falling out of favor in the Python community.

Fortunately, PEP 517 also provides a new method for abstractly delegating the metadata determination to whatever compliant build backend the project has chosen to use, which paves the way for towncrier to continue to provide the very convenient project-version-inference feature. Specifically, the build backend must provide a build_wheel() hook, so a wheel file can always be generated and the metadata read from that, and it may also provide a lighter prepare_metadata_for_build_wheel() hook, which provides the same metadata without the full build. Happily, PyPA (the Python Packaging Authority) provides an official package for abstracting these operations: pep517 (compatible with Python 2.7 and all versions of Python 3).

Using this package, determining the project version from its project directory (AFAIK, towncrier is always executed from there) is as simple as:

from pep517 import meta

...
project_version = meta.load(".").version
...

This will create a temporary environment in which the build system is installed, and attempt to call the backend's lightweight call first, then fall back to the wheel build if necessary. Also per PEP 517, if the project doesn't have a pyproject.toml, or it doesn't specify a build backend, then it will fall back to the setuptools.build_meta:__legacy__ backend, which provides a PEP 517-compliant implementation of the legacy setup.py behavior. Because of these semantics, I believe that in addition to supporting all modern PEP 517 build systems, this approach can actually replace the existing __version__ support in towncrier, as well, without breaking legacy setup.py projects. In fact, it supports them better, since it will honor whatever version-delegation might be implemented in setup.py instead of requiring the specific __version__ attribute at a specific place.

[adiroiban]

I am +1 for using pep517 and encourage the usage of pyproject.toml

---

I think that we tried pep517 meta in a separate Twisted related project (pydoctor) and the feedback from other developers was that running pip wheel in the background to get the project meta is too heavy

twisted/pydoctor#332 (comment)

[bennyrowland]

This is very nearly relevant so I will comment here in the hopes of provoking some interesting discussion. I am currently using setuptools_scm for my version management, which relies on the use of a git tag to define the base version, plus any number of commits since the latest tag (each version tag then corresponds to a release). However, this approach requires the tag to be applied after a release version is committed to the repo, whereas towncrier obviously requires the version string to build the file before committing to the repo. So far I have found myself performing a release commit, where I take the version of the code that I want to release and then run towncrier with the version I want to use for the new release, commit the new file (and the deletions of fragments) and tag that commit with the release version. Currently this is a manual process (albeit a simple one), but I intend to create a script to do it for me - that would remove the possibility of getting the versions out of sync with each other.

[tomster]

Since this issue seems to have gone stale: is there any other way to keep a canonical version information in pyproject.toml? for instance, poetry has this nice feature, where you can simply bump the version (https://python-poetry.org/docs/cli/#version) from the commandline but that obviously does not work with towncrier, if towncrier expects the version information to reside in its own section of pyproject.toml. is this something that simply hasn't been considered (yet) by the authors of towncrier or is there some other configuration / invocation that I'm missing to address this? (I guess one work around would be to parse the version from poetry's section on the command line and pass that string into towncriers build command i.e. poetry run towncrier build --version (poetry version -s) but that seems a bit cumbersome, especially in contrast to how simple and elegent towncrier works in every other regard...
An approach as suggested would remedy this quite nicely. So I guess I'm just saying in a very roundabout way: +1 :)

[adiroiban]

I think that this is not resolved since nobody has pushed a PR :)

regarding pep517 and https://www.python.org/dev/peps/pep-0621/ ... I tried using it in a relate project but the general feedback was negative.

It sound good in theory, but the implementation is super slow.

for towncrier, the fact that pep517 package is slow might not be such a big issue as you don't do a release every 10 minutes :)

[mikenerone]

It sound good in theory, but the implementation is super slow.

This is why I hope it becomes common for PEP 517 implementations to provide the optional prepare_metadata_for_build_wheel() hook. For any given build system, simply returning the metadata is likely to be quite light (e.g. for Poetry it's all in the Poetry section of pyproject.toml, ready to be parsed and returned almost directly). The problem is just that when that hook is not provided, a full build is required in order to get the metadata.

But in any case, as you said, this probably isn't a big problem for towncrier's purpose, so it's a good candidate for early adoption, which in turn will encourage implementation of that optional hook.

[adiroiban]

Now... at the same time, you don't run towncrier that often ... unless it is integrated into your CI system :)

So, even if it's slow, maybe it is worth to wait longer for towncrier to do its job.

as a mentioned I am +1 for using this and I would be happy to review and approve such a PR :)

I don't have much time for writing towncrier code :|

[mikesongming]

As far as setuptools being used as the build backend, dynamic version can be retrieved by running a light job "egg-info".

Or for dynamic versioning tools like versioningit, versioning is called by iterating an entry group "setuptools.finalize_distribution_options", as in "dist.py":

 815     def finalize_options(self):
 816         """
 817         Allow plugins to apply arbitrary operations to the
 818         distribution. Each hook may optionally define a 'order'
 819         to influence the order of execution. Smaller numbers
 820         go first and the default is 0.
 821         """
 822         group = 'setuptools.finalize_distribution_options'
 823
 824         def by_order(hook):
 825             return getattr(hook, 'order', 0)
 826
 827         defined = pkg_resources.iter_entry_points(group)
 828         filtered = itertools.filterfalse(self._removed, defined)
 829         loaded = map(lambda e: e.load(), filtered)
 830         for ep in sorted(loaded, key=by_order):
 831             ep(self)

[setuptools.finalize_distribution_options]
 versioningit = versioningit.hook:setuptools_finalizer