PEP 777: How to Re-invent the Wheel (python#4036)

Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Carol Willing <carolcode@willingconsulting.com>

.github/CODEOWNERS

@@ -640,6 +640,8 @@ peps/pep-0759.rst  @warsaw
 peps/pep-0760.rst  @pablogsal @brettcannon
 peps/pep-0761.rst  @sethmlarson @hugovk
 # ...
+peps/pep-0777.rst  @warsaw
+# ...
 peps/pep-0789.rst  @njsmith
 # ...
 peps/pep-0801.rst  @warsaw

peps/pep-0777.rst

@@ -0,0 +1,303 @@
+PEP: 777
+Title: How to Re-invent the Wheel
+Author: Ethan Smith <ethan@ethanhs.me>
+Sponsor: Barry Warsaw <barry@python.org>
+PEP-Delegate: Paul Moore <p.f.moore@gmail.com>
+Status: Draft
+Type: Standards Track
+Topic: Packaging
+Created: 09-Oct-2024
+Post-History:
+
+Abstract
+========
+
+The current :pep:`wheel 1.0 specification <427>` was written over a decade ago,
+and has been extremely robust to changes in the Python packaging ecosystem.
+Previous efforts to improve the wheel specification 
+:pep:`were deferred <491#pep-deferral>` to focus on other packaging
+specifications. Meanwhile, the use of wheels has changed dramatically in the
+last decade. There have been many requests for new wheel features over the
+years; however, a fundamental obstacle to evolving the wheel specification has
+been that there is no defined process for how to handle adding
+backwards-incompatible features to wheels. Therefore, to enable other PEPs to
+describe new enhancements to the wheel specification, **this PEP prescribes**
+**compatibility requirements on future wheel revisions**. This PEP does *not*
+specify a new wheel revision. The specification of a new wheel format
+(“Wheel 2.0”) is left to a future PEP.
+
+Rationale
+=========
+
+Currently, wheel specification changes that require new installer behavior are backwards incompatible and require a major version increase in
+the wheel metadata format. An increase of the wheel major version has yet to
+happen, partially because such a change has the potential to be
+catastrophically disruptive. Per
+`the wheel specification <https://packaging.python.org/en/latest/specifications/binary-distribution-format/#installing-a-wheel-distribution-1-0-py32-none-any-whl>`_,
+any installer that does not support the new major version must abort at install
+time. This means that if the major version were to be incremented without
+further planning, many users would see installation failures as older installers reject new wheels
+uploaded to public package indices like the Python Package Index (PyPI). It is
+critically important to carefully plan the interactions between build tools,
+package indices, and package installers to avoid incompatibility issues,
+especially considering the long tail of users who are slow to update their
+installers.
+
+The backward compatibility concerns have prevented valuable improvements
+to the wheel file format, such as
+`better compression <https://discuss.python.org/t/improving-wheel-compression-by-nesting-data-as-a-second-zip/1747>`_,
+`wheel data format improvements <https://discuss.python.org/t/should-there-be-a-new-standard-for-installing-arbitrary-data-files/7853/7>`_,
+`better information about what is included in a wheel <https://discuss.python.org/t/record-the-top-level-names-of-a-wheel-in-metadata/29494>`_,
+and `JSON formatted metadata in the ".dist-info" folder <https://discuss.python.org/t/is-was-there-a-goal-with-pep-566s-json-encoding-section/12324/3>`_.
+
+This PEP describes constraints and behavior for new wheel revisions to preserve
+stability for existing tools that do not support a new major version of the wheel format.
+This ensures that backwards incompatible changes to the wheel specification
+will only affect users and tools that are properly set up to use the newer
+wheels. With a clear path for evolving the wheel specification, future PEPs
+will be able to improve the wheel format without needing to re-define a
+completely new compatibility story.
+
+Specification
+=============
+
+Add Wheel-Version Metadata Field to Core Metadata
+-------------------------------------------------
+
+Currently, the :pep:`wheel 1.0 PEP <427>`, PEP 427, specifies that wheel files
+must contain a ``WHEEL`` metadata file that contains the version of the wheel
+specification that the file conforms to. PEP 427 stipulates that installers
+MUST warn on installation of a wheel with a minor version greater than supported,
+and MUST abort on installation of wheels with a major version that is greater than
+what the installer supports. This ensures that users do not get invalid
+installations from wheels that installers cannot properly install.
+
+However, resolvers do not currently exclude wheels with an incompatible wheel
+version. There is also currently no way for a resolver to check a wheel's
+version without downloading the wheel directly. To make wheel version filtering
+easy for resolvers, the wheel version **MUST** be included in the relevant
+metadata file (currently METADATA). This will allow resolvers to efficiently
+check the wheel version using the :pep:`658` metadata API without needing to
+download and inspect the ``.dist-info/WHEEL`` file.
+
+To accomplish this, a new core metadata field is introduced called
+``Wheel-Version``. While this field is optional for metadata included in a
+wheel of major version 1, it is a mandatory field for metadata in wheels of major
+version 2 or higher. This enforces that future revisions of the wheel
+specification can rely on resolvers skipping incompatible wheels by checking
+the ``Wheel-Version`` field.
+
+The ``Wheel-Version`` field in the metadata file shall contain the exact same entry as the
+``Wheel-Version`` entry in the ``WHEEL`` file, or any future replacement file
+defining metadata about the wheel file. Installers **MUST** verify that these
+entries match when installing a wheel. If ``Wheel-Version`` is absent from the
+metadata file, then the implied major version of the wheel is 1.
+
+Resolver Behavior Regarding ``Wheel-Version``
+---------------------------------------------
+
+Resolvers, in the process of selecting a wheel to install, **MUST** check a
+candidate wheel's ``Wheel-Version``, and ignore incompatible wheel files.
+Without ignoring these files, older installers might select a wheel that uses
+an unsupported wheel version for that installer, and force the installer to
+abort per :pep:`427`. By skipping incompatible wheel files, users will not see
+installation errors when a project adopts a new wheel major version. As already
+specified in PEP 427, installers **MUST** abort if a user tries to directly
+install a wheel that is incompatible. If, in the process of resolving packages
+found in multiple indices, a resolver comes across two wheels of the same
+distribution and version, resolvers should prioritize the wheel of the highest
+compatible version.
+
+While the above protects users from unexpected breakages, users may miss a new
+release of a distribution if their installer does not support the wheel version
+used in the release. Imagine in the future that a package publishes 3.0 wheel
+files. Downstream users won't see that there is a new release available if
+their installers only support 2.x wheels. Therefore, installers **SHOULD** emit
+a warning if, in the process of resolving packages, they come across an incompatible wheel
+and skip it.
+
+First Major Version Bump Must Change File Extension
+---------------------------------------------------
+
+Unfortunately, existing resolvers do not check the compatibility of wheels
+before selecting them as installation candidates. Until a majority of users
+update to installers that properly check for wheel compatibility, it is unsafe
+to allow publishing wheels of a new major version that existing resolvers might
+select. It could take upwards of four years before the majority of users are on
+updated resolvers, based on current data about PyPI installer usage (See the
+:ref:`777-pypi-download-analysis`, for
+details). To allow for experimentation and faster adoption of 2.0 wheels,
+this PEP proposes a one time change to the file extension of the
+wheel file format, from ``.whl`` to ``.whlx``. This resolves the initial
+transition issue of 2.0 wheels breaking users on existing installers that do
+not implement ``Wheel-Version`` checks. By using a different file extension,
+2.0 wheels can immediately be uploaded to PyPI, and users will be able to
+experiment with the new features right away. Users on older installers will
+simply ignore these new files.
+
+One rejected alternative would be to keep the ``.whl`` extension, but delay the
+publishing of wheel 2.0 to PyPI. For more on that, please see Rejected Ideas.
+
+Recommended Build Backend Behavior with New Wheel Formats
+---------------------------------------------------------
+
+Build backends are recommended to generate the most compatible wheel based on
+features a project uses. For example, if a wheel does not use symbolic links,
+and such a feature was introduced in wheel 5.0, the build backend could
+generate a wheel of version 4.0. On the other hand, some features will want to
+be adopted by default. For example, if wheel 3.0 introduces better compression,
+the build backend may wish to enable this feature by default to improve the
+wheel size and download performance.
+
+Limitations on Future Wheel Revisions
+-------------------------------------
+
+While it is difficult to know what future features may be planned for the wheel
+format, it is important that certain compatibility promises are maintained.
+
+Wheel files, when installed, **MUST** stay compatible with the Python standard
+library's ``importlib.metadata`` for all supported CPython versions.  For
+example, replacing ``.dist-info/METADATA`` with a JSON formatted metadata file
+MUST be a multi-major version migration with one version introducing the new
+JSON file alongside the existing email header format, and another future
+version removing the email header format metadata file. The version to remove
+``.dist-info/METADATA`` also **MUST** be adopted only after the last CPython
+release that lacked support for the new file reaches end of life. This ensures
+that code using ``importlib.metadata`` will not break with wheel major version
+revisions.
+
+Wheel files **MUST** remain ZIP format files as the outer container format.
+Additionally, the ``.dist-info`` metadata directory **MUST** be placed at the
+root of the archive without any compression, so that unpacking the wheel file
+produces a normal ``.dist-info`` directory holding any metadata for the wheel.
+Future wheel revisions **MAY** modify the layout, compression, and other
+attributes about non-metadata components of a wheel such as data and code. This
+assures that future wheel revisions remain compatible with tools operating on
+package metadata, while allowing for improvements to code storage in the wheel,
+such as adopting compression.
+
+Package tooling **MUST NOT** assume that the contents and format of the wheel
+file will remain the same for future wheel major versions beyond the
+limitations above about metadata folder contents and outer container format.
+For example, newer wheel major versions may add or remove filename components,
+such as the build tag or the platform tag. Therefore it is incumbent upon
+tooling to check the metadata for the ``Wheel-Version`` before attempting to
+install a wheel.
+
+Finally, future wheel revisions **MUST NOT** use any compression formats not in
+the CPython standard library of at least the latest release. Wheels generated
+using any new compression format should be tagged as requiring at least the
+first released version of CPython to support the new compression format,
+regardless of the Python API compatibility of the code within the wheel.
+
+Backwards Compatibility
+=======================
+
+Backwards compatibility is an incredibly important issue for evolving the wheel
+format. If adopting a new wheel revision is painful for downstream users,
+package creators will hesitate to adopt the new standards, and users will be
+stuck with failed CI pipelines and other installation woes.
+
+Several choices in the above specification are made so that the adoption of a
+new feature is less painful. For example, today wheels of an incompatible major
+version are still selected by pip as installation candidates, which causes
+installer failures if a project starts publishing 2.0 wheels. To avoid this
+issue, this PEP requires resolvers to filter out wheels with major versions or
+features incompatible with the installer.
+
+This PEP also defines constraints on future wheel revisions, with the goal of
+maintaining compatibility with CPython, but allowing evolution of wheel
+contents. Wheel revisions shouldn't cause package installations to break on
+older CPython revisions, as not only would it be frustrating, it would be
+incredibly hard to debug for users.
+
+The main compatibility limitation of this PEP is for projects that start
+publishing solely new wheels alongside a source distribution. If a user on an
+older installer tries to install the package, it will fall back to the source
+distribution, because the resolver will skip all newer wheels. Users are often
+poorly set up to build projects from source, so this could lead to some failed
+builds users would not see otherwise. There are several approaches to resolving
+this issue, such as allowing dual-publishing for the initial migration, or
+marking source distributions as not intended to be built.
+
+Rejected Ideas
+==============
+
+The Wheel Format is Perfect and Does not Need to be Changed
+-----------------------------------------------------------
+The wheel format has been around for over 10 years, and in that time, Python
+packages have changed a lot. It is much more common for packages to include
+Rust or C extension modules, increasing the size of packages. Better
+compression, such as lzma or zstd, could save a lot of time and bandwidth for
+PyPI and its users. Compatibility tags cannot express the wide variety of
+hardware used to accelerate Python code today, nor encode shared library
+compatibility information. In order to address these issues, evolution of the
+wheel package format is necessary.
+
+Wheel Format Changes Should be Tied to CPython Releases
+-------------------------------------------------------
+I do not believe that tying wheel revisions to CPython
+releases is beneficial. The main benefit of doing so is to make adoption of new
+wheels predictable - users with the latest CPython get the latest package
+format! This choice has several issues however. First, tying the new format
+to the latest CPython makes adoption much slower. Users on LTS versions of
+Linux with older Python installations are free to update their pip in a virtual
+environment, but cannot update the version of Python as easily.  While some
+changes to the wheel format must be tied to CPython changes necessarily, such
+as adding new compression formats or changing the metadata format, many changes
+do not need to be tied to the Python version, such as symlinks, enhanced
+compatibility tags, and new formats that use existing compression formats in
+the standard library. Additionally, wheels are used across multiple different
+language implementations, which lag behind the CPython version. It seems unfair
+to prevent their users from using a feature due to the Python version. Lastly,
+while this PEP does not suggest tying the wheel version to CPython releases, a
+future PEP may still do so at any time, so this choice does not need to be made
+in this PEP.
+
+Keep Using ``.whl`` as the File Extension
+-----------------------------------------
+While keeping the extension ``.whl`` is appealing for many reasons, it presents
+several problems that are difficult to surmount.  First, current installers
+would still pick a new wheel and fail to install the package. Furthermore,
+the file name of a wheel would not be able to change without breaking existing
+installers that expect a set wheel file name format. While the current filename
+specification for wheels is sufficient for current usage, the optional
+build tag in the middle of the file name makes any extensions ambiguous (i.e.
+``foo-0.3-py3-none-any-fancy_new_tag.whl`` would parse as the build tag being
+``py3``). This limits changes to information stored in the wheel file name.
+
+Discussion Topics
+=================
+
+Should Indices Support Dual-publishing for the First Migration?
+---------------------------------------------------------------
+Since ``.whl`` and ``.whlx`` will look different in file name, they could be
+uploaded side-by-side to package indices like PyPI. This has some nice
+benefits, like dual-support for older and newer installers, so users who can
+get the latest features, while users who don't upgrade still can install the
+latest version of a package.
+
+There are many complications however. Should we allow wheel 2 uploads to
+existing wheel 1-only releases? Should we put any requirements on the
+side-by-side wheels, such as:
+
+.. admonition:: Constraints on dual-published wheels
+
+    A given index may contain identical-content wheels with different wheel
+    versions, and installers should prefer the newest-available wheel format,
+    with all other factors held constant.
+
+Should we only allow uploading both with :pep:`694` allowing "atomic"
+dual-publishing?
+
+Acknowledgements
+================
+
+The author of this PEP is greatly indebted to the incredibly valuable review,
+advice, and feedback of Barry Warsaw and Michael Sarahan.
+
+Copyright
+=========
+
+This document is placed in the public domain or under the
+CC0-1.0-Universal license, whichever is more permissive.