Python 3.11 in FreeBSD Ports

A month ago I’ve landed Python 3.11 (alpha2) in FreeBSD Ports tree. This is a continuation of my past work on Python 3.10 support, and likewise, it allows curious ones to play with the upcoming release, maintainers to prepare their ports and developers to make sure their code is compatible with the language changes. I’d like to share some details on latest Python support in FreeBSD and the story of porting Python 3.10.

Python 3.10 status

Python 3.10 support is not production ready yet, as some critical ports do not support it. The most important is math/py-numpy (update PR since Nov 04, no reaction from python@ for a month). A lot of ports which depend in it are not available in Python 3.10 flavor.

Some other important dependencies lacking 3.10 support:

Expectedly, there’s a few dozen of leaf port failures as well.

Still, it may work for you depending on which modules you need. For instance, everything I need for my Python projects including Repology work fine and I’ve already switched my production to Python 3.10.

DEFAULT_VERSIONS+=python=3.10 python3=3.10

Python 3.11 status

All the issues with 3.10 apply here too (but the good part is that they will likely be fixed for both versions), and a bunch of new problems is expected. Since this is still an alpha, thus a moving target, there’s not much sense in polishing 3.11 support explicitly, but all new failures may reveal generally problematic ports which need attention.

Not much is expected to work here, but still I’ve already managed to fix most of ports I use (the only remaining is www/uwsgi, which will be fixed by an update this week). A poudriere build which will reveal 3.11 specific failures is currently running.

Further work

Fix more individual failures

The workflow: run a poudriere build with DEFAULT_VERSIONS=python3.10 python3=3.10, identify failures, fix. Most fixes are trivial (so you can help too), for instance:

  • Need to force cython files to be regenerated (examples: asyncpg, aiohttp). Notably, these changes prevent breakages with future python versions (as soon as cython keeps up with Python C API changes).
  • Fix some imports after some modules were moved around (example: spidermonkey).
  • Fix build scripts to use fixed version of Python instead of iterating through (not) all versions. From multimedia/gstreamer1-editing-services failure log: 
    checking for python... no
checking for python2... no
checking for python3... no
checking for python3.9... no
checking for python3.8... no
checking for python3.7... no
checking for python3.6... no
checking for python3.5... no
  • Force build to use versioned Python scripts. From fix for math/py-gmpy2-devel: 
    -   (cd ${WRKSRC}/docs && ${GMAKE} html)
+   (cd ${WRKSRC}/docs && ${GMAKE} SPHINXBUILD=sphinx-build-${PYTHON_VER} html)
    Similar fix for devel/cmake.
  • Fix incorrect version handling in Python code, such as platform.python_version()[:3] construct which returns 3.1 for Python 3.10 (example: libSEDML, abseil).
  • As a last resort, mark a port not compatible with newer python versions: 
    -USES=python:3.6+
+USES=python:3.6-3.9

Force pytest and numpy updates

Updates for important ports handing in the bugzilla for months is not acceptable. In most cases they are blocked by testing or fixing the affected ports, which may last forever. So instead it could be solved by preserving an older version of the port (e.g. copy py-pytest → py-pytest4) and switching all the consumers to it (this is merely a cosmetic change not expected to cause any breakages), then update actual version of the port and gradually switch consumers onto it. Given enough time, I’d have to take responsibility and perform these updates since python@ team is not generally active.

The quest of adding Python 3.10

Finally I’d like to share a story of porting Python 3.10, which required a bunch of not directly related changes.

Apart from Python 3.10 included a usual set of incompatible changes, there was one special thing about this release: it’s minor version contained two digits.

FreeBSD ports framework has internal machinery for comparing Python versions, and since it’s make based (and make lacks useful tools for that purpose like tuples), versions have to be converted to plain numbers, where e.g. 3.9.1 would be represented as 3901. You can notice that there’s no place for 10 minor version here, that is exactly why it’s had to be extended.

Funnily enough, it had already been extended in the past for patch version. From CHANGES file:

20150526:
AUTHOR: antoine@FreeBSD.org

  PYTHON_REL has been switched from a 3 digits number to a 4 digits number to
  handle python 2.7.10.  Ports checking for python 2.7.9 should compare
  PYTHON_REL against 2709 and ports checking for python 2.7.10 should compare
  PYTHON_REL against 2710.

just to repeat the same mistake which needed another fix.

There was another part of the framework which did not expect two-digit minor version and had to be fixed as well.

Lesson learned: do not spare digits for numeric representation of a version or another thing you try to pack into it.

Then I’ve had to fix the way a version is handled for Python ports to allow pre-release versions. FreeBSD port may define either PORTVERSION (version as used by FreeBSD) or DISTVERSION (version as defined by upstream), and Python ports used the former. This led to incorrect comparison for pre-release versions:

% pkg version -t 3.11.0a2 3.11.0
>

So I’ve had to switch everything to DISTVERSION, which implies normalization of upstream version into something compatible with FreeBSD. That is, for DISTVERSION=3.11.0a2 the framework generates PORTVERSION=3.11.0.a2 which compares correctly:

% pkg version -t 3.11.0.a2 3.11.0
<

Then I’ve had to update setuptools module. The port was at 44.1.1 which supported Python 2 (which we still need for some consumers), but not Python 3.10. The updated version 57.0.0 supported Python 3.10, but not Python 2. So I’ve had to preserve the older version as devel/py-setuptools44 and add some machinery which switches to it for python2 ports.

A lot of Python ports were also fixed and updated in the meantime.

Finally, in almost two months after the first commit, Python 3.10.0beta4 has landed.

Python 3.11 update went much smoother, as most prerequisites were already in place. Apart from landing Python itself, I’ve just had to add lang/cython-devel port (as stable cython had no support for python 3.11 yet) and apply a minor fix to databases/py-sqlite3. Hopefully, my work will make future updates easier too.