Cleanup setup.py metadata for all core packages

[thet]

• Fix trove classifiers (Plone version, Python version)
• remove ez_setup reference
• Add Framework :: Zope :: 4 trove classifier
• Change author information to Plone Foundation
• Change author email from "plone-developers@lists.sourceforge.net" to something we use. decision needs to be made
• ...

@plone/framework-team

[mauritsvanrees]

• Fix pypi.python.org package links. https://pypi.python.org/pypi/package-name -> https://pypi.org/project/package-name
• Fix pypi.python.org classifiers links: https://pypi.org/classifiers/

[jensens]

I think this needs a script, at least semi-automated...

[mauritsvanrees]

If anyone is doing cleanups like this which don't really touch code and cannot break anything, please include [ci skip] in your commit message to avoid running Jenkins.

[mauritsvanrees]

I have found this script on my laptop to remove some classifiers from setup.py. That may be useful as blueprint for similar scripts:

#!/bin/sh
# Remove Python 2.6 and 3.2 classifiers (and lower) from setup.py.
cat setup.py | grep -v 'Programming Language :: Python :: 2.3' | grep -v 'Programming Language :: Python :: 2.4' | grep -v 'Programming Language :: Python :: 2.5' | grep -v 'Programming Language :: Python :: 2.6' | grep -v 'Programming Language :: Python :: 3.0' | grep -v 'Programming Language :: Python :: 3.1' | grep -v 'Programming Language :: Python :: 3.2' > setup.py.tmp
mv setup.py.tmp setup.py
git diff setup.py
echo "Commit this? [ENTER means yes, anything else means revert] "
read answer
if test "x$answer" == 'x'; then
   echo "committing"
   git commit setup.py -m "Removed Python 2.6 and 3.2 classifiers (and lower)."
else
   echo "Reverting setup.py."
   git checkout -- setup.py
fi

[mauritsvanrees]

I am using this script to update the pypi links of a single package:

#!/bin/sh
# Remove pypi.python.org links from setup.py.
HTTP='http://pypi.python.org'
HTTPS='https://pypi.python.org'
OLD_CLASSIFIERS='http://pypi.python.org/pypi?:action=list_classifiers'
OLD_CLASSIFIERS2='https://pypi.python.org/pypi?%3Aaction=list_classifiers'
NEW_CLASSIFIERS='https://pypi.org/classifiers/'
OLD_PACKAGE='https://pypi.python.org/pypi/'
NEW_PACKAGE='https://pypi.org/project/'
sed "s|${HTTP}|${HTTPS}|;s|${OLD_CLASSIFIERS}|${NEW_CLASSIFIERS}|;s|${OLD_CLASSIFIERS2}|${NEW_CLASSIFIERS}|;s|${OLD_PACKAGE}|${NEW_PACKAGE}|" setup.py > setup.py.tmp
mv setup.py.tmp setup.py
git diff setup.py
if test $(git status --porcelain | wc -l) -eq 0; then
    echo "No changes"
    exit 0
fi
echo "Commit and PUSH this? [ENTER means yes, anything else means revert] "
read answer
if test "x$answer" == 'x'; then
    echo "committing"
    git commit setup.py -m """Fixed PyPI links in setup.py.

[ci skip]"""
    git push
else
    echo "Reverting setup.py."
    git checkout -- setup.py
    # exit 1
fi

and then this to run it on all checkouts in coredev 5.2 in the src directory:

export ORIGDIR='/Users/maurits/community/plone-coredev/5.2/src'
for i in */; do cd $i && echo; echo; echo "#############" && echo "### $i" && git pull && cleanup-pypi-links.sh || echo "No changes or could not push"; cd $ORIGDIR; done

[mauritsvanrees]

Okay, PyPI links are fixed on all packages on coredev 5.2 that end up in the src directory with auto-checkout = *.

[gforcada]

Should we remove the exclude=['ez_setup'] on packages=find_packages('src', exclude=['ez_setup']), as well? 🤔

[mauritsvanrees]

Yes, that can be removed. I see that lots of packages have that, and I see only ZODB3 which actually has a ez_setup.py.
I can do that.
First I am adding 5.2 classifiers to all packages in 5.2/src that have 5.1 in their classifiers.

[mauritsvanrees]

Currently using this script for the 5.2 classifiers:

#!/bin/sh
# Add Plone 5.2 classifier if there is a 5.1 classifier.
OLD_CLASSIFIER='Framework :: Plone :: 5.1'
NEW_CLASSIFIER='Framework :: Plone :: 5.2'
if test $(grep -c "${NEW_CLASSIFIER}" setup.py) -ge 1; then
    echo "5.2 already in classifiers"
    exit 0
fi
if test $(grep -c "${OLD_CLASSIFIER}" setup.py) -eq 0; then
    echo "5.1 not in classifiers"
    exit 0
fi
sed "/${OLD_CLASSIFIER}/p; s/${OLD_CLASSIFIER}/${NEW_CLASSIFIER}/" setup.py > setup.py.tmp
mv setup.py.tmp setup.py
git diff setup.py
if test $(git status --porcelain | wc -l) -eq 0; then
    echo "No changes"
    exit 0
fi
echo "Commit and PUSH this? [ENTER means yes, anything else means revert] "
read answer
if test "x$answer" == 'x'; then
    echo "committing"
    git commit setup.py -m """Added Plone 5.2 to classifiers in setup.py.

[ci skip]"""
    git push
else
    echo "Reverting setup.py."
    git checkout -- setup.py
    # exit 1
fi

I forgot to change the commit message, so the first couple dozen packages have the wrong commit message, but okay.

[mauritsvanrees]

5.2 has been added to the classifiers, when 5.1 was already in there.

[mauritsvanrees]

Using this for ez_setup.py:

#!/bin/sh
# Remove the exclude=['ez_setup'] on
# packages=find_packages('src', exclude=['ez_setup'])
#  Only one is using double quotes.
# OLD='exclude=\["ez_setup"\]'
OLD="exclude=\['ez_setup'\]"
NEW=''
if test $(grep -c "${OLD}" setup.py) -eq 0; then
    echo "ez_setup.py not excluded in setup.py"
    exit 0
fi
sed "s/${OLD}/${NEW}/" setup.py > setup.py.tmp
mv setup.py.tmp setup.py
git diff setup.py
if test $(git status --porcelain | wc -l) -eq 0; then
    echo "No changes"
    exit 0
fi
echo "Commit and PUSH this? [ENTER means yes, anything else means revert] "
read answer
if test "x$answer" == 'x'; then
    echo "committing"
    git commit setup.py -m """Removed unneeded exclude ez_setup.py from setup.py.

[ci skip]"""
    git push
else
    echo "Reverting setup.py."
    git checkout -- setup.py
    # exit 1
fi

Done all.

[mauritsvanrees]

Towncrier checklist of packages moved to issue #2548.

[jensens]

We should also add "Framework :: Zope :: 4" I suppose?

[mauritsvanrees]

We should also add "Framework :: Zope :: 4" I suppose?

Yes, that would be good.

[gforcada]

@thet I took over your first message to add all the requests to change setup.py, I hope it is fine.

Regarding email address, any suggestion? Should we point them to the release/security/framework teams? Any preference

Regarding the author, if we assume that plone packages are owned by the foundation, then yes there should be no problem, let's hope no one gets offended due to the change 😅

[mauritsvanrees]

I updated the setup.cfg of (almost) all packages to contain [bdist_wheel] universal =1. I removed [zest.releaser] create-wheel = yes because this is superfluous when bdist_wheel/universal is there.

We used to have a comment in setup.cfg of a lot of packages to remind us to enable universal wheels when the package is Python 3 compatible. But actually this is fine for all packages, except the ones that have C code (like AccessControl and Products.ZCTextIndex).

It is also fine for say Products.Archetypes: it is not Python 3 compatible, but when you run the fullrelease script with Python 3, it cannot know this and it will create a Python-3-only wheel, and no Python 2 wheel. With universal = 1, you will always get a py2-py3 wheel, so it can actually be used on Python 2.

For the record, this is the script I used to update a single repository branch. There are a few corner cases that the script does not catch, so you need to pay attention. But worked okay here.

#!/bin/sh
# Set [bdist_wheel] universal = 1
cat setup.cfg | grep -v '# When Python 2-3 compatible:' | grep -v '# \[bdist_wheel\]' | grep -v '# universal = 1' | grep -v '\[zest.releaser\]' | grep -v 'create-wheel = yes' > setup.cfg.tmp
if test $(grep -c "universal" setup.cfg.tmp) -eq 1; then
    echo "universal already in setup.cfg"
    rm setup.cfg.tmp
    exit 0
fi
echo "" >> setup.cfg.tmp
echo "[bdist_wheel]" >> setup.cfg.tmp
echo "universal = 1" >> setup.cfg.tmp
while test "$(head -1 setup.cfg.tmp)" == ""; do
    tail -n +2 setup.cfg.tmp > setup.cfg
    cp setup.cfg setup.cfg.tmp
done
mv setup.cfg.tmp setup.cfg
git diff setup.cfg
echo "Commit this? [ENTER means yes, anything else means revert] "
read answer
if test "x$answer" == 'x'; then
   echo "committing"
   git commit setup.cfg -m "Release as universal (Py2/3 compatible) wheel.
   
[ci skip]"
   git push
else
   echo "Reverting setup.cfg."
   git checkout -- setup.cfg
fi

[mauritsvanrees]

I added the Zope 4 classifier to the setup.py of most packages. Script:

#!/bin/sh
# Add Zope 4 classifier if there is a Zope classifier.
OLD_CLASSIFIER='Framework :: Zope2'
NEW_CLASSIFIER='Framework :: Zope :: 4'
if test $(grep -c "${NEW_CLASSIFIER}" setup.py) -ge 1; then
    echo "Zope 4 already in classifiers"
    exit 0
fi
if test $(grep -c "${OLD_CLASSIFIER}" setup.py) -eq 0; then
    echo "Zope2 not in classifiers"
    exit 0
fi
sed "/${OLD_CLASSIFIER}/p; s/${OLD_CLASSIFIER}/${NEW_CLASSIFIER}/" setup.py > setup.py.tmp
mv setup.py.tmp setup.py
git diff setup.py
if test $(git status --porcelain | wc -l) -eq 0; then
    echo "No changes"
    exit 0
fi
echo "Commit and PUSH this? [ENTER means yes, anything else means revert] "
read answer
if test "x$answer" == 'x'; then
    echo "committing"
    git commit setup.py -m """Added Zope 4 to classifiers in setup.py.

[ci skip]"""
    git push
else
    echo "Reverting setup.py."
    git checkout -- setup.py
    # exit 1
fi

[jensens]

I close this, the plone/meta project covers this, right?

[gforcada]

Actually so far not, but eventually we might 🤞🏾 meanwhile almost all changes requested here were already done 😄