RFC: Make maintenance of Zope eggs versions by pulling them from upstream.

I don't see any reason why we don't use upstream zope *versions.cfg files. This
just duplicates work and makes maintenance tedious especially now that we have
zope2, zope4py2 and zope4py3.

Also:
  * Remove pinned versions on eggs with newer version or already defined in
    Zope versions.cfg files.
  * Make sure that pinned versions in stack/slapos.cfg do not override Zope versions
    (for example ZConfig versions set in stack/slapos.cfg was creating a version
    conflict while installing another egg).

stack/erp5/buildout.cfg

 [buildout]
-
 extends =
 # Exact version of Zope
-  ztk-versions.cfg
-  zope-versions.cfg
   buildout.hash.cfg
+# This has to be before Zope versions.cfg otherwise eggs such as ZConfig have
+# their versions overridden when slapos.cfg is extends by some component...
+  ../../stack/slapos.cfg
+  https://raw.githubusercontent.com/zopefoundation/Zope/4.5.3/versions.cfg
   ../../component/fonts/buildout.cfg
   ../../component/git/buildout.cfg
   ../../component/ghostscript/buildout.cfg
@@ -676,9 +677,7 @@ Zope = 4.5.3+SlapOSPatched002
 pylint = 1.4.4
 
 # use newer version than specified in ZTK
-PasteDeploy = 1.5.2
 argparse = 1.4.0
-zope.dottedname = 4.1.0
 
 # modified version that works fine for buildout installation
 SOAPpy = 0.12.0nxd001
@@ -719,7 +718,6 @@ WSGIUtils = 0.7
 astroid = 1.3.8
 erp5diff = 0.8.1.7
 five.formlib = 1.0.4
-five.localsitemanager = 2.0.5
 google-api-python-client = 1.6.1
 httplib2 = 0.10.3
 huBarcode = 1.0.0
@@ -761,8 +759,6 @@ zbarlight = 2.3
 cloudpickle = 0.5.3
 dask = 0.18.1
 toolz = 0.9.0
-zope.globalrequest = 1.5
-waitress = 1.4.4
 xlrd = 1.1.0
 
 # Re-add for as it is required to be there for uninstallation
@@ -791,20 +787,12 @@ jedi = 0.15.1
 parso = 0.5.1
 yapf = 0.28.0
 z3c.etestbrowser = 3.0.1
-zope.testbrowser = 5.5.1
-WSGIProxy2 = 0.4.6
-WebTest = 2.0.33
-beautifulsoup4 = 4.8.2
-WebOb = 1.8.5
-soupsieve = 1.9.5
 eggtestinfo = 0.3
 
 oic = 0.15.1
 Beaker = 1.11.0
 Mako = 1.1.4
 pyjwkest = 1.4.2
-alabaster = 0.7.12
-future = 0.18.2
 pycryptodomex = 3.10.1
 
 strict-rfc3339 = 0.7
@@ -814,8 +802,6 @@ jsonpointer = 2.2
 
 
 # WIP Zope 4 ⚠
-zope.interface = 5.2.0 
-ZConfig = 3.5.0
 Products.CMFCore = 2.4.0
 Products.StandardCacheManagers = 4.1.0
 Products.ZSQLMethods = 3.14

stack/erp5/zope-versions.cfg

-[versions]
-#Zope = 4.5.3
-Zope2 = 4.0
-# AccessControl 5+ no longer supports Zope 4.
-AccessControl = 4.2
-Acquisition = 4.7
-AuthEncoding = 4.2
-BTrees = 4.7.2
-Chameleon = 3.8.1
-DateTime = 4.3
-# DocumentTemplate 4+ no longer supports Zope 4.
-DocumentTemplate = 3.4
-ExtensionClass = 4.5.0
-Missing = 4.1
-MultiMapping = 4.1
-Paste = 3.5.0
-PasteDeploy = 2.1.1
-Persistence = 3.0
-Products.BTreeFolder2 = 4.2
-# ZCatalog 6+ no longer supports Zope 4.
-Products.ZCatalog = 5.2
-Record = 3.5
-RestrictedPython = 5.1
-WSGIProxy2 = 0.4.6
-WebOb = 1.8.6
-WebTest = 2.0.35
-ZConfig = 3.5.0
-ZEO = 5.2.2
-ZODB = 5.6.0
-ZServer = 4.0.2
-five.globalrequest = 99.1
-five.localsitemanager = 3.2.2
-funcsigs = 1.0.2
-future = 0.18.2
-ipaddress = 1.0.23
-# mock 4.0 and up requires Python 3
-mock = 3.0.5
-pbr = 5.5.1
-persistent = 4.6.4
-pytz = 2020.4
-roman = 3.3
-shutilwhich = 1.1.0
-six = 1.15.0
-transaction = 3.0.0
-waitress = 1.4.4
-z3c.pt = 3.3.0
-zExceptions = 4.1
-zc.lockfile = 2.0
-zdaemon = 4.3
-zodbpickle = 2.0.0
-zope.annotation = 4.7.0
-zope.browser = 2.3
-zope.browsermenu = 4.4
-zope.browserpage = 4.4.0
-zope.browserresource = 4.4
-zope.cachedescriptors = 4.3.1
-zope.component = 4.6.2
-zope.componentvocabulary = 2.2.0
-zope.configuration = 4.4.0
-zope.container = 4.4.0
-zope.contentprovider = 4.2.1
-zope.contenttype = 4.5.0
-zope.datetime = 4.2.0
-zope.deferredimport = 4.3.1
-zope.deprecation = 4.4.0
-zope.dottedname = 4.3
-zope.event = 4.5.0
-zope.exceptions = 4.4
-zope.filerepresentation = 5.0.0
-zope.formlib = 4.7.1
-zope.globalrequest = 1.5
-zope.hookable = 5.0.1
-zope.i18n = 4.7.0
-zope.i18nmessageid = 5.0.1
-zope.interface = 5.2.0
-zope.lifecycleevent = 4.3.0
-zope.location = 4.2
-zope.pagetemplate = 4.5.0
-zope.processlifetime = 2.3.0
-zope.proxy = 4.3.5
-zope.ptresource = 4.2.0
-zope.publisher = 5.2.1
-zope.ramcache = 2.3
-zope.schema = 6.0.0
-zope.security = 5.1.1
-zope.sendmail = 5.1
-zope.sequencesort = 4.1.2
-zope.site = 4.4.0
-zope.size = 4.3
-zope.structuredtext = 4.3
-zope.tal = 4.4
-zope.tales = 5.1
-zope.testbrowser = 5.5.1
-# Version 4.8+ dropped support for Python 3.5
-zope.testing = 4.7
-zope.testrunner = 5.2
-zope.traversing = 4.4.1
-zope.viewlet = 4.2.1

stack/erp5/ztk-versions.cfg

-[buildout]
-extends =
-    zope-versions.cfg
-versions = versions
-
-[versions]
-# Version pins for development and optional dependencies.
-Babel = 2.8.1
-Jinja2 = 2.11.2
-MarkupSafe = 1.1.1
-# Pygments 2.6.0 and up require Python 3
-Pygments = 2.5.2
-# Version 2.0+ needs Python 3.x
-Sphinx = 1.8.5
-alabaster = 0.7.12
-appdirs = 1.4.4
-attrs = 20.3.0
-backports.functools-lru-cache = 1.6.1
-beautifulsoup4 = 4.9.3
-bleach = 3.2.1
-buildout.wheel = 0.2.0
-# Version 2020.4.5.2 and up claim no Python 2 support
-certifi = 2020.4.5.1
-cffi = 1.14.3
-chardet = 3.0.4
-cmarkgfm = 0.4.2
-collective.recipe.cmd = 0.11
-collective.recipe.sphinxbuilder = 1.1
-collective.recipe.template = 2.1
-colorama = 0.4.4
-# configparser 5 and up require Python 3
-configparser = 4.0.2
-contextlib2 = 0.6.0.post1
-coverage = 5.3
-distlib = 0.3.1
-docutils = 0.16
-filelock = 3.0.12
-idna = 2.10
-imagesize = 1.2.0
-# tox and pluggy require importlib-metadata <1
-importlib-metadata = 0.23
-lxml = 4.6.1
-manuel = 1.10.1
-# Version 6+ needs Python 3.x
-more-itertools = 5.0.0
-mr.developer = 2.0.1
-nose = 1.3.7
-packaging = 20.4
-pathlib2 = 2.3.5
-pip = 20.2.4
-pkginfo = 1.6.1
-plone.recipe.command = 1.1
-pluggy = 0.13.1
-py = 1.9.0
-pycparser = 2.20
-pyparsing = 2.4.7
-python-gettext = 4.0
-readme-renderer = 28.0
-repoze.sphinx.autointerface = 0.8
-requests = 2.25.0
-requests-toolbelt = 0.9.1
-scandir = 1.10.0
-snowballstemmer = 2.0.0
-# soupsieve 2 needs Python 3
-soupsieve = 1.9.6
-sphinx-rtd-theme = 0.5.0
-sphinxcontrib-serializinghtml = 1.1.4
-sphinxcontrib-websupport = 1.2.4
-# tempstorage is only needed because the Sphinx documentation parses
-# ZConfig xml configurations, which still contain references to it
-tempstorage = 5.1
-toml = 0.10.2
-tox = 3.20.1
-tqdm = 4.51.0
-# Version 2+ needs Python 3.x
-twine = 1.15.0
-typing = 3.7.4.3
-urllib3 = 1.26.1
-virtualenv = 20.1.0
-webencodings = 0.5.1
-wheel = 0.35.1
-z3c.checkversions = 1.2
-zc.recipe.egg = 2.0.7
-zc.recipe.testrunner = 2.1
-zest.releaser = 6.22.1
-# Version 2 requires Python 3
-zipp = 1.1.1
-zodbupdate = 1.5

[Rafael Monnerat]

This isn't a good idea, since this url on extends isn't cacheable on shacache (or anywhere else). So, if we do have problems to access it, the software release won't work anymore.

/cc @tomo @Daetalus @romain

[Jérome Perrin]

ah, this was also my first impression ( there's also discussion below BTW ), but then I thought that because it uses zc.buildout.download.Download it would automatically use shacache.

extends are downloaded here: https://lab.nexedi.com/nexedi/slapos.buildout/blob/78ced4136882b38355642a2b017c9c7de993d4b9/src/zc/buildout/buildout.py#L1929-1932

Even if it uses shacache, what I am not so comfortable with is that we cannot use md5sum mechanism here, I tried this:

[buildout]
extends = https://raw.githubusercontent.com/zopefoundation/Zope/4.5.3/versions.cfg#91f3d70f6ffd114f23f1e71f170a81dx

91f3d70f6ffd114f23f1e71f170a81dx is not the md5sum of this profile, but there's no error, which means this is ignored.

[Jérome Perrin]

We use to extends from github directly and we stopped in 8f222b41 (without explanation on commit message, but maybe @kazuhiko remembers ?).

By having the profiles fetched from an external source, there's a risk that one day they are not available or that they became different. I don't think networkcache applies for buildout extends and buildout extends are not verified (there's no md5sum).

By having the profiles copied in the repository, it's a little bit easier to look at the history (that point is not really important)

My preference would be to continue copying these profiles in slapos repository. If it helps, maybe we can use different filenames ( something like Zope-4.5.3-versions.cfg )

BTW, this seem to break the build because we use a different version of zc.buildout

[Arnaud Fontaine]

Thanks for the review ! I don't have a strong preference between an URL or a copy, except that it was not obvious to me that this was a copy from Zope buildout and I started to do modifications... So let's wait for @kazuhiko to reply then (@kazuhiko: it would have been nice to write an useful commit message BTW).

[Kazuhiko Shiozaki]

The reason why I stopped direct extend in 8f222b41 is simply because versions.cfg was no longer included in repository in 2.13.23 or later in 2.13 branch, by https://github.com/zopefoundation/Zope/commit/56b6790d1fd3a6e0105925adcc7cca29e9ab21cf

[Jérome Perrin]

For reference: @arnau and I discussed a bit and there were not clearly a "best way" of handling zope versions.

Using zope versions like ( https://raw.githubusercontent.com/zopefoundation/Zope/4.5.3/versions.cfg ) is good because we use the "canonical source of information", so it's easy to look for new version. One problem is that we don't use exactly these versions.

Sometimes slapos.cfg uses a different version ( zc.buildout of course, but also ZConfig = 2.9.3 or zope.interface = 5.2.0 ) sometimes ERP5 uses a different version from Zope: for security reasons like 571e78f2 or because we need to patch, like with Acquisition.

We could use versions from github and have explicit overrides in stack/erp5/buildout.cfg, but there are two questions:

• shall we use https://raw.githubusercontent.com/zopefoundation/Zope/4.5.3/versions.cfg or https://raw.githubusercontent.com/zopefoundation/Zope/4.5.3/versions-prod.cfg ?
• what shall be the order of extends ( or maybe why slapos.cfg needs ZConfig ? )

I hope I summarized correctly, I wanted to update also because my first comment sounds like I'm against this, but I'm not against.

[Jérome Perrin]

In a7e370de we updated Zope, I have used some curl commands to download zope profiles https://lab.nexedi.com/nexedi/slapos/blob/a7e370de393797dbb1f373622acfacdf51b36ed1/stack/erp5/buildout.cfg#L4-6

# versions pins from zope, vendored with
# curl https://zopefoundation.github.io/Zope/releases/4.8.2/versions.cfg | sed -e s/versions-prod.cfg/zope-versions.cfg/g > ztk-versions.cfg
# curl https://zopefoundation.github.io/Zope/releases/4.8.2/versions-prod.cfg > zope-versions.cfg

there are still many open questions, but I have noted there the procedure used