1. 20 May, 2024 18 commits
    • Kazuhiko Shiozaki's avatar
    • Kazuhiko Shiozaki's avatar
      [why?] Ignore empty value for 'download-cache' in 'buildout' section · c9ab3010
      Kazuhiko Shiozaki authored
      Even though such configuration is wrong...
      c9ab3010
    • Julien Muchembled's avatar
      [feat] Enable partial support for += / -= with <= · d1dc3b1c
      Julien Muchembled authored
      Before this, += and -= worked well when applied to keys defined in the
      current section or in a section of the same name inherited from a file
      extended by the current file, but not to keys inherited from another
      section using macro extension <=, because <= substitution happens later
      and all the += and -= are already resolved.
      
      e.g.:
      
      ```
      [macro]
      a = 1
      
      [part]
      <= macro
      a += 2
      ```
      
      was equivalent to
      
      ```
      [part]
      a = 1
      a = 2
      ```
      
      instead of
      
      ```
      [part]
      a = 1
      a += 2
      ```
      
      Now a partial and brittle support for this is enabled by postponing
      += and -= resolution until <= substitution happens when the current
      section contains a <= extension. But this is not guaranteed: if the
      current section is overloaded in another file extending the current
      one, then += and -= resolution will happend then, before <=.
      
      A consistent solution would be to unify the implementation of
      extends= and <= and update sections by taking both into account,
      instead of updating based only on extends and computing <= later,
      leading to inconsistencies. This could be achieved e.g. by computing
      section updates on demand during substitution.
      Co-authored-by: Xavier Thompson's avatarXavier Thompson <xavier.thompson@nexedi.com>
      d1dc3b1c
    • Julien Muchembled's avatar
      [fix] Fix +=/-= support for keys in the same file · a53abd21
      Julien Muchembled authored
      Before this, += and -= worked correctly when applied to a key defined
      in a file extended by the current file, but not when the key was in
      the same file.
      
      e.g.:
      
      ```
      [part]
      a = this case is
      ```
      
      ```
      [buildout]
      extends = a.cfg
      
      [part]
      a += ok
      b = this case is
      b += not-ok
      ```
      Co-authored-by: Xavier Thompson's avatarXavier Thompson <xavier.thompson@nexedi.com>
      a53abd21
    • Xavier Thompson's avatar
      [test] Disable extends-cache.txt test · 70993203
      Xavier Thompson authored
      This test asserts buildout's behavior with regards to download options,
      and this was changed by the new algorithm for extends. Tests for the
      new behavior have not been written yet.
      70993203
    • Xavier Thompson's avatar
      [feat] Reimplement the extends algorithm · 1e00cce7
      Xavier Thompson authored
      The new algorithm avoids fetching the same extended file more than once
      and correctly handles overriding values and += and -=:
      
      The new algorithm starts as if there was a buildout file containing
      
      ```
      [buildout]
      extends =
        user/defaults.cfg # if it exists
        buildout.cfg # if it exists
        command_line_extends.cfg # if passed on the command line
      ```
      
      The files are then fetched in depth-first-search postorder and fetching
      child nodes in the order given by the extends directive, ignoring files
      that have already been fetched.
      
      The buildout dicts are then collected in order, and this linearisation
      is then merged at the end, overriding the first configs collected with
      the later ones. The first dict in the linearisation is not from a file,
      but the dict of buildout's (hardcoded) defaults. This is equivalent to
      acting as though every file that does not extend anything extends these
      defaults.
      
      The first time a file must be downloaded from a url, the linearisation
      is merged with the configs already collected, and the resulting options
      are then used to determine the download options for this download, and
      every subsequent download.
      
      This is a break with buildout's current logic for download options.
      
      By analogy with classes in Python, we are computing a linearisation of
      the class hierarchy to determine the method resolution order (MRO).
      This algorithm is not the same as Python's MRO since Python 2.3 (C3).
      
      It could be good to switch to a C3 linearisation like Python.
      1e00cce7
    • Xavier Thompson's avatar
      [opti] Unannotate sections in-place · 7cde53c4
      Xavier Thompson authored
      This avoids unecessary copies. This is a preparatory step to
      reimplementing the extends algorithm. It may be that this breaks
      the extends algorithm as it is currently implemented.
      7cde53c4
    • Xavier Thompson's avatar
      [opti] Update sections in-place · 6b0ea6d0
      Xavier Thompson authored
      This avoids unecessary deepcopies. This is a preparatory step to
      reimplementing the extends algorithm. It may be that this breaks
      the extends algorithm as it is currently implemented.
      6b0ea6d0
    • Xavier Thompson's avatar
      [opti] Remove redundant deepcopies · 25140175
      Xavier Thompson authored
      25140175
    • Xavier Thompson's avatar
      [fix] zc.recipe.egg: Fix ls(eggs) tests · 4f666926
      Xavier Thompson authored
      Depending on the test environment, pip, setuptools and wheel dists
      may be installed as .egg or .egg-link and may be files or folders.
      4f666926
    • Xavier Thompson's avatar
      [fix] Fix ls(eggs) tests · 7e6afe30
      Xavier Thompson authored
      Depending on the test environment, pip, setuptools and wheel dists
      may be installed as .egg or .egg-link and may be files or folders.
      7e6afe30
    • Xavier Thompson's avatar
      [fix] Fix picked versions tests in test_all.py · f886a69c
      Xavier Thompson authored
      The general case is that buildout will pick versions for setuptools,
      pip and wheel when they are not pinned: it's only when these are
      installed as .egg-link - either as actual develop eggs or as a
      side-effect of being found in .dist-info format in the environment
      e.g. in site-packages - that buildout does not consider it picked
      a version for these eggs.
      
      This affects tests that set `allow-picked-versions = false` or
      `show-picked-versions = true` or `update-versions-file = ...`.
      f886a69c
    • Xavier Thompson's avatar
      [fix] Fix repeatability tests · 6905f409
      Xavier Thompson authored
      6905f409
    • Xavier Thompson's avatar
      [tool] Add dev.py --isolate option · 19fa1dbd
      Xavier Thompson authored
      This creates a environment for tests that is isolated from the current
      environment (as long as there are no .egg-link to e.g. site-packages).
      19fa1dbd
    • Kazuhiko Shiozaki's avatar
      [feat] Revert "Removing develop-eggs/ directory before bootstrapping." · b323afe2
      Kazuhiko Shiozaki authored
      In SlapOS, bootstrap is called each time software release is invoked and
      we do not want to delete develop-eggs directory.
      
      This reverts commit 55d76b34.
      b323afe2
    • Xavier Thompson's avatar
      [fix] zc.recipe.egg: Fix tests with recent setuptools · a7e56604
      Xavier Thompson authored
      With recent setuptools, the way the test package index is laid out
      causes a warning log to be emitted, which needs to be filtered out
      to keep matching the expected test output.
      a7e56604
    • Xavier Thompson's avatar
      [cleanup] Remove superfluous code after pip wheel · 395c6f36
      Xavier Thompson authored
      Now that we use `pip wheel` + `Wheel.install_as_egg` to produce
      genuine eggs, we don't need to workaround the fact that we have
      not-actually-egg `.egg` bundles anymore.
      
      If we still do need to allow for non-eggs being installed in `~/.eggs`,
      via extensions or others, we should revert this.
      See https://github.com/buildout/buildout/pull/352
      395c6f36
    • Xavier Thompson's avatar
      [dev] Raise setuptools requirement to >=38.2.3 · 322bdd43
      Xavier Thompson authored
      Needed for pip wheel + setuptools.wheel.Wheel.install_as_egg.
      322bdd43
  2. 17 May, 2024 15 commits
    • Xavier Thompson's avatar
      [feat] Use pip wheel + Wheel.install_as_egg · b08cf932
      Xavier Thompson authored
      `pip install <package>` produces a `<package-name>` package folder
      and a `<package-name>.dist-info` metadata folder, which is another
      format than eggs. Then buildout bundles both folders into a parent
      folder `<package.egg>` and tries to act as though it were an egg.
      
      Instead, use `pip wheel` to produce a wheel - which `pip install`
      does internally anyway - and `setuptools.Wheel.install_as_egg` to
      produce a genuine egg.
      
      This is much cleaner: it consistently produces genuine eggs instead
      of sometimes true eggs, sometimes `.dist-info` bundles depending on
      whether `pip install` is called or the package was installed from a
      `.whl` or `.egg` archive directly.
      
      The only downside it this requires setuptools >= 38.2.3.
      b08cf932
    • Xavier Thompson's avatar
      [fix] Add each .egg and .dist-info to environment · 58b8ec8d
      Xavier Thompson authored
      Since zc.buildout uses `pip` to install dists, most .egg in ./eggs and
      ./develop-eggs (in case of dists installed with zc.recipe.egg:custom)
      are not actually eggs, but installed wheels with .dist-info format
      which have been bundled into a .egg containing .dist-info metadata
      folder as well as the actual package folder directly alongside.
      
      Unlike actual eggs, such */*.dist-info bundles are not stepped into
      by package_resources when the containing directory is add to the
      environment, so each .egg bundle needs to be added individually.
      
      Buildout already does this for bundles in the target directory of
      egg installation, usually ./eggs, but e.g. in offline mode there
      is no target directory and then already installed .egg bundles
      are not detected.
      
      So now every dist path is added to the environment individual,
      including for paths where buildout looks for already installed
      eggs but will not install eggs to, such as ./develop-eggs, and
      also ./eggs in offline mode or when using zc.recipe.egg:custom
      and the target is ./develop-eggs and ./eggs is such a scan-from
      but don't-install-to path.
      58b8ec8d
    • Xavier Thompson's avatar
      bfedbc34
    • Xavier Thompson's avatar
      9f45ce37
    • Julien Muchembled's avatar
      [feat] Omit Python library in script paths · 134008a5
      Julien Muchembled authored
      This is useful when using OS Python & eggs.
      
      Useless for SlapOS.
      134008a5
    • Xavier Thompson's avatar
      77ca6153
    • Xavier Thompson's avatar
      [doc] Show getting-started with modern python venv · 19f6addd
      Xavier Thompson authored
      Also show `pip install <url-for-tar.gz-of-master-branch-on-gitlab>`.
      19f6addd
    • Xavier Thompson's avatar
      [fix] Allow only installable DEVELOP_DIST in index · 85619ff1
      Xavier Thompson authored
      The package index considers may yield develop dists found on its local
      search_path - sys.path by default - in addition to its index_url.
      
      This commit discards such dists when they do not contain a setup.py
      nor a pyproject.toml, as a good heuristic of whether they will be
      installable or not.
      85619ff1
    • Xavier Thompson's avatar
      [fix] Fix package index selection indexing · 07a270fd
      Xavier Thompson authored
      Index constructed package indexes by all their parameters.
      07a270fd
    • Xavier Thompson's avatar
      [feat] Add buildout:extra-paths option · 1f3977f8
      Xavier Thompson authored
      This option determines what paths zc.buildout will scan for already
      installed distributions. It can be set to an empty value to enable
      isolation. The special values 'sys.path', 'legacy' and 'zc.buildout'
      expand to particular paths. The default is 'zc.buildout'.
      
      The 'sys.path' mode simply expands to the currrent sys.path.
      
      The 'legacy' mode reproduces the previous behavior of specifically
      scanning the paths of the current zc.buildout distribution and its
      dependencies using a weird ordering logic that bears no relation to the
      order they appear in sys.path - which can produce unexpected results as
      other distributions for zc.buildout, pip, wheel and setuptools may take
      precedence over the ones currently running.
      
      The 'zc.buildout' mode is similar to 'legacy' as it uses only the paths
      of the currently running zc.buildout and dependencies, but unlike
      'legacy' mode it respects the order in which these appear in sys.path,
      avoiding unexpected results.
      
      This mode is set to the default because it is closest to 'legacy' mode
      and because it has a nice property: running in succession
      
        `buildout buildout:extra-paths= bootstrap` (1)
        `bin/buildout` (2)
      
      will result in (1) installing zc.buildout and its dependencies in
      ./eggs in isolation from the environment, and (2) using only the
      paths of these in ./eggs, i.e. continuing to operate in isolation,
      even without setting extra-paths explictly.
      1f3977f8
    • Xavier Thompson's avatar
      [fix] Fix bootstrap working set order · bec365d0
      Xavier Thompson authored
      In bootstrap we potentially copy eggs from the working set to ./eggs.
      We then reconstruct the same working set using the moved locations.
      
      This commits ensures we keep a correct working set order throughout
      and that we avoid activating unintended dists.
      bec365d0
    • Xavier Thompson's avatar
      [fix] Fix working set sorting · b05e8a41
      Xavier Thompson authored
      If a dist in the computed working set is at a location shared with
      other dists - such as site-packages - then when generating scripts
      these other packages may overshadow the next items on the sys.path
      and result in importing a different version than the one installed
      and intended by buildout.
      
      To avert this, a sort of the working set was introduced at various
      points just before generating a script.
      
      However, that sort put the paths referenced from an `.egg-link` in
      ./develop-eggs first. This is truly problematic because dists from
      site-packages which are not eggs - e.g. dists installed with pip -
      can become referenced as `.egg-link` during buildout bootstrap and
      the sort then causes site-packages to be one of the first items in
      sys.path.
      
      In particular when running buildout bootstrap from a venv in which
      zc.buildout was installed by pip, if any one of zc.buildout or its
      dependencies from the venv meets the version requirements, then it
      can cause the generated bin/buildout to import the dists only from
      the venv's site-packages, even when some do not meet requirements.
      
      To fix this, the sort now puts the dists from `./eggs` first as we
      know their locations contain only a single dist, and then puts the
      dists from ./develop-eggs which have locations inside the buildout
      directory before the others.
      
      The previous sort was also activating all the dists from the paths
      of the already activated dists.
      
      Note that this also means that any working set must be manipulated
      with care in general to avoid activating unintended dists from the
      locations of the already activated dists.
      b05e8a41
    • Xavier Thompson's avatar
      [fix] Use only ws.find in install · 815cdf75
      Xavier Thompson authored
      Replace `pkg_resources.Environment(ws.entries).best_match(req, ws)`
      with `ws.find(req)`.
      
      The first already starts by calling `ws.find(req)` to attempt to find
      an already activated dist in the working set, but if none is found it
      then proceeds to scan through the entries of the environment - i.e.
      the locations of the directly-requested distributions, `ws.entries` -
      to activate a dist at these locations if one matches.
      
      This is problematic when directly-requested distributions are found in
      a location shared by multiple dists, such as site-packages: this gives
      that location precedence over the normal order of locations scanned by
      easy_install, and can result in undesired versions being chosen over
      versions available in ./eggs or in ./develop-eggs. The random aspect
      of this is also problematic, as the order of paths considered will
      depend on the order of the directly-requested distributions and where
      they are found.
      815cdf75
    • Xavier Thompson's avatar
      [feat] Respect pinned versions in bootstrap · 765fd23e
      Xavier Thompson authored
      If zc.buildout or its dependencies have pinned versions that do not
      match the currently running versions, they are now installed in the
      local eggs directory from scratch according to the pinned versions.
      
      In offline mode this merely ensures that versions that satisfy the
      requirements are already available. This is the case when the eggs
      are already installed, or when the running versions are a match to
      the pinned versions or the absence of a pinned version.
      
      If after this matching versions of zc.buildout and its dependencies
      are not located in the local eggs or develop-eggs directories, they
      are copied there as was already the case before.
      765fd23e
    • Xavier Thompson's avatar
      58cbdbd5
  3. 13 Mar, 2024 1 commit
  4. 08 Nov, 2022 2 commits
  5. 06 Nov, 2022 4 commits