Added topic on reusing option values and buildout config options reference (#347)

* Added topic on reusing option values and buildout config options reference

* unbroke capitalization

* added another todo

But maybe I should stop doing this because sphinx tells me about
missing references.

* note that paths are whitespace separated.

* fixed some minor wording bugs.

* Added a hidden test to check my understanding of interaction with install-from-cache

* Clarify modes and semantics of extends-cache

* emphasize that in newst mode, we always download configuration

* expanded on logging control

* clarified socket-timeout

* spelling and it's

* simplify user-default example

* warn about unpinning when using develop eggs

* discuss unpinning

* explain why unpinning with empty pin is useful.

* reverted examples to use longer more explict forms

* typos

* Added tests for variables-extending-and-substitutions.rst

Also added more test helpers.

doc/getting-started.rst

@@ -56,7 +56,6 @@ with a parts option.  If we run Buildout:
    >>> run_buildout(src)
 
    >>> import os
-   >>> ls = lambda d='.': os.listdir(d)
    >>> eqs(ls(), 'buildout.cfg', 'bin', 'eggs', 'develop-eggs', 'parts', 'out')
 
    >>> eqs(ls('bin'))
@@ -403,6 +402,8 @@ Buildout to *not* check for newer versions of Python requirements:
 This relaxes repeatability, but with little risk if there was a recent
 run without this option.
 
+.. _pinned-versions:
+
 Pinned versions
 _______________
 
@@ -567,6 +568,30 @@ that your component only works a range of versions of some dependency,
 the express the range in your project requirements. Don't require
 specific versions.
 
+.. _unpinning-versions:
+
+Unpinning versions
+__________________
+
+You can unpin a version by just removing it (or commenting it out of)
+your ``versions`` section.
+
+You can also unpin a version by setting the version to an empty
+string:
+
+.. code-block:: ini
+
+  [versions]
+  ZEO =
+
+In an extending configuration (``buildout.cfg`` in the example above), or
+:ref:`on the buildout command line <unpinning-on-command-line>`.
+
+You might do this if pins are shared between projects and you want to
+unpin a requirement for one of the projects, or want to remove a pin
+while using a requirement in :ref:`development mode
+<python-development-projects>`.
+
 Buildout versions and automatic upgrade
 ---------------------------------------
 
@@ -602,6 +627,8 @@ Then Buildout will upgrade or downgrade to be consistent with version
 requirements.  See the :doc:`bootstrapping topic
 <topics/bootstrapping>` to learn more about bootstrapping.
 
+.. _python-development-projects:
+
 Python development projects
 ===========================
 

doc/reference.rst

@@ -2,6 +2,8 @@
 Reference
 =========
 
+.. _buildout-command-line:
+
 The Buildout command line
 =========================
 
@@ -11,6 +13,15 @@ A Buildout execution is of the form:
 
   buildout [buildout-options] [settings] [subcommand [subcommand-arguments]]
 
+Settings take the form ``section:option=value`` and override (or
+augment) settings in configuration files.  For example, to pin a
+version of ZEO you could use ``versions:ZEO=4.3.1``.  The section
+defaults to the ``buildout`` section.  So, for example: ``parts=test``
+sets the ``buildout`` section ``parts`` option.
+
+Command-line settings overrides can use ``+=`` and ``-=`` to
+:ref:`merge values with existing values <merge-values-with-existing-values>`
+
 Buildout options
 ----------------
 
@@ -42,3 +53,294 @@ It then runs the resulting buildout.
 
 See :ref:`Bootstrapping <init-generates-buildout.cfg>` for examples.
 
+
+.. _buildout-configuration-options-reference:
+
+Buildout configuration options
+===============================
+
+The standard buildout options are shown below.  Values of options with
+defaults shown can be used in :ref:`value substitutions
+<value-substitutions>`.
+
+abi-tag-eggs
+  A flag (true/false) indicating whether the eggs directory should be
+  divided into subdirectories by `ABI tag
+  <https://www.python.org/dev/peps/pep-0425/#abi-tag>`_.  This may be
+  useful if you use multiple Python builds with different build
+  options or different Python implementations.  It's especially
+  useful if you switch back and forth between PyPy and C Python.
+
+allow-hosts, default: '*'
+  Specify which hosts (as globs) you're willing to download
+  distributions from when following :ref:`dependency links
+  <use-dependency-links>`.
+
+allow-picked-versions, default: 'true'
+  Indicate whether it should be possible to install requirements whose
+  `versions aren't pinned <pinned-versions>`.
+
+bin-directory, default: bin
+  The directory where generated scripts should be installed. If this
+  is a relative path, it's evaluated relative to the buildout
+  directory.
+
+.. _develop-option:
+
+develop
+  One or more (whitespace-separated) paths to `distutils setup scripts
+  <https://docs.python.org/3.6/distutils/setupscript.html>`_ or (more
+  commonly) directories containing setup scripts named ``setup.py``.
+
+  See: :ref:`Python development projects <python-development-projects>`.
+
+develop-eggs-directory, default: 'develop-eggs'
+  The directory where :ref:`develop eggs
+  <python-development-projects>` should be installed. If this is a
+  relative path, it's evaluated relative to the buildout directory.
+
+directory, default: directory containing top-level buildout configuration
+  The top of the buildout.  Other directories specified (or
+  defaulting) with relative paths are created relative to this directory.
+
+.. _download-cache:
+
+download-cache
+  An optional directory in which to cache downloads. Python
+  distributions are cached in the ``dist`` subdirectory of this
+  directory.  Recipes may also cache downloads in this directory, or
+  in a subdirectory.
+
+  This is often set in a :ref:`User-default configuration
+  <user-default-configuration>` to share a cache between buildouts.
+  See the section on :doc:`Optimizing buildouts with shared eggs and
+  download caches <topics/optimizing>`.
+
+  If the value is a relative path and doesn't contain value
+  substitutions, it's interpreted relative to the directory containing
+  the configuration file that defined the value. (If it contains value
+  substitutions, and the result is a relative path, then it will be
+  interpreted relative to the buildout directory.)
+
+eggs-directory, default: 'eggs'
+  The directory where :ref:`eggs <eggs-label>` are installed.
+
+  This is often set in a :ref:`User-default configuration
+  <user-default-configuration>` to share eggs between buildouts.
+  See the section on :doc:`Optimizing buildouts with shared eggs and
+  download caches <topics/optimizing>`.
+
+  If the value is a relative path and doesn't contain value
+  substitutions, it's interpreted relative to the directory containing
+  the configuration file that defined the value. (If it contains value
+  substitutions, and the result is a relative path, then it will be
+  interpreted relative to the buildout directory.)
+
+executable, default: sys.executable, read-only
+  The full path to the Python executable used to run the buildout.
+
+.. _extends-option-ref:
+
+extends
+  The names, separated by whitespace, of one or more configurations
+  that the configuration containing the ``extends`` option should
+  :ref:`extend <extends_option>`. The names may be file paths, or
+  URLs.  If they are relative paths, they are interpreted relative to
+  the configuration containing the ``extends`` option.
+
+.. _extends-cache-buildout-option:
+
+extends-cache
+  An optional directory to cache remote configurations in.  Remote
+  configuration is configuration specified using a URL in an
+  :ref:`extends option <extends_option>` or as the argument to the
+  :ref:`-C buildout command-line option <-C-option>`. How the
+  extends-cache behaves depends on the buildout mode:
+
+  +---------------------------------+------------------------------+
+  | Mode                            | Behavior                     |
+  +=================================+==============================+
+  | :ref:`install-from-cache        | Configuration is retrieved   |
+  | <install-from-cache-mode>` or   | from cache if possible. If   |
+  | :ref:`offline <offline-mode>`   | configuration isn't cached,  |
+  |                                 | the buildout fails.          |
+  +---------------------------------+------------------------------+
+  | :ref:`non-newest                | Configuration is retrieved   |
+  | <non-newest-mode>`              | from cache if possible. If   |
+  |                                 | configuration isn't cached,  |
+  |                                 | then it is downloaded        |
+  |                                 | and saved in the cache.      |
+  +---------------------------------+------------------------------+
+  | Default                         | Configuration is downloaded  |
+  | (:ref:`newest <newest-mode>`)   | and saved in the cache, even |
+  |                                 | if it is already cached, and |
+  |                                 | the previously cached value  |
+  |                                 | is replaced.                 |
+  +---------------------------------+------------------------------+
+
+  If the value is a relative path and doesn't contain value
+  substitutions, it's interpreted relative to the directory containing
+  the configuration file that defined the value. (If it contains value
+  substitutions, and the result is a relative path, then it will be
+  interpreted relative to the buildout directory.)
+
+find-links, default: ''
+  Extra locations to search for distributions to download.
+
+  These may be file paths or URLs.  These may name individual
+  distributions or directories containing
+  distributions. Subdirectories aren't searched.
+
+index
+  An alternate index location.
+
+  This can be a local directory name or an URL.  It can be a flat
+  collection of distributions, but should be a "simple" index, with
+  subdirectories for distribution `project names
+  <https://packaging.python.org/distributing/#name>`_ containing
+  distributions for those projects.
+
+  If this isn't set, then ``https://pypi.python.org/simple/`` is used.
+
+.. _install-from-cache-mode:
+
+install-from-cache, default: 'false'
+  Enable install-from-cache mode.
+
+  In install-from-cache mode, no network requests should be made.
+
+  It's a responsibility of recipes to adhere to this.  Recipes that
+  would need to download files may use the :ref:`download cache
+  <download-cache>`.
+
+  The original purpose of the install-from-cache mode was to support
+  source-distribution of buildouts that could be built without making
+  network requests (mostly for security reasons).
+
+  This mode may only be used if a :ref:`download-cache
+  <download-cache>` is specified.
+
+installed, default: '.installed.cfg'
+  The name of the file used to store information about what's installed.
+
+  Buildout keeps information about what's been installed so it can
+  remove files created by parts that are removed and so it knows
+  whether to update or install new parts from scratch.
+
+  If this is a relative path, then it's interpreted relative to the
+  buildout directory.
+
+log-format, default: ''
+  `Format
+  <https://docs.python.org/3/library/logging.html#formatter-objects>`_
+  to use for log messages.
+
+  If ``log-format`` is blank, the default, Buildout will use the format::
+
+    %(message)s
+
+  for its own messages, and::
+
+    %(name)s: %(message)s
+
+  for the root logger [#root-logger]_.
+
+  If ``log-format`` is non-blank, then it will be used for the root logger
+  [#root-logger]_ (and for Buildout's messages).
+
+log-level, default: 'INFO'
+  The `logging level
+  <https://docs.python.org/3/library/logging.html#logging-levels>`_.
+
+  This may be adjusted with the :ref:`-v option <-v-option>` or the
+  :ref:`-q option <-q-option>`, which are the more common ways to control
+  the logging level.
+
+  The ``log-level`` option is rarely used.
+
+.. _newest-mode:
+
+.. _non-newest-mode:
+
+newest, default: 'true'
+  If true, check for newer distributions.  If false, then only look
+  for distributions when installed distributions don't satisfy requirements.
+
+  The goal of non-newest mode is to speed Buildout runs by avoiding
+  network requests.
+
+.. _offline-mode:
+
+offline, default: 'false'
+  If true, then offline mode is enabled.
+
+  .. Warning:: Offline mode is deprecated.
+
+     Its purpose has evolved over time and the end result doesn't
+     make much sense, but it is retained for backward compatibility.
+
+     If you think you want an offline mode, you probably want the
+     :ref:`install-from-cache <install-from-cache-mode>` mode instead.
+
+  In offline mode, no network requests should be made.  It's the
+  responsibility of recipes to adhere to this.  Recipes that would
+  need to download files may use the :ref:`download
+  cache <download-cache>`.
+
+  No distributions are installed in offline mode. If installed
+  distributions don't satisfy requirements, the the buildout will
+  error in offline mode.
+
+parts-directory, default: 'parts'
+  The directory where generated part artifacts should be installed. If this
+  is a relative path, it's evaluated relative to the buildout
+  directory.
+
+  If a recipe creates a file or directory, it will normally create it
+  in the parts directory with a name that's the same as the part name.
+
+prefer-final, default: 'true'
+  If true, then only `final distribution releases
+  <https://www.python.org/dev/peps/pep-0440/#final-releases>`_ will be
+  used unless no final distributions satisfy requirements.
+
+show-picked-versions, default: 'false'
+  If true, when Buildout finds a newest distribution for a
+  requirement that `wasn't pinned <pinned-versions>`, it will print
+  lines it would write to a versions configuration if the
+  :ref:`update-versions-file <update-versions-file>` option was used.
+
+socket-timeout, default: ''
+  Specify a socket timeout [#socket-timeout]_, in seconds, to use when
+  downloading distributions and other artifacts.  If non-blank, the
+  value must be a positive non-zero integer. If left blank, the socket
+  timeout is system dependent.
+
+  This may be useful if downloads are attempted from very slow
+  sources.
+
+.. _update-versions-file:
+
+update-versions-file, default: ''
+  If non-blank, this is the name of a file to write versions to when
+  selecting a distribution for a requirement whose version `wasn't
+  pinned <pinned-versions>`.  This file, typically ``versions.cfg``,
+  should end with a ``versions`` section (or whatever name is
+  specified by the ``versions`` option).
+
+.. _use-dependency-links:
+
+use-dependency-links, default: true
+  Distribution meta-data may include URLs, called dependency links, of
+  additional locations to search for distribution dependencies.  If
+  this option is set to ``false``, then these URLs will be ignored.
+
+versions, default 'versions'
+  The name of a section that contains :ref:`version pins <pinned-versions>`.
+
+.. [#root-logger] Generally, the root logger format is used for all
+   messages unless it is overridden by a lower-level logger.
+
+.. [#socket-timeout] This timeout reflects how long to wait on
+   individual socket operations. A slow request may take much longer
+   than this timeout.

doc/topics/history.rst

@@ -70,6 +70,8 @@ To achieve Buildout's goals, it was necessary to interact with
 setuptools at a much lower level and to write quite a bit more
 packaging logic than planned.
 
+.. _eggs-label:
+
 Eggs
 ----
 

doc/topics/index.rst

@@ -6,9 +6,11 @@ Buildout Topics
    :maxdepth: 2
 
    history
+   variables-extending-and-substitutions
    bootstrapping
 
 .. todo:
 
-   variables-extending-and-substitutions
    writing-recipes
+   optimizing
+

doc/topics/variables-extending-and-substitutions.rst

+======================================================================
+Staying DRY with value substitutions, extending, and macros
+======================================================================
+
+A buildout configuration is a collection of sections, each holding a
+collection of options.  It's common for option values to be repeated
+across options.  For examples, many file-path options might start
+with common path prefixes. Configurations that include clients and
+servers might share server-address options.  This topic presents
+various ways you can reuse option values without repeating yourself.
+
+.. _value-substitutions:
+
+Value substitutions
+======================
+
+When supplying values in a configuration, you can include values from
+other options using the syntax::
+
+  ${SECTION:OPTION}
+
+For example: ``${buildout:directory}`` refers to the value of the
+``directory`` option in the in the ``buildout`` section of the
+configuration.  The value of the referenced option will be substituted
+for the referencing text.
+
+You can simplify references to options in the current section by omitting the
+section name.  If we wanted to use the ``buildout`` ``directory``
+option from within the ``buildout`` section itself, we could use
+``${:directory}``.  This convenience is especially useful in
+:ref:`macros <macros-label>`, which we'll discuss later in this topic.
+
+There's a special value that's also useful in macros, named
+``_buildout_section_name_``, which has the name of the current
+section. We'll show how this is used when we discuss :ref:`macros
+<macros-label>`.
+
+Default and computed option values
+===================================
+
+Many sections have option values that can be used in substitutions
+without being defined in a configuration.
+
+The ``buildout`` section, where settings for the buildout as a whole
+are provided has many default option values. For example, the
+directory where scripts are installed is configurable and the value is
+available as ``${buildout:bin-directory}``.  See the :ref:`Buildout
+options reference <buildout-configuration-options-reference>` for a
+complete list of Buildout options that can be used in substitutions.
+
+Many recipes also have options that have defaults or that are computed and
+are available for substitutions.
+
+Sources of configuration options
+====================================
+
+Configuration option values can come from a number of sources (in
+increasing precedence):
+
+software default values
+  These are defined by buildout and recipe sources.
+
+user default values
+  These are set in :ref:`per-user default configuration files
+  <user-default-configuration>` and override default values.
+
+options from one or more configuration files
+  These override user defaults and each other, as described below.
+
+option values in the :ref:`buildout command line <buildout-command-line>`
+  These override configuration-file settings.
+
+.. _extends_option:
+
+Extending configuration files
+================================
+
+The :ref:`extends <extends-option-ref>` option in a ``buildout``
+section can be used to extend one or more configuration files.  There
+are a number of applications for this. For example, common options for
+a set of projects might be kept in a common base configuration.  A
+production buildout could extend a development buildout, or they could
+both extend a common base.
+
+The option values in the extending configuration file override those
+in the files being extended.  If multiple configurations are named in
+the ``extends`` option (separated by whitespace), then the
+configurations are processed in order from left/top to right/bottom,
+with the later (right/bottom) configurations overriding earlier
+(left/top) ones. For example, in:
+
+.. code-block:: ini
+
+   extends = base1.cfg base2.cfg
+             base3.cfg
+
+.. -> src
+
+    >>> write("[buildout]\na=11\nb=12\n", 'base1.cfg')
+    >>> write("[buildout]\nb=21\nc=22\n", 'base2.cfg')
+    >>> write("[buildout]\nc=31\nd=32\n", 'base3.cfg')
+    >>> write("[buildout]\nparts=\n" + src, 'buildout.cfg')
+    >>> run_buildout("buildout -vv")
+    >>> print(read()) # doctest: +ELLIPSIS
+    Creating ...
+    [buildout]
+    a = 11
+    allow-hosts = *
+    allow-picked-versions = true
+    b = 21
+    bin-directory = ...
+    c = 31
+    d = 32
+    develop-eggs-directory = ...
+
+    >>> clear_here()
+
+The options in the configuration using the extends option override the
+options in ``base3.cfg``, which override the options in ``base2.cfg``,
+which override the options in ``base1.cfg``.
+
+Base configurations may be extended multiple times. For example, in
+the example above, ``base1.cfg`` might, itself, extend ``base3.cfg``,
+or they might both extend a common base configuration.  Of course, cycles
+are not allowed.
+
+Configurations may be named with URLs in the ``extends`` option, in
+which case they may be downloaded from remote servers.  See :ref:`The
+extends-cache buildout option <extends-cache-buildout-option>`.
+
+When a relative path is used in an extends option, it's interpreted
+relative to the path of the extending configuration.
+
+.. _user-default-configuration:
+
+User-default configuration
+==============================
+
+A per-user default configuration may be defined in the ``default.cfg``
+file in the ``.buildout`` subdirectory of a user's home directory
+(``~/.buildout/default.cfg`` on Mac OS and Linux).  This configuration
+is typically used to set up a shared egg or cache directory, as in:
+
+.. code-block:: ini
+
+  [buildout]
+  eggs-directory = ~/.buildout/eggs
+  download-cache = ~/.buildout/download-cache
+  abi-tag-eggs = true
+
+.. -> src
+
+    >>> import os
+    >>> os.makedirs(join('home', '.buildout'))
+    >>> write(src, 'home', '.buildout', 'default.cfg')
+    >>> write("""\
+    ... [buildout]
+    ... parts = bobo
+    ... [bobo]
+    ... recipe=zc.recipe.egg
+    ... eggs=bobo
+    ... """, "buildout.cfg")
+    >>> run_buildout()
+    >>> eqs(ls(),
+    ...     'out', 'home', '.installed.cfg', 'buildout.cfg',
+    ...     'develop-eggs', 'parts', 'bin')
+    >>> eqs(ls(join('home', '.buildout')),
+    ...     'default.cfg', 'eggs', 'download-cache')
+    >>> [abieggs] = ls(join('home', '.buildout', 'eggs'))
+    >>> eqs([n.split('-', 1)[0]
+    ...      for n in ls('home', '.buildout', 'eggs', abieggs)],
+    ...     'bobo', 'WebOb')
+
+    >>> clear_here()
+
+See the section on :doc:`optimizing buildouts with shared eggs and
+download caches <topics/optimizing>` for an explanation of the options
+used in the example above.
+
+.. _merge-values-with-existing-values:
+
+Merging, rather than overriding values
+========================================
+
+Normally, values in extending configurations override values in
+extended configurations by replacing them, but it's also possible to
+augment or trim overridden values.  If ``+=`` is used rather than
+``=``, the overriding option value is appended to the original. So,
+for example if we have a base configuration, ``buildout.cfg``:
+
+.. code-block:: ini
+
+   [buildout]
+   parts =
+     py
+     test
+     server
+   ...
+
+.. -> src
+
+   >>> py_part = """
+   ... [{name}]
+   ... recipe = zc.recipe.egg
+   ... eggs = bobo
+   ... scripts = {name}
+   ... interpreter = {name}
+   ... """
+   >>> parts = (py_part.format(name='py')
+   ...        + py_part.format(name='test')
+   ...        + py_part.format(name='server'))
+   >>> write(src.replace('...', parts), 'buildout.cfg')
+   >>> run_buildout()
+   >>> eqs(ls('bin'), 'py', 'test', 'server')
+
+And a production configuration ``prod.cfg``, we can add another part,
+``monitor``, like this:
+
+.. code-block:: ini
+
+   [buildout]
+   extends = buildout.cfg
+   parts += monitor
+   ...
+
+.. -> src
+
+   >>> write(src.replace('...', py_part.format(name='monitor')), 'e.cfg')
+   >>> run_buildout("buildout -N -c e.cfg")
+   >>> eqs(ls('bin'), 'py', 'test', 'server', 'monitor')
+
+In this example, we didn't have to repeat (or necessarily know) the
+base parts to add the ``monitor`` part.
+
+We can also subtract values using ``-=``, so if we wanted to exclude
+the ``test`` part in production:
+
+.. code-block:: ini
+
+   [buildout]
+   extends = buildout.cfg
+   parts += monitor
+   parts -= test
+   ...
+
+.. -> src
+
+   >>> write(src.replace('...', py_part.format(name='monitor')), 'e.cfg')
+   >>> run_buildout("buildout -N -c e.cfg")
+   >>> eqs(ls('bin'), 'py', 'server', 'monitor')
+
+   >>> clear_here()
+
+Something to keep in mind is that this works by *lines*.  The ``+=``
+form adds the lines in the new data to the lines of the
+old. Similarly, ``-=`` removes *lines* in the overriding option from the
+original *lines*. This is a bit delicate.  In the example above,
+we were careful to put the base values on separate lines, in
+anticipation of using ``-=``.
+
+Merging values also works with option settings provided via the
+:ref:`buildout command line <buildout-command-line>`.  For example, if
+you want to temporarily use a :ref:`development version
+<python-development-projects>` of another project, you can augment the
+buildout :ref:`develop option <develop-option>` on the command-line
+when running buildout:
+
+.. code-block:: console
+
+   buildout develop+=/path/to/other/project
+
+.. -> src
+
+   >>> write("import setuptools; setuptools.setup(name='a')", "setup.py")
+   >>> write("""
+   ... [buildout]
+   ... develop=.
+   ... parts=py
+   ... [py]
+   ... recipe=zc.recipe.egg
+   ... eggs = a
+   ...        b
+   ... [versions]
+   ... b=1
+   ... """, "buildout.cfg")
+   >>> os.mkdir('b')
+   >>> write("import setuptools; setuptools.setup(name='b', version=1)",
+   ...       "b", "setup.py")
+   >>> run_buildout(src.replace('/path/to/other/project', 'b'))
+   >>> eqs(ls('develop-eggs'), 'b.egg-link', 'a.egg-link')
+
+.. _unpinning-on-command-line:
+
+Although, if you've pinned the version of that project, you'll need to
+:ref:`unpin it <unpinning-versions>`, which you can also do on the command-line:
+
+.. code-block:: console
+
+   buildout develop+=/path/to/other/project versions:projectname=
+
+.. -> src
+
+   >>> write("import setuptools; setuptools.setup(name='b', version=2)",
+   ...       "b", "setup.py")
+   >>> run_buildout(src.replace('/path/to/other/project', 'b')
+   ...                 .replace('projectname', 'b'))
+   >>> eqs(ls('develop-eggs'), 'b.egg-link', 'a.egg-link')
+
+   >>> clear_here()
+
+.. _macros-label:
+
+Extending sections using macros
+===============================
+
+We can extend other sections in a configuration as macros by naming
+then using the ``<`` option.  For example, perhaps we have to create
+multiple server processes that listen on different ports.  We might
+have a base ``server`` section, and some sections that use it as a
+macro:
+
+.. code-block:: ini
+
+   [server]
+   recipe = zc.zdaemonrecipe
+   port = 8080
+   program =
+     ${buildout:bin-directory}/serve
+        --port ${:port}
+        --name ${:_buildout_section_name_}
+
+   [server1]
+   <= server
+   port = 8081
+
+   [server2]
+   <= server
+   port = 8082
+
+.. -> src
+
+   >>> write("[buildout]\nparts=server server1 server2\n" + src, "buildout.cfg")
+   >>> run_buildout("buildout -vv")
+   >>> print(read()) # doctest: +ELLIPSIS
+   Creating ...
+   [server]
+   ...
+   port = 8080
+   program = .../bin/serve...--port 8080...--name server
+   ...
+   recipe = zc.zdaemonrecipe
+   ...
+   [server1]
+   ...
+   port = 8081
+   program = .../bin/serve...--port 8081...--name server1
+   ...
+   recipe = zc.zdaemonrecipe
+   ...
+   [server2]
+   ...
+   port = 8082
+   program = .../bin/serve...--port 8082...--name server2
+   ...
+   recipe = zc.zdaemonrecipe
+   ...
+
+In the example above, the ``server1`` and  ``server2`` sections use the
+``server`` section, getting its ``recipe`` and ``program`` options.
+The resulting configuration is equivalent to:
+
+.. code-block:: ini
+
+   [server]
+   recipe = zc.zdaemonrecipe
+   port = 8080
+   program =
+     ${buildout:bin-directory}/serve
+        --port ${:port}
+        --name ${:_buildout_section_name_}
+
+   [server1]
+   recipe = zc.zdaemonrecipe
+   port = 8081
+   program =
+     ${buildout:bin-directory}/serve
+        --port ${:port}
+        --name ${:_buildout_section_name_}
+
+   [server2]
+   recipe = zc.zdaemonrecipe
+   port = 8082
+   program =
+     ${buildout:bin-directory}/serve
+        --port ${:port}
+        --name ${:_buildout_section_name_}
+
+.. -> src
+
+   >>> write("[buildout]\nparts=server server1 server2\n" + src, "buildout.cfg")
+   >>> run_buildout("buildout -vv")
+   >>> print(read()) # doctest: +ELLIPSIS
+   Installing ...
+   [server]
+   ...
+   port = 8080
+   program = .../bin/serve...--port 8080...--name server
+   ...
+   recipe = zc.zdaemonrecipe
+   ...
+   [server1]
+   ...
+   port = 8081
+   program = .../bin/serve...--port 8081...--name server1
+   ...
+   recipe = zc.zdaemonrecipe
+   ...
+   [server2]
+   ...
+   port = 8082
+   program = .../bin/serve...--port 8082...--name server2
+   ...
+   recipe = zc.zdaemonrecipe
+   ...
+
+Value substitutions in the base section are applied after its
+application as a macro, so the substitutions are applied using data
+from the sections that used the macro (using the ``<`` option).
+
+You can extend multiple sections by listing them in the ``<`` option
+on separate lines, as in:
+
+.. code-block:: ini
+
+   [server2]
+   <= server
+      monitored
+   port = 8082
+
+.. -> src
+
+   >>> old = read('buildout.cfg')
+   >>> write(old + src + """
+   ... [monitored]
+   ... name = ${:_buildout_section_name_}
+   ... mport = 1${:port}
+   ... """, "buildout.cfg")
+   >>> run_buildout("buildout -vv")
+   >>> print(read()) # doctest: +ELLIPSIS
+   Installing ...
+   [server2]
+   ...
+   mport = 18082
+   name = server2
+   port = 8082
+   program = .../bin/serve...--port 8082...--name server2
+   ...
+   recipe = zc.zdaemonrecipe
+   ...
+
+If multiple sections are extended, they're processed in order, with
+later ones taking precedence.  In the example above, if both
+``server`` and ``monitored`` provided an option, then the value from
+``monitored`` would be used.
+
+A section that's used as a macro can extend another section.

src/zc/buildout/extends-cache.txt

@@ -107,6 +107,11 @@ base.cfg from the cache:
 >>> print_(system(buildout + ' -o'))
 Unused options for buildout: 'foo'.
 
+.. verify same behavior for install-from-cache:
+
+   >>> print_(system(buildout + ' install-from-cache=true download-cache=.'))
+   Unused options for buildout: 'foo'.
+
 In online mode, buildout will download and use the modified version:
 
 >>> print_(system(buildout))

src/zc/buildout/tests.py

@@ -3455,7 +3455,8 @@ normalize_S = (
     )
 
 def run_buildout(command):
-    os.environ['HOME'] = os.getcwd() # Make sure we don't get .buildout
+    # Make sure we don't get .buildout
+    os.environ['HOME'] = os.path.join(os.getcwd(), 'home')
     args = command.strip().split()
     import pkg_resources
     buildout = pkg_resources.load_entry_point(
@@ -3751,18 +3752,32 @@ def test_suite():
                 with open(path) as f:
                     return f.read()
 
-            def write(text, path):
-                with open(path, 'w') as f:
+            def write(text, *path):
+                with open(os.path.join(*path), 'w') as f:
                     f.write(text)
 
+            def eqs(a, *b):
+                a = set(a); b = set(b)
+                return None if a == b else (a - b, b - a)
+
+            def clear_here():
+                for name in os.listdir('.'):
+                    if os.path.isfile(name):
+                        os.remove(name)
+                    else:
+                        shutil.rmtree(name)
+
             test.globs.update(
                 run_buildout=run_buildout_in_process,
                 yup=lambda cond, orelse='Nope': None if cond else orelse,
                 nope=lambda cond, orelse='Nope': orelse if cond else None,
                 eq=lambda a, b: None if a == b else (a, b),
-                eqs=lambda a, *b: None if set(a) == set(b) else (a, b),
+                eqs=eqs,
                 read=read,
                 write=write,
+                ls=lambda d='.', *rest: os.listdir(os.path.join(d, *rest)),
+                join=os.path.join,
+                clear_here=clear_here
                 )
             setupstack.setUpDirectory(test)
 
@@ -3771,6 +3786,9 @@ def test_suite():
                 manuel.doctest.Manuel() + manuel.capture.Manuel(),
                 os.path.join(docdir, 'getting-started.rst'),
                 os.path.join(docdir, 'topics', 'bootstrapping.rst'),
+                os.path.join(
+                    docdir,
+                    'topics', 'variables-extending-and-substitutions.rst'),
                 setUp=docSetUp, tearDown=setupstack.tearDown
                 ))